Skip to content

Commit

Permalink
[#6504] feat(iceberg): Support s3 path-style-access properties for Ic…
Browse files Browse the repository at this point in the history 
…eberg catalog and Iceberg REST server (#6541)

### What changes were proposed in this pull request?

Support s3 path-style-access properties in Iceberg REST server

### Why are the changes needed?

Fix: #6504

### Does this PR introduce _any_ user-facing change?

N/A

### How was this patch tested?

Manual
  • Loading branch information
@zhoukangcn
zhoukangcn authored Feb 27, 2025
1 parent 1650c20 commit 6da2f79
Show file tree
Hide file tree
Showing 7 changed files with 22 additions and 5 deletions.
1 change: 1 addition & 0 deletions ...common/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergConstants.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ public class IcebergConstants {
public static final String ICEBERG_S3_ACCESS_KEY_ID = "s3.access-key-id";
public static final String ICEBERG_S3_SECRET_ACCESS_KEY = "s3.secret-access-key";
public static final String ICEBERG_S3_TOKEN = "s3.session-token";
public static final String ICEBERG_S3_PATH_STYLE_ACCESS = "s3.path-style-access";
public static final String AWS_S3_REGION = "client.region";

public static final String ICEBERG_OSS_ENDPOINT = "oss.endpoint";
Expand Down
2 changes: 2 additions & 0 deletions .../src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergPropertiesUtils.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,8 @@ public class IcebergPropertiesUtils {
map.put(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID, IcebergConstants.ICEBERG_S3_ACCESS_KEY_ID);
map.put(
S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY, IcebergConstants.ICEBERG_S3_SECRET_ACCESS_KEY);
map.put(
S3Properties.GRAVITINO_S3_PATH_STYLE_ACCESS, IcebergConstants.ICEBERG_S3_PATH_STYLE_ACCESS);
// OSS
map.put(OSSProperties.GRAVITINO_OSS_ENDPOINT, IcebergConstants.ICEBERG_OSS_ENDPOINT);
map.put(OSSProperties.GRAVITINO_OSS_ACCESS_KEY_ID, IcebergConstants.ICEBERG_OSS_ACCESS_KEY_ID);
Expand Down
3 changes: 3 additions & 0 deletions catalogs/catalog-common/src/main/java/org/apache/gravitino/storage/S3Properties.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,5 +40,8 @@ public class S3Properties {
// The S3 credentials provider class name.
public static final String GRAVITINO_S3_CREDS_PROVIDER = "s3-creds-provider";

// The S3 path style access flag.
public static final String GRAVITINO_S3_PATH_STYLE_ACCESS = "s3-path-style-access";

private S3Properties() {}
}
11 changes: 6 additions & 5 deletions docs/iceberg-rest-service.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -107,11 +107,12 @@ Please refer to [Credential vending](./security/credential-vending.md) for more

#### S3 configuration

| Configuration item | Description | Default value | Required | Since Version |
|----------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|------------------------------------------------|------------------|
| `gravitino.iceberg-rest.io-impl` | The IO implementation for `FileIO` in Iceberg, use `org.apache.iceberg.aws.s3.S3FileIO` for S3. | (none) | No | 0.6.0-incubating |
| `gravitino.iceberg-rest.s3-endpoint` | An alternative endpoint of the S3 service, This could be used for S3FileIO with any s3-compatible object storage service that has a different endpoint, or access a private S3 endpoint in a virtual private cloud. | (none) | No | 0.6.0-incubating |
| `gravitino.iceberg-rest.s3-region` | The region of the S3 service, like `us-west-2`. | (none) | No | 0.6.0-incubating |
| Configuration item | Description | Default value | Required | Since Version |
|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|------------------------------------------------|------------------|
| `gravitino.iceberg-rest.io-impl` | The IO implementation for `FileIO` in Iceberg, use `org.apache.iceberg.aws.s3.S3FileIO` for S3. | (none) | No | 0.6.0-incubating |
| `gravitino.iceberg-rest.s3-endpoint` | An alternative endpoint of the S3 service, This could be used for S3FileIO with any s3-compatible object storage service that has a different endpoint, or access a private S3 endpoint in a virtual private cloud. | (none) | No | 0.6.0-incubating |
| `gravitino.iceberg-rest.s3-region` | The region of the S3 service, like `us-west-2`. | (none) | No | 0.6.0-incubating |
| `gravitino.iceberg-rest.s3-path-style-access` | Whether to use path style access for S3. | false | No | 0.9.0-incubating |

For other Iceberg s3 properties not managed by Gravitino like `s3.sse.type`, you could config it directly by `gravitino.iceberg-rest.s3.sse.type`.

Expand Down
2 changes: 2 additions & 0 deletions docs/lakehouse-iceberg-catalog.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,8 @@ Supports using static access-key-id and secret-access-key to access S3 data.
| `s3-secret-access-key` | The static secret access key used to access S3 data. | (none) | No | 0.6.0-incubating |
| `s3-endpoint` | An alternative endpoint of the S3 service, This could be used for S3FileIO with any s3-compatible object storage service that has a different endpoint, or access a private S3 endpoint in a virtual private cloud. | (none) | No | 0.6.0-incubating |
| `s3-region` | The region of the S3 service, like `us-west-2`. | (none) | No | 0.6.0-incubating |
| `s3-path-style-access` | Whether to use path style access for S3. | false | No | 0.9.0-incubating |


For other Iceberg s3 properties not managed by Gravitino like `s3.sse.type`, you could config it directly by `gravitino.bypass.s3.sse.type`.

Expand Down
1 change: 1 addition & 0 deletions docs/spark-connector/spark-catalog-iceberg.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ Gravitino spark connector will transform below property names which are defined
| `s3-region` | `client.region` | The region of the S3 service, like `us-west-2`. | 0.6.0-incubating |
| `s3-access-key-id` | `s3.access-key-id` | The static access key ID used to access S3 data. | 0.8.0-incubating |
| `s3-secret-access-key` | `s3.secret-access-key` | The static secret access key used to access S3 data. | 0.8.0-incubating |
| `s3-path-style-access` | `s3.path-style-access` | Whether to use path style access for S3. | 0.9.0-incubating |
| `oss-endpoint` | `oss.endpoint` | The endpoint of Aliyun OSS service. | 0.7.0-incubating |
| `oss-access-key-id` | `client.access-key-id` | The static access key ID used to access OSS data. | 0.8.0-incubating |
| `oss-secret-access-key` | `client.access-key-secret` | The static secret access key used to access OSS data. | 0.8.0-incubating |
Expand Down
7 changes: 7 additions & 0 deletions iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/IcebergConfig.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,13 @@ public class IcebergConfig extends Config implements OverwriteDefaultConfig {
.stringConf()
.create();

public static final ConfigEntry<Boolean> S3_PATH_STYLE_ACCESS =
new ConfigBuilder(S3Properties.GRAVITINO_S3_PATH_STYLE_ACCESS)
.doc("Whether to use path style access for S3")
.version(ConfigConstants.VERSION_0_9_0)
.booleanConf()
.createWithDefault(false);

public static final ConfigEntry<String> OSS_ENDPOINT =
new ConfigBuilder(OSSProperties.GRAVITINO_OSS_ENDPOINT)
.doc("The endpoint of Aliyun OSS service")
Expand Down

0 comments on commit 6da2f79

Please sign in to comment.