docs: expand GCS production configuration for sysadmins (#1325) - #5432
docs: expand GCS production configuration for sysadmins (#1325)#5432AzazelSensei wants to merge 2 commits into
Conversation
Bring the GCS page in line with the S3 and Azure production pages: server identity, optional gcpServiceAccount impersonation, catalog storageConfigInfo, client wiring, and a verification snippet.
| how to bootstrap and issue admin tokens). | ||
|
|
||
| ```bash | ||
| curl -X POST https://<polaris-host>/management/v1/catalogs \ |
There was a problem hiding this comment.
This copy-paste flow cannot create the catalog because the Management API is served under /api/management/v1, while this URL omits /api. Could we change it to https:///api/management/v1/catalogs and verify the command against the Management OpenAPI? Otherwise the setup stops before any GCS configuration is exercised.
There was a problem hiding this comment.
Updated the curl to https:///api/management/v1/catalogs. That matches the Management OpenAPI server URL plus /catalogs.
| Polaris requires both IAM roles and | ||
| [Hierarchical Namespace (HNS)](https://docs.cloud.google.com/storage/docs/hns-overview) ACLs (if |
There was a problem hiding this comment.
This section still describes HNS access as a separate ACL layer, but , IIUC, HNS requires uniform bucket-level access and does not support object-level ACLs. Prefix-scoped access has to come from IAM, for example through an associated managed folder or an IAM condition, while the Credential Access Boundary can only narrow permissions the source identity already has. Could we replace this with one supported IAM setup and keep source authorization separate from token downscoping? Otherwise the troubleshooting guidance sends operators to a mechanism HNS does not provide.
There was a problem hiding this comment.
Replaced the HNS ACL section. HNS buckets use uniform bucket-level access, so prefix access has to come from IAM (managed folder or a condition). The Credential Access Boundary only narrows what the source identity already has.
| `GcpStorageConfigurationInfo` also records `gcpServiceAccount`; omit it only when the Polaris | ||
| process identity should be downscoped directly. | ||
|
|
||
| Iceberg I/O for GCS catalogs is `org.apache.iceberg.gcp.gcs.GCSFileIO`. Polaris returns that |
There was a problem hiding this comment.
I believe Polaris does not return the GCSFileIO implementation name with vended credentials. GcpStorageConfigurationInfo selects GCSFileIO for Polaris's own server-side FileIO, while GcpCredentialsStorageIntegration returns the token, expiry, and optional refresh endpoint to the client. Could we separate those responsibilities here and state that each engine must select its own GCS filesystem or FileIO? The current wording makes the client setup look automatic when it is not.
There was a problem hiding this comment.
Split that. GcpStorageConfigurationInfo still picks GCSFileIO for Polaris server-side FileIO. Vended credentials are only the token, expiry, and optional refresh endpoint. Each engine has to select its own GCS filesystem or FileIO.
| iceberg.rest-catalog.oauth2.credential=<client-id>:<client-secret> | ||
| iceberg.rest-catalog.oauth2.scope=PRINCIPAL_ROLE:ALL | ||
| iceberg.rest-catalog.oauth2.server-uri=https://<polaris-host>/api/catalog/v1/oauth/tokens | ||
| iceberg.rest-catalog.vended-credentials-enabled=true |
There was a problem hiding this comment.
I think this Trino example cannot initialize with vended credentials as written. Trino requires a native cloud filesystem when vending is enabled, and fs.gcs.enabled defaults to false, so validation fails before the connector can call Polaris. Could we add fs.gcs.enabled=true and verify this exact catalog file against the supported Trino version?
There was a problem hiding this comment.
Added fs.gcs.enabled=true. Trino 483 (the getting-started image) rejects vended credentials unless a native cloud filesystem is on, and that property defaults to false.
| to GCS directly with `gcs.oauth2.token`. | ||
|
|
||
| Identity 1 is configured once at Polaris deployment time. Identity 2 is created per catalog and | ||
| its email is registered when the catalog is created. Identity 3 is generated on every table load |
There was a problem hiding this comment.
The “generated on every table load” part is not accurate for the production path. CachingStorageIntegration routes matching vending requests through StorageCredentialCache, so Polaris mints on a cache miss and reuses the in-memory credential until cache or credential expiry. Could we describe that lifecycle explicitly, including that credentials are cached in process but not durably persisted? This affects operator expectations around issuance frequency and revocation timing.
There was a problem hiding this comment.
Updated that paragraph. CachingStorageIntegration mints on a cache miss and reuses the in-memory token until cache or credential expiry. Nothing is persisted, so a restart issues new tokens.
Correct the management API path, describe in-process credential caching, replace HNS ACL guidance with IAM/managed-folder prefix access, separate server-side GCSFileIO from vended tokens, and enable Trino fs.gcs.
2 participants
The GCS production page was still a short HNS note after #4451 landed the S3 and Azure pages. This fills the remaining gap from #1325 so a sysadmin can take a Polaris server from ADC through catalog
storageConfigInfoto a Spark / Trino / PyIceberg client.Fields and vending behavior match
GcpStorageConfigurationInfoandGcpCredentialsStorageIntegration(ADC + optionalgcpServiceAccountimpersonation, downscopedgcs.oauth2.token). I did not run this against a live GCS bucket.Checklist
CHANGELOG.md(if needed)site/content/in-dev/unreleased(if needed)