docs: add production configuration pages for AWS S3 and Azure Blob storage (#1325) by mj006648 · Pull Request #4451 · apache/polaris

mj006648 · 2026-05-15T06:07:26Z

Summary

Adds production configuration pages for AWS S3 and Azure Blob storage backends, mirroring the existing GCS page. Implements Option 1 of #1325 as agreed with @flyrain. GCS page enhancement and skill.md are intentionally left for follow-up PRs.

Two new pages under site/content/in-dev/unreleased/configuration/configuring-polaris-for-production/:

configuring-aws-s3-cloud-storage-specific.md
configuring-azure-blob-cloud-storage-specific.md

Each covers server-side storageConfigInfo fields, client wiring for Spark / Trino / PyIceberg, and a verification SQL snippet.

Verified end-to-end while writing this

MinIO + Polaris + Spark / Trino / PyIceberg 0.11.1: CREATE / INSERT / SELECT round-trips. Findings folded into the page (the required vended-credentials-enabled and native-S3 properties for Trino, and the header.X-Iceberg-Access-Delegation REST header for PyIceberg).
Ceph RGW with STS disabled: produced Failed to get subscoped credentials STS 400 — informed the new "Backends without STS support" subsection.
ADLS Gen2 (HNS) with a service principal: happy-path round-trip.
ADLS Gen2 with the SP's Storage Blob Data Contributor removed: reproduced AuthorizationPermissionMismatch 403 — informed the Azure troubleshooting bullet.

AWS IAM trust-policy / KMS specifics and HNS directory-ACL behavior are described conservatively based on docs and the corresponding Polaris classes rather than from a live test against those exact features.

Test plan

Hugo site build succeeds locally (186 pages, no errors)
Both new pages appear in the production-config sidebar
CI Hugo build passes
Reviewers confirm the conservative bullets (AWS IAM/KMS, Azure HNS ACL) match operational reality

…orage Adds two new pages under configuring-polaris-for-production that mirror the existing GCS page and walk a system administrator through end-to-end storage backend setup: - configuring-aws-s3-cloud-storage-specific.md: IAM role, trust policy (userArn/externalId), KMS settings, S3-compatible endpoint options (endpoint/pathStyleAccess/stsUnavailable), and Spark/Trino/PyIceberg client wiring. - configuring-azure-blob-cloud-storage-specific.md: multi-tenant app consent flow (tenantId/multiTenantAppName/consentUrl), HNS-aware SAS token scoping, and ADLS Gen2 client wiring. Fields used in the examples match AwsStorageConfigurationInfo and AzureStorageConfigurationInfo. Refs: apache#1325 Signed-off-by: mj006648 <uckdekf@gmail.com>

Add oauth2-server-uri and token-refresh-enabled=false to the Spark spark-sql examples on the AWS S3 and Azure Blob storage pages. These match the property set used by the existing MinIO / RustFS guides, which is the configuration verified end-to-end against a local polaris+minio docker stack while preparing these docs. The oauth2-server-uri is also called out in prose because the Iceberg REST client logs a deprecation warning when the fallback path is used. Refs: apache#1325 Signed-off-by: mj006648 <uckdekf@gmail.com>

Replace the multi-tenant 'consent URL' framing with the service-principal flow that self-hosted Polaris deployments actually use. Polaris loads credentials via DefaultAzureCredential (AzureCredentialsStorageIntegration), so the docs now show 'az ad sp create-for-rbac' with Storage Blob Data Contributor and the three AZURE_* environment variables on the server. multiTenantAppName / consentUrl are still mentioned but described as optional fields for managed deployments, not the primary path. Verified end-to-end against ADLS Gen2 (HNS enabled) with a service principal: catalog creation, CREATE NAMESPACE / TABLE, INSERT, SELECT all succeed through the vended SAS-token path. Refs: apache#1325 Signed-off-by: mj006648 <uckdekf@gmail.com>

…nd PyIceberg S3 page changes informed by additional end-to-end checks: - 'S3-compatible endpoints' is now split into two cases. With STS support (AWS S3, MinIO) Polaris vends short-lived credentials as before. Without STS support (Apache Ozone S3 gateway, Ceph RGW with STS disabled) the client must omit the X-Iceberg-Access-Delegation header and supply its own long-lived credentials. A test against a Ceph RGW deployment without STS produced 'Failed to get subscoped credentials' (STS 400), which is the symptom the new section now warns about. - The Trino example now includes 'iceberg.rest-catalog.vended-credentials-enabled', 'fs.native-s3.enabled', and 's3.region'/'s3.endpoint'/'s3.path-style-access'. Verified end-to-end against the MinIO compose with Trino: SHOW SCHEMAS, CREATE SCHEMA, CREATE TABLE, INSERT, SELECT all succeed. - A real PyIceberg snippet is added showing the required options, including the 'header.X-Iceberg-Access-Delegation' REST header. Verified against the same MinIO compose with pyiceberg 0.11.1. Azure page changes: - A real PyIceberg snippet is added mirroring the S3 example. - The troubleshooting bullet is split: the 'Failed to get subscoped credentials / AuthorizationPermissionMismatch' pattern now points at the data-plane RBAC role assignment, which was reproduced by removing Storage Blob Data Contributor from the service principal and observing the exact error message during INSERT. The pre-existing HNS-ACL bullet is kept for the narrower ACL-mismatch case. Refs: apache#1325 Signed-off-by: mj006648 <uckdekf@gmail.com>

dimas-b

Thanks for your contribution, @mj006648 ! Very useful docs... I hope you do not mind my many comments 😅

S3 page: - retitle to 'Configuring S3 Storage' since the page also covers MinIO, Ozone, Ceph and similar S3-compatible backends. - drop userArn from the storageConfigInfo example and the surrounding text; the field is not used by AwsStorageConfigurationInfo in current code. - recommend externalId as the default and remove the inline 'if you skip the external ID' guidance, since omitting it is not best practice and knowledgeable users can omit it on their own. - add a one-line note that the bearer token in the catalog-create curl is the Polaris admin OAuth2 token from /v1/oauth/tokens, with a relref to the parent production-config page that explains bootstrapping. - clarify the KMS section: currentKmsKey and allowedKmsKeys are processed independently in AwsCredentialsStorageIntegration, so the write key has to be included in allowedKmsKeys explicitly. IAM action wording fixed accordingly (kms:GenerateDataKey only on currentKmsKey). - reconcile the STS subsection with the page intro: STS-unsupported backends turn off vended credentials entirely, and the client must drop the X-Iceberg-Access-Delegation header rather than fall back to long-lived credentials inside the vending flow. - add a short note that this page covers native Polaris authentication; external identity providers are supported but documented elsewhere. - soften the client-side s3.* block: Polaris returns endpoint / path-style-access / region in the catalog config response, so the block is only needed where Trino requires it to be explicit (e.g. S3-compatible endpoints). Azure page: - recommend storing AZURE_CLIENT_SECRET in a Kubernetes Secret rather than as a plain env in the pod spec, and apply the same to POLARIS_BOOTSTRAP_CREDENTIALS. - rewrite the HNS bullet: HNS is not required by Polaris or Iceberg for table operations themselves; its main effect is that Polaris can downscope vended SAS tokens to the directory level rather than the container level. Atomic-commit framing removed. - add a bullet that the hierarchical field in storageConfigInfo must match the actual HNS state of the storage account, since a mismatch causes runtime access errors. - mark multiTenantAppName and consentUrl as informational only, since current Apache Polaris code does not use them when communicating with Azure APIs. Refs: apache#1325 Signed-off-by: mj006648 <uckdekf@gmail.com>

mj006648 · 2026-05-16T07:26:25Z

Thanks for the thorough review, @dimas-b! Pushed f5eae4e addressing your comments:

S3 page

L62: removed the "if you skip external ID" guidance and recommended externalId as default.
L67/79: dropped userArn from both the JSON example and the surrounding prose since AwsStorageConfigurationInfo does not use it.
L71: added a one-line note about the bearer token being the Polaris admin OAuth2 token, with a relref to the parent production-config page.
L102 (KMS): clarified that currentKmsKey and allowedKmsKeys are processed independently in AwsCredentialsStorageIntegration, so the write key has to be repeated in allowedKmsKeys. Fixed the IAM action wording accordingly.
L120: retitled the page to "Configuring S3 Storage".
L130: reconciled the STS subsection — STS-unsupported backends turn off vended credentials entirely and the client must drop the X-Iceberg-Access-Delegation header, instead of "falling back to long-lived credentials inside the vending flow".
L187: added a short paragraph noting the page covers native Polaris auth and external IdPs are supported but documented elsewhere.
L217: softened the client-side s3.* block — Polaris returns endpoint/path-style-access/region in the catalog config response, and the explicit block is now framed as "only needed where Trino requires it to be explicit (e.g. S3-compatible endpoints)". Please double-check that wording matches what you had in mind; if Polaris's response is enough on its own for the AWS S3 case, I am happy to drop the explicit block from the AWS path entirely.

Azure page

L53: recommended Kubernetes Secret refs rather than plain env for AZURE_CLIENT_SECRET (and POLARIS_BOOTSTRAP_CREDENTIALS).
L63/64 (HNS): rewrote the bullet — HNS is not required by Polaris/Iceberg for table operations themselves; the main effect is SAS-token downscoping (directory vs. container). Atomic-commit framing removed.
L102 (multiTenantAppName / consentUrl): marked as informational only since current Apache Polaris code does not use them when communicating with Azure APIs.
L118: added a bullet that the hierarchical field must match the actual HNS state, otherwise runtime access errors occur.

Re: L217 (Trino S3 block) — would be glad to drop more of it if Polaris's catalog config response is sufficient end-to-end on the Trino side. I have only verified the Trino path against MinIO with the explicit block in place; I have not tested removing it.

dimas-b · 2026-05-20T19:55:30Z

+
+```properties
+s3.endpoint=https://s3.internal.example.com
+s3.path-style-access=true


@mj006648 : Do you think this is still necessary?

dimas-b · 2026-05-20T19:58:28Z

+
+If `INSERT` or `SELECT` fails with a 403, the most common causes are:
+
+- The IAM role's trust policy does not match the `userArn` / `externalId` Polaris is presenting.


Did you mean roleArn here?

- Drop the `s3.region` line and the separate `s3.endpoint` / `s3.path-style-access` block from the Trino example. Polaris already vends those keys in the LoadTable response config via StorageAccessProperty (s3.endpoint, s3.path-style-access, client.region), so they are redundant on the client. Keep `fs.native-s3.enabled=true` since it activates Trino's native S3 filesystem and is not something Polaris can vend. - Rewrite the surrounding paragraph to describe the split between what Polaris vends and what the client still has to set. - Replace the residual `userArn` reference in the 403 troubleshooting bullet with `roleArn`, matching what AwsStorageConfigurationInfo exposes to AssumeRoleRequest. Confidence: medium Scope-risk: narrow Signed-off-by: mj006648 <uckdekf@gmail.com>

mj006648 · 2026-05-21T02:50:53Z

Thanks for the second pass, @dimas-b. Pushed a follow-up.

For the Trino block (L220): I traced what Polaris actually vends in the LoadTable response. From polaris-core/src/main/java/org/apache/polaris/core/storage/StorageAccessProperty.java (lines 41-44) the keys exposed to the client are s3.endpoint, s3.path-style-access, and client.region, populated by AwsCredentialsStorageIntegration (lines 172-193) from endpoint / pathStyleAccess / region on the storage config.

So I dropped s3.region and the separate s3.endpoint / s3.path-style-access block — those really are redundant on the client. I kept fs.native-s3.enabled=true since that flag turns on Trino's native S3 filesystem and isn't something Polaris can vend. Rewrote the surrounding paragraph to reflect that split.

L261: replaced userArn with roleArn (leftover from the first pass — AwsStorageConfigurationInfo.getRoleARN() is what gets threaded into the AssumeRoleRequest).

Diff is +5/-13, docs-only.

dimas-b · 2026-05-21T15:34:52Z

@mj006648 : line references to review-based changes are nice and helpful, but replying to GH comments in-thread is even nicer for humans 😉

dimas-b

Thanks again for your contribution, @mj006648 !

I'll leave the PR open for a couple more days in case other reviewers want to comment.

flyrain · 2026-05-26T05:50:27Z

+This page is limited to native Polaris authentication. External identity providers are also
+supported but are not yet covered here; the configuration patterns below remain otherwise the same.
+
+## IAM role and trust policy


Thanks for adding it! I think there are three different IAM roles involved:

The IAM role used for starting a new Polaris instance

The IAM role used for creating a new catalog

The IAM role used for vended credentials.

This section describes the third one. Would it be possible to describe the other two?

Thanks @flyrain. Took a shot at this in 0837f0a — added an
overview section naming the three identities and a "Polaris service
identity" section covering IRSA / EC2 / static creds plus the
sts:AssumeRole policy. Also renamed the old section to "Catalog
access role and trust policy" so the labels line up.

Quick check: I read "creating a new catalog" as the IAM role that
gets bound to the catalog at creation time. If you meant the admin
permissions to call the catalogs API, let me know and I'll handle
it separately.

…dentities

mj006648 · 2026-05-26T09:43:13Z

@dimas-b apologies for the new churn — pushed 0837f0a to address @flyrain's review. Could you take another look when convenient?

flyrain

LGTM. Thanks @mj006648 !

Verification environment used while writing the AWS S3 production configuration page (apache/polaris#4451). Credentials are MinIO-local dummy values, not real secrets. Confidence: high Scope-risk: narrow Not-tested: ./trino-catalog volume is referenced but not committed yet

…orage (apache#1325) (apache#4451)

mj006648 added 4 commits May 15, 2026 02:42

github-project-automation Bot added this to Basic Kanban Board May 15, 2026

github-project-automation Bot moved this to PRs In Progress in Basic Kanban Board May 15, 2026

dimas-b reviewed May 15, 2026

View reviewed changes

dimas-b reviewed May 20, 2026

View reviewed changes

dimas-b previously approved these changes May 21, 2026

View reviewed changes

github-project-automation Bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board May 21, 2026

flyrain reviewed May 26, 2026

View reviewed changes

docs(s3): describe Polaris service identity and label the three IAM i…

0837f0a

…dentities

mj006648 dismissed dimas-b’s stale review via 0837f0a May 26, 2026 09:32

dimas-b approved these changes May 26, 2026

View reviewed changes

dimas-b requested a review from flyrain May 26, 2026 23:41

flyrain approved these changes May 27, 2026

View reviewed changes

flyrain merged commit 6566296 into apache:main May 27, 2026
24 checks passed

github-project-automation Bot moved this from Ready to merge to Done in Basic Kanban Board May 27, 2026

mj006648 mentioned this pull request May 27, 2026

Discrepency in the fetched table properties apache/iceberg-python#3247

Open

3 tasks

MonkeyCanCode pushed a commit to MonkeyCanCode/polaris that referenced this pull request Jun 4, 2026

docs: add production configuration pages for AWS S3 and Azure Blob st…

1a5c63d

…orage (apache#1325) (apache#4451)

dimas-b mentioned this pull request Aug 12, 2026

CLI: Setup Azure missing parameter during setup/export #5220

Merged

6 tasks

This was referenced Sep 1, 2026

docs: expand GCS production configuration for sysadmins (#1325) #5432

Open

Document how Polaris works with different cloud storage(S3, Azure Blob Storage, GCS). #1325

Open


		If `INSERT` or `SELECT` fails with a 403, the most common causes are:

		- The IAM role's trust policy does not match the `userArn` / `externalId` Polaris is presenting.

Uh oh!

Conversation

mj006648 commented May 15, 2026

Summary

Verified end-to-end while writing this

Test plan

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mj006648 commented May 16, 2026

Uh oh!

dimas-b May 20, 2026

Choose a reason for hiding this comment

Uh oh!

dimas-b May 20, 2026

Choose a reason for hiding this comment

Uh oh!

mj006648 commented May 21, 2026

Uh oh!

dimas-b commented May 21, 2026

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

flyrain May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mj006648 May 26, 2026

Choose a reason for hiding this comment

Uh oh!

mj006648 commented May 26, 2026

Uh oh!

flyrain left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

flyrain May 26, 2026 •

edited

Loading