fix: prevent duplicate AWS SDK S3 classloading when s3 and iceberg extensions are both loaded

[akashchamp]

Fixes #19636.

Description

When druid-s3-extensions and druid-iceberg-extensions are both loaded, an Iceberg batch ingestion task that reads S3/Glue can fail before it starts running subtasks, while Iceberg constructs its AWS S3 client, with:

java.lang.ExceptionInInitializerError
Caused by: java.lang.IllegalArgumentException: No duplicate IdentityProperty names allowed but both IdentityProperties 5fb0a09e and 536fc38 have the same namespace (java.lang.String) and name (Bucket). IdentityProperty should be referenced from a shared static constant to protect against erroneous or unexpected collisions.

Root cause. Both extensions declare their own direct dependency on software.amazon.awssdk:s3, so pull-deps bundles a private copy of the s3 jar into each extension's own directory, and each extension is loaded through its own StandardURLClassLoader (druid.extensions.useExtensionClassloaderFirst defaults to false, i.e. parent-first delegation). software.amazon.awssdk.services.s3.internal.s3express.S3ExpressPlugin (inside the s3 jar) registers a static IdentityProperty constant into a registry that lives in identity-spi. identity-spi is already loaded once, from the shared parent classpath (/opt/druid/lib), because druid-server has a runtime dependency on druid-aws-common, which depends on identity-spi transitively. But s3 itself is not on that shared classpath, so each extension classloader loads its own copy of S3ExpressPlugin, and each copy's static IdentityProperty field is a distinct object with the same name/namespace — which the shared, singleton registry rejects as a "duplicate".

Fix. This follows the same pattern already used for sts (see the "WebIdentityTokenProvider requires runtime dependency on sts" comment in cloud/aws-common/pom.xml, and sts being provided in both extensions):

• Add software.amazon.awssdk:s3 as a runtime dependency of druid-aws-common, so it is resolved once, from the shared classpath (druid-server → druid-aws-common → s3), instead of being extension-scoped.
• Mark software.amazon.awssdk:s3 as provided in both extensions-core/s3-extensions/pom.xml and extensions-contrib/druid-iceberg-extensions/pom.xml, so neither extension bundles its own copy any more (it's still on the compile/test classpath, just not packaged).

No licenses.yaml change is needed: software.amazon.awssdk:s3 (like sts, identity-spi, sdk-core, etc.) is already covered by the existing "AWS SDK for Java 2" entry, which is module: java-core and lists the whole SDK, independent of which module physically bundles which artifact.

Verification

• mvn dependency:tree -Dscope=runtime on cloud/aws-common now shows software.amazon.awssdk:s3:jar:2.55.1:runtime.
• mvn dependency:tree -Dscope=runtime on extensions-core/s3-extensions and extensions-contrib/druid-iceberg-extensions no longer lists software.amazon.awssdk:s3 anywhere in either module's own subtree (confirmed by grepping the tree output for awssdk:s3: and checking the matches are outside those modules' sections).
• mvn test-compile for cloud/aws-common, extensions-core/s3-extensions, and extensions-contrib/druid-iceberg-extensions (plus their -am prerequisites) succeeds.
• Full existing unit test suites pass unchanged:
  • druid-s3-extensions: 247 tests, 0 failures, 0 errors.
  • druid-iceberg-extensions: 61 tests, 0 failures, 0 errors (including GlueIcebergCatalogTest).
• Manual reproduction of the actual crash mechanism, outside the repo, using the real software.amazon.awssdk:s3 and identity-spi jars resolved by this build (2.55.1) and two URLClassLoaders configured exactly like Druid's default StandardURLClassLoader (parent-first delegation, one parent "shared classpath" loader plus two child "extension" loaders): with a private copy of s3-2.55.1.jar given to each child loader (today's pre-fix layout), building an S3Client through the second loader reproduces the exact reported exception, verbatim:

  java.lang.IllegalArgumentException: No duplicate IdentityProperty names allowed but both IdentityProperties 5a021cb9 and 2e52fb3e have the same namespace (java.lang.String) and name (Bucket). IdentityProperty should be referenced from a shared static constant to protect against erroneous or unexpected collisions.
  

  With s3-2.55.1.jar moved to the shared/parent loader instead (this fix's layout — one copy, both extensions delegate to it), building an S3Client through both loaders succeeds with no exception. This isn't a Druid integration test I could reasonably check in (it stands up its own throwaway classloaders against raw SDK jars rather than a running Druid process), but it directly confirms the classloading mechanism described above and that this change removes it.

I did not add real S3/Glue infrastructure to run an actual end-to-end Iceberg ingestion task; the above reproduces the exact failing code path (S3Client.builder().build() under duplicated extension classloaders) that the issue's stack trace points to.

Release note

Fixed: Iceberg batch ingestion tasks no longer fail with IllegalArgumentException: No duplicate IdentityProperty names allowed ... name (Bucket) when druid-s3-extensions and druid-iceberg-extensions are loaded together.

---

Key changed/added classes in this PR

• None (build/packaging change only: cloud/aws-common/pom.xml, extensions-core/s3-extensions/pom.xml, extensions-contrib/druid-iceberg-extensions/pom.xml)

---

This PR has:

• been self-reviewed.
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml (not needed — see Description above)
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

---

This PR was prepared by an AI coding agent (Claude, via Claude Code) under human supervision, from investigation through implementation and verification, as disclosed above; the reproduction methodology and verification results are genuine and were actually executed, not simulated.

[FrankChen021]

🟢 Approval recommended

The merge-base diff and relevant surrounding contracts were reviewed for all three changed Maven files. Promoting software.amazon.awssdk:s3 to the shared runtime through druid-aws-common is consistent with druid-server's runtime dependency and Druid's parent-first extension classloader; marking the direct S3 dependencies as provided prevents the two extension packages from contributing private copies while retaining compile and test visibility. I also inspected the pull-deps runtime dependency filter, distribution runtime dependency set, task classpath defaults, all direct S3 declarations in the repository, and the existing AWS SDK license entry. No actionable correctness, compatibility, lifecycle, security, or missing-test issue was found in this static review.

Reviewed 3 of 3 changed files.

---

This is an automated review by Codex GPT-5.6-Luna(max)

[FrankChen021]

Follow-up assessment

The author addressed the prior review feedback by shortening the shared-classpath comments in all three changed POM files. I reviewed the latest author reply, the current PR head, the incremental and full diffs, and the surrounding Maven dependency/classpath handling; no further inline response or actionable finding is needed.

Reviewed 3 of 3 changed files; 0 findings.

---

This is an automated review by Codex GPT-5.6-Luna(max)

[FrankChen021]

Thanks for the change

[FrankChen021]

🟢 Approval recommended

The current head preserves the intended classpath fix: software.amazon.awssdk:s3 is supplied once through the shared runtime dependency path, while both extension-local declarations remain compile/test-visible but are excluded from extension packaging. I rechecked the empty incremental diff, the full merge-base diff, the three changed POM files, the server-to-aws-common runtime edge, pull-deps' runtime/provided filtering, the parent-first and extension-first loader behavior, all repository S3 consumers/declarations, and the existing AWS SDK v2 license entry. No actionable correctness, compatibility, classloading, packaging, security, or missing-test issue was found in this static review.

Reviewed 3 of 3 changed files.

---

This is an automated review by Codex GPT-5.6-Luna(max)