Native scans resolve the AWS SDK ProfileCredentialsProvider names differently from Hadoop

[dwsmith1983]

Describe the bug

When fs.s3a.aws.credentials.provider names an AWS SDK profile provider, software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider or com.amazonaws.auth.profile.ProfileCredentialsProvider, native scans resolve the profile with aws-config's ProfileFileCredentialsProvider. With Hadoop 3.4 and later (Spark 4.x), Hadoop maps both names to the Java SDK v2 ProfileCredentialsProvider, and the two SDKs resolve profiles differently. Some of these differences change which identity signs requests or which STS endpoint is called:

• a source profile with both static keys and credential_process: Java uses credential_process, aws-config the keys;
• assume-role STS region: Java uses each role's own region, else the default region chain, else the global sts.amazonaws.com; the native side uses one region, from aws-config's default region chain, for every role in the chain;
• a role whose source_profile names itself: Java reports a cycle, aws-config assumes the role with the profile's own keys;
• SSO keys together with role_arn: Java uses SSO, aws-config assumes the role.

This is the current behavior on main and branch-1.1. Hadoop's own org.apache.hadoop.fs.s3a.auth.ProfileAWSCredentialsProvider is handled in #5872 and is not part of this issue.

Steps to reproduce

On Spark 4.x, set fs.s3a.aws.credentials.provider=software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider and select an assume-role profile whose source profile has both aws_access_key_id and credential_process. A native read signs AssumeRole with the static key, while Spark with Comet disabled signs with the process credentials.

Expected behavior

Native scans pick the same credentials and STS endpoints as Hadoop for the SDK profile provider names. One option is to route them through HadoopS3ACredentialProviderAdapter, which builds Hadoop's own provider list on the executor. That would add the adapter's classpath requirement and a JNI call per request for static keys to configurations that work today.

Additional context

Found while reviewing #5872.

[dwsmith1983]

Proposal once #5872 lands: route the two SDK names through HadoopS3ACredentialProviderAdapter the same way #5872 routes Hadoop's ProfileAWSCredentialsProvider. Hadoop maps both names to the Java SDK's ProfileCredentialsProvider, so they would resolve exactly as in Spark, including the cases above, and on Spark 3.x through the adapter's SDK v1 build. The change itself is small: the provider class lookup, its table test, and the docs.

The trade-off is that these names work today, unlike the profile provider #5872 handles, so routing them adds the adapter's requirements to configurations that run now:

• hadoop-aws and the AWS SDK must be visible to Comet's class loader; with Comet on extraClassPath and hadoop-aws only through --packages, the scan would fail with NoClassDefFoundError where it works today;
• a profile with static keys asks the provider through JNI on each request.

The cases that differ are uncommon profile shapes (keys plus credential_process in one source profile, multi-role chains with different regions, a role naming itself as source_profile, SSO keys with role_arn); a plain profile or a single role mostly works today. I would not fall back to the native path when the adapter cannot load, since the signing identity would then depend on the classpath.

@sunchao @andygrove does the classpath requirement for existing configurations look acceptable? If so I'll open the PR after #5872 merges.