Describe the bug
The native Parquet scan resolves Azure credentials from the Hadoop fs.azure.* settings, and the goal is to pick the identity Hadoop's ABFS driver would. After #6059 a few gaps remain where the two can disagree. None of them comes from #6059; they were found while checking it against Hadoop 3.4.2's AbfsConfiguration.
- Container-scoped OAuth keys. Since Hadoop 3.4.2,
getPasswordString probes <key>.<container>.<host> before the account and global forms, and getTokenProvider reads the client id, secret and endpoint, the MSI tenant, endpoint and authority, and the token file through it. The native scan reads that form only for the SAS fixed token. With fs.azure.account.oauth2.client.secret.data.myacct.dfs.core.windows.net set next to an account-level secret, Hadoop authenticates as the container's principal and the native scan as the account's. With only the container-scoped secret, the native scan fails.
- Fabric hosts.
ENDPOINT_SUFFIXES lists only the core.windows.net suffixes, but object_store also accepts onelake.dfs.fabric.microsoft.com. Account-scoped keys for a Fabric host are never found, so a configured principal is missed and the scan falls through to ambient AZURE_* variables or managed identity.
- Account-scoped key forms. Hadoop's
accountConf builds <key>.<host> from the raw URL authority, port included. The native scan also probes the other endpoint suffix and the short <key>.<account> form, ahead of the global key, and drops an explicit port. A stale key Hadoop never reads can therefore win over the global one.
- Credential provider stores.
Configuration.getPassword consults hadoop.security.credential.provider.path (for example JCEKS) before plain configuration. The native scan only sees plain configuration, so a token kept in a store is invisible and a lower-priority plaintext value is used instead.
- OAuth provider class lookup.
fs.azure.account.oauth.provider.type does not apply the getTokenProviderClass auth-type guard that the SAS provider class now does. Hadoop fails ("Failed to initialize null") where the native scan builds a store, so this only matters if the driver path changes.
- Error text. The "Scheme of URL is not Azure" error formats the whole URL, which would echo a password in the userinfo.
Steps to reproduce
Unit-level: build the store with create_store_with_env and an empty environment for each configuration above, and compare with what Hadoop 3.4.2 AbfsConfiguration resolves.
Expected behavior
The native scan either resolves the same credential Hadoop does or fails with an error naming the key, and never falls through to the environment when Hadoop configured a mechanism.
Additional context
Hadoop 3.4.1 and older have no container-scoped lookup at all, so items 1 and 3 depend on the Hadoop version on the driver.
Describe the bug
The native Parquet scan resolves Azure credentials from the Hadoop
fs.azure.*settings, and the goal is to pick the identity Hadoop's ABFS driver would. After #6059 a few gaps remain where the two can disagree. None of them comes from #6059; they were found while checking it against Hadoop 3.4.2'sAbfsConfiguration.getPasswordStringprobes<key>.<container>.<host>before the account and global forms, andgetTokenProviderreads the client id, secret and endpoint, the MSI tenant, endpoint and authority, and the token file through it. The native scan reads that form only for the SAS fixed token. Withfs.azure.account.oauth2.client.secret.data.myacct.dfs.core.windows.netset next to an account-level secret, Hadoop authenticates as the container's principal and the native scan as the account's. With only the container-scoped secret, the native scan fails.ENDPOINT_SUFFIXESlists only thecore.windows.netsuffixes, butobject_storealso acceptsonelake.dfs.fabric.microsoft.com. Account-scoped keys for a Fabric host are never found, so a configured principal is missed and the scan falls through to ambientAZURE_*variables or managed identity.accountConfbuilds<key>.<host>from the raw URL authority, port included. The native scan also probes the other endpoint suffix and the short<key>.<account>form, ahead of the global key, and drops an explicit port. A stale key Hadoop never reads can therefore win over the global one.Configuration.getPasswordconsultshadoop.security.credential.provider.path(for example JCEKS) before plain configuration. The native scan only sees plain configuration, so a token kept in a store is invisible and a lower-priority plaintext value is used instead.fs.azure.account.oauth.provider.typedoes not apply thegetTokenProviderClassauth-type guard that the SAS provider class now does. Hadoop fails ("Failed to initialize null") where the native scan builds a store, so this only matters if the driver path changes.Steps to reproduce
Unit-level: build the store with
create_store_with_envand an empty environment for each configuration above, and compare with what Hadoop 3.4.2AbfsConfigurationresolves.Expected behavior
The native scan either resolves the same credential Hadoop does or fails with an error naming the key, and never falls through to the environment when Hadoop configured a mechanism.
Additional context
Hadoop 3.4.1 and older have no container-scoped lookup at all, so items 1 and 3 depend on the Hadoop version on the driver.