Skip to content

Native Azure credential lookup still differs from Hadoop ABFS in a few configurations #6605

Description

@dwsmith1983

Describe the bug

The native Parquet scan resolves Azure credentials from the Hadoop fs.azure.* settings, and the goal is to pick the identity Hadoop's ABFS driver would. After #6059 a few gaps remain where the two can disagree. None of them comes from #6059; they were found while checking it against Hadoop 3.4.2's AbfsConfiguration.

  1. Container-scoped OAuth keys. Since Hadoop 3.4.2, getPasswordString probes <key>.<container>.<host> before the account and global forms, and getTokenProvider reads the client id, secret and endpoint, the MSI tenant, endpoint and authority, and the token file through it. The native scan reads that form only for the SAS fixed token. With fs.azure.account.oauth2.client.secret.data.myacct.dfs.core.windows.net set next to an account-level secret, Hadoop authenticates as the container's principal and the native scan as the account's. With only the container-scoped secret, the native scan fails.
  2. Fabric hosts. ENDPOINT_SUFFIXES lists only the core.windows.net suffixes, but object_store also accepts onelake.dfs.fabric.microsoft.com. Account-scoped keys for a Fabric host are never found, so a configured principal is missed and the scan falls through to ambient AZURE_* variables or managed identity.
  3. Account-scoped key forms. Hadoop's accountConf builds <key>.<host> from the raw URL authority, port included. The native scan also probes the other endpoint suffix and the short <key>.<account> form, ahead of the global key, and drops an explicit port. A stale key Hadoop never reads can therefore win over the global one.
  4. Credential provider stores. Configuration.getPassword consults hadoop.security.credential.provider.path (for example JCEKS) before plain configuration. The native scan only sees plain configuration, so a token kept in a store is invisible and a lower-priority plaintext value is used instead.
  5. OAuth provider class lookup. fs.azure.account.oauth.provider.type does not apply the getTokenProviderClass auth-type guard that the SAS provider class now does. Hadoop fails ("Failed to initialize null") where the native scan builds a store, so this only matters if the driver path changes.
  6. Error text. The "Scheme of URL is not Azure" error formats the whole URL, which would echo a password in the userinfo.

Steps to reproduce

Unit-level: build the store with create_store_with_env and an empty environment for each configuration above, and compare with what Hadoop 3.4.2 AbfsConfiguration resolves.

Expected behavior

The native scan either resolves the same credential Hadoop does or fails with an error naming the key, and never falls through to the environment when Hadoop configured a mechanism.

Additional context

Hadoop 3.4.1 and older have no container-scoped lookup at all, so items 1 and 3 depend on the Hadoop version on the driver.

Activity

  1. added
    bugSomething isn't working
    priority:mediumFunctional bugs, performance regressions, broken features
    area:scanParquet scan / data reading
    and removed on Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:scanParquet scan / data readingbugSomething isn't workingpriority:mediumFunctional bugs, performance regressions, broken features

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions