fix(ingest): drop the sqlglot requirement from auto_resolve_lineage_urns

[aviraj-gour]

What changed

auto_resolve_lineage_urns failed config parse when sqlglot was not installed:

auto_resolve_lineage_urns is enabled but the SQL parser it relies on is not installed.

The processor never parses SQL. It uses exactly two things from datahub.sql_parsing:

• SchemaResolver.resolve_urn(urn) — an exact-URN cache lookup plus graph fetch
• match_columns_to_schema() — a lowercase column map

Neither needs sqlglot. The requirement came from an import chain: schema_resolver.py
imports _TableName from sql_parsing/_models.py, which did a module-level
import sqlglot for three conversion methods. _TableName's data core is just
(database, db_schema, table, parts) and none of that needs sqlglot.

_models.py is now sqlglot-free, so importing schema_resolver no longer pulls
sqlglot and the config-parse guard is unnecessary.

The feature becomes usable on any connector, not only those whose extra bundles
sql-parser. No behaviour change when sqlglot is installed.

Code removed

_TableName.from_sqlglot_table — dead. sqlglot_lineage.py already has
_table_name_from_sqlglot_table, which is what every production path calls; the
classmethod had no callers outside tests. The two duplicated the same SemanticView and
Dot traversal, with a comment stating they were kept in sync by hand.

The surviving one handles two cases the deleted copy got wrong: it restores MSSQL
temp-table prefixes (#, ##) and splits Snowflake IDENTIFIER('db.sch.tbl') into its
parts. It is not a strict superset — for Anonymous table expressions (table-valued
functions, BigQuery EXTERNAL_QUERY) it yields table='' where the deleted method
yielded the function name. That is pre-existing production behaviour, since the deleted
method had no production callers, so this PR does not change it.

The 19 test call sites moved across with no assertion changes.

_TableName.qualified(dialect=...) — the dialect parameter was never read in the
method body. Dropped, along with it at the four call sites.

AutoResolveLineageUrnsConfig._require_sql_parser_when_enabled — the validator that
raised when sqlglot was missing, plus the matching sentence in the enabled field
description and the caveat in lineage_urn_casing.md.

Code moved

_TableName.as_sqlglot_table → sqlglot_lineage._table_name_as_sqlglot_table(),
which is where its single caller lives.

Tests

test_module_import_does_not_pull_sqlglot now also asserts that importing
datahub.sql_parsing.schema_resolver leaves sqlglot out of sys.modules. This is the
guard for the invariant the change relies on; previously nothing covered it, because
the processor defers its schema_resolver import and the test passed either way.

Two tests removed:

• test_config_requires_sql_parser_only_when_enabled asserted the deleted validator.
• test_qualified_preserves_temp_prefix was vacuous — qualified() copies
  self.table verbatim and never inspected the dialect, so the # it checked for was
  simply its own input.

Verified with tests/unit/sql_parsing, tests/unit/dremio, tests/unit/cube,
tests/unit/dbt, tests/unit/kafka_connect, tests/unit/sdk_v2/test_dataset.py and
tests/unit/workunit_processors (1901 passed), plus ./gradlew :metadata-ingestion:lint
(ruff and mypy clean).

Checklist

• The PR conforms to DataHub's Contributing Guideline
• Tests for the changes have been added/updated
• Docs related to the changes have been added/updated
• Breaking change — n/a

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

[datahub-connector-tests]

Connector Tests Results

All connector tests passed for commit 30185ba

View full test logs →

To skip connector tests, add the skip-connector-tests label (org members only).

Autogenerated by the connector-tests CI pipeline.

[cursor]

PR Summary

Overview
auto_resolve_lineage_urns no longer requires sqlglot at install or config parse. The processor only uses URN/schema helpers from schema_resolver, but those used to pull in sqlglot because _TableName in _models.py imported it for conversion helpers. That module-level dependency is removed, along with the AutoResolveLineageUrnsConfig validator and docs that mandated acryl-datahub[sql-parser].

SQLGlot-specific table conversion is consolidated in sqlglot_lineage. Dead _TableName.from_sqlglot_table / as_sqlglot_table and the unused qualified(dialect=...) parameter are dropped; production code uses _table_name_from_sqlglot_table and new _table_name_as_sqlglot_table. Tests and Dremio/schema-resolver unit tests follow those APIs.

Import-safety tests are tightened so importing auto_resolve_lineage_urns, schema_resolver, and schema_resolver_provider does not load sqlglot in a fresh interpreter.

^{Reviewed by Cursor Bugbot for commit 30185ba. Bugbot is set up for automated code reviews on this repo. Configure here.}

[sgomezvillamor]

Approving — no blockers, and I confirmed there's no breaking change.

I traced every removed or changed surface against origin/master rather than relying on the description:

• _TableName.from_sqlglot_table — zero production callers on master (every hit is a test, plus its own recursion and two comments). So the documented non-equivalence for Anonymous table expressions (table='' via _restore_mssql_temp_table_prefix vs. the function name) isn't reachable from any production path.
• as_sqlglot_table — one production caller, updated; the moved body is identical.
• qualified(dialect=…) — all four call sites use keyword args. Worth noting the downstream failure modes are loud either way: a keyword caller gets TypeError, a positional one hits a pydantic ValidationError on Optional[str]. And _TableName is private with nothing re-exported from datahub/sql_parsing/__init__.py.
• The config validator — strictly relaxes validation, so nothing that parsed before fails now. No updating-datahub.md entry needed.
• The runtime premise — schema_resolver.py has zero sqlglot occurrences, schema_resolver_provider pulls only schema_resolver / graph filters / PerfTimer, sql_parsing_common mentions it only in comments and dialect strings, and graph/client.py's imports are TYPE_CHECKING + function-local. The processor's runtime path uses only provide_schema_resolver, SchemaResolver(...), resolve_urn() and match_columns_to_schema.

Nice side effect worth calling out: cube_lineage, dbt_common, kafka_connect/common and dremio_source all import _models, which is no longer an sqlglot carrier for any of them.

Both test deletions look right to me — one covered code that no longer exists, and test_qualified_preserves_temp_prefix was genuinely vacuous since qualified() copies self.table verbatim.

Two non-blocking nits inline. One thing to watch before merge: connector-tests failed on 32fb216 and was still in progress on head when I looked.

---

Generated by Claude Code

[sgomezvillamor]

Nit: this guard lives somewhere a sql_parsing/ developer won't look.

Extending the assertion here is the right call — it's the invariant the whole PR rests on. But the test now constrains datahub.sql_parsing.schema_resolver while sitting under tests/unit/workunit_processors/. Someone adding a module-level import sqlglot to schema_resolver.py has no reason to look in this file, and CI will point them at a workunit-processor test for a change they made in sql_parsing/.

Either a mirrored guard under tests/unit/sql_parsing/, or — cheaper — a one-line note at the top of schema_resolver.py:

# Must stay sqlglot-free: auto_resolve_lineage_urns relies on this module being
# importable without the [sql-parser] extra. Guarded by
# tests/unit/workunit_processors/test_auto_resolve_lineage_urns.py::test_module_import_does_not_pull_sqlglot

Non-blocking.

---

Generated by Claude Code