Skip to content

Migrate exasol provider to pyexasol 2 - #69431

Open
youdie006 wants to merge 2 commits into
apache:mainfrom
youdie006:fix/exasol-pyexasol-2
Open

Migrate exasol provider to pyexasol 2#69431
youdie006 wants to merge 2 commits into
apache:mainfrom
youdie006:fix/exasol-pyexasol-2

Conversation

@youdie006

@youdie006 youdie006 commented Jul 6, 2026

Copy link
Copy Markdown

Closes #69123.

Migrates the Exasol provider to pyexasol 2 and addresses @potiuk's review.

Dependency

  • providers/exasol/pyproject.toml: pyexasol>=0.26.0,<2pyexasol>=2,<3 (keep an upper bound).

Honest type handling (no more cast)
pyexasol 2 ships a py.typed marker, so ExaConnection.execute / export_to_pandas are now typed as (query: str, query_params: dict | None). Instead of casting the hook's broader sql / parameters to the driver types (which lies to mypy), the hook now handles the mismatch at runtime:

  • _serialize_query_params: converts a Mapping to a plain dict and raises a clear TypeError for a positional sequence. Exasol binds named parameters only, so a positional sequence could never have worked — it now fails at the boundary with a helpful message instead of deep inside the driver.
  • _execute_statements: resolves sql the way run() does — a str is one statement, a list[str] is executed sequentially and records are fetched from the last cursor. This keeps the base get_records / get_first str | list[str] contract (a list has been valid input since Fix UnboundLocalError when sql is empty list in ExasolHook #23812) without casting sql to str.
  • run() and _get_pandas_df route parameters through the same helper.

Breaking change + version

  • Bumped the provider to 5.0.0 (major) and added a Breaking changes note under the Changelog header: pyexasol 1.x support is dropped, and a positional parameters sequence now raises. ExasolHook's public API (run / get_records / get_first / pandas export arguments) is otherwise unchanged.

Tests

  • Added coverage in providers/exasol/tests/unit/exasol/hooks/test_exasol.py for the parameter-passing paths (mapping forwarded as a dict; positional sequence rejected) and the list-sql handling. test_run_with_parameters now uses a mapping, since its previous positional tuple is rejected by design.

Verification

  • ruff format --check is clean and there are no new ruff check findings on the changed files; the boundary helpers were unit-checked in isolation. I could not stand up the full Airflow toolchain in my environment this round, so the integrated mypy and provider unit tests run in CI here — I am watching them.

Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines


Drafted-by: Claude Code (Opus 4.8) (no human review before posting)


Important

Maintainer triage note for @youdie006 - by @potiuk - 2026-07-11 14:59 UTC

Helpful heads-up from the maintainers — please address before this PR can be reviewed (see our Pull Request quality criteria):

  • Other failing CI checks. See docs.
  • Pre-commit / static checks. See docs.
  • Provider tests. See docs.

The ball is in your court — you've been assigned to this PR. Fix the above, then mark it Ready for review.

Automated triage — may be imperfect; a maintainer takes the next look.

@boring-cyborg

boring-cyborg Bot commented Jul 6, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@potiuk

potiuk commented Jul 20, 2026

Copy link
Copy Markdown
Member

@youdie006 This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

  • Image build: Failing: Additional CI image checks / Check that image builds quickly. Check Dockerfiles and dependency changes.
  • Pre-commit / static checks: Failing: CI image checks / Static checks. Run prek run --from-ref main --stage pre-commit locally to reproduce and fix.
  • mypy (type checking): Failing: MyPy providers checks. Run the relevant mypy hook locally, e.g. prek run mypy-airflow-core --all-files (providers: breeze run mypy path/to/code).

Note: Your branch is 312 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

  • Fix each issue listed above.
  • Make sure static checks pass locally (prek run --from-ref main --stage pre-commit).
  • Mark the PR as "Ready for review" when you're done.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack.


Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

pyexasol 2.x adds stricter typing to ExaConnection.execute and
export_to_pandas; cast the hook's parameters/sql to the driver's expected
types at the call sites. No public pyexasol symbols were removed.
@youdie006
youdie006 force-pushed the fix/exasol-pyexasol-2 branch from b134946 to 18649ae Compare July 21, 2026 00:07
@youdie006
youdie006 marked this pull request as ready for review July 21, 2026 00:07
@youdie006

Copy link
Copy Markdown
Author

Thanks for the triage @potiuk. I've rebased onto current main (the branch was 328 commits behind) and force-pushed — the rebase was clean, since main hadn't touched either changed file.

Local checks on the rebased branch:

  • Static checks: ruff check and ruff format --check both pass on the changed hook (providers/exasol/src/airflow/providers/exasol/hooks/exasol.py). No stale generated dependencies reference the old pyexasol<2 cap — the only remaining constraint is providers/exasol/pyproject.toml itself (pyexasol>=2).
  • mypy: the cast(...) calls added in this PR exist specifically to satisfy pyexasol 2.x's stricter types in the hook; I couldn't run the full breeze mypy locally, so the fresh CI run should now give an accurate result.

Marked ready for review. Let me know if anything else comes up.

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up — removing the <2 cap is genuinely wanted (#69123) and the rebase onto current main is appreciated.

My concern is that this satisfies mypy rather than actually migrating. Every incompatibility surfaced by pyexasol 2.x's stricter types is silenced with a cast(...) that asserts something the signatures say isn't true — so the runtime mismatch the <2 cap was protecting against is still there, just no longer visible to the type checker. Details inline.

There's also a release-mechanics gap: pyexasol>=2 is a hard floor on a provider that supports apache-airflow>=2.11.0, so anyone currently on pyexasol 1.x breaks on upgrade. That's a breaking change for a provider sitting at 4.10.4 — it needs a major version bump and an entry under the Changelog header in providers/exasol/docs/changelog.rst explaining the required user action. (No newsfragment — providers don't use those.)

And nothing in providers/exasol/tests/ changed. A major dependency migration with zero test movement is hard to gain confidence in; at minimum the parameter-passing paths below want coverage against the 2.x API.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

with closing(self.get_conn()) as conn, closing(conn.execute(sql, parameters)) as cur:
with (
closing(self.get_conn()) as conn,
closing(conn.execute(cast("str", sql), cast("dict | None", parameters))) as cur,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get_records is declared sql: str | list[str], so cast("str", sql) tells mypy the list case cannot happen when the signature explicitly allows it.

To be fair this isn't a regression — conn.execute(sql, ...) already passed a list straight through to pyexasol before this PR. But that pre-existing inconsistency is exactly what the stricter 2.x types just surfaced, and the cast re-buries it. Either narrow the signature to str (if lists were never really supported here), or handle the list branch explicitly the way run() does with sql_list.

Same applies to get_first just below.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

"""
with closing(self.get_conn()) as conn:
df = conn.export_to_pandas(sql, query_params=parameters, **kwargs)
df = conn.export_to_pandas(sql, query_params=cast("dict | None", parameters), **kwargs)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parameters is Iterable | Mapping[str, Any] | None, so cast("dict | None", ...) is a stronger claim than the signature supports: a tuple or list of positional params satisfies Iterable and would reach pyexasol as a non-dict.

If pyexasol 2.x genuinely only accepts a mapping, the honest fix is to tighten the parameter type (and convert or reject sequences at the boundary) rather than assert the narrower type at the call site.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

Comment thread providers/exasol/pyproject.toml Outdated
# Capped to 1.x: pyexasol 2.x ships stricter types that break the exasol hook.
# Remove the cap after migrating; tracked at https://github.com/apache/airflow/issues/69123
"pyexasol>=0.26.0,<2",
"pyexasol>=2",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pyexasol>=2 drops the upper bound entirely — worth keeping one (pyexasol>=2,<3) so the next major doesn't repeat this exercise unannounced.

Also, the removed comment pointed at the tracking issue; once this lands, #69123 should be closed as part of the PR (closes: #69123 in the description).


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

…hange docs + tests

Replace the cast(...) hacks with runtime handling at the boundary, per review:
- _serialize_query_params: convert Mapping parameters to a plain dict and reject
  positional sequences with a clear TypeError (Exasol binds named parameters only)
  instead of casting an unsupported value to dict.
- _execute_statements: handle the sql list branch the way run() does (execute
  sequentially, fetch from the last cursor) so get_records/get_first keep the base
  str | list[str] contract without casting sql to str.
- run() and _get_pandas_df route parameters through _serialize_query_params.

Also:
- pyproject: keep the upper bound with pyexasol>=2,<3.
- Bump the provider to 5.0.0 (major) and document the breaking change in
  changelog.rst under the Changelog header.
- Add tests for the parameter-passing paths and list-sql handling.
@youdie006 youdie006 changed the title Migrate exasol provider to pyexasol 2.x and remove the <2 cap Migrate exasol provider to pyexasol 2 Jul 31, 2026
@youdie006

Copy link
Copy Markdown
Author

Thanks for the careful review @potiuk — reworked in 761fb2b.

cast on parameters → convert/reject at the boundary. Added _serialize_query_params: a Mapping becomes a plain dict, and a positional sequence now raises a clear TypeError instead of being cast to dict. Exasol binds named parameters only, so a positional sequence never actually worked — it now fails at the boundary with a helpful message rather than deep inside the driver. I went with "reject" over "convert" here because there is no meaningful positional-to-named conversion.

cast("str", sql) in get_records / get_first. Took your second option — handle the list branch the way run() does. _execute_statements treats a str as one statement and a list[str] as sequential statements, returning the last cursor to fetch from. That keeps the base str | list[str] contract (a list has been valid input since #23812) with no cast. run()'s own sql_list path was already correct; I only routed its parameters through the new helper.

Upper bound. Restored: pyexasol>=2,<3.

Version + changelog. Bumped to 5.0.0 (major) and added a Breaking changes note under the Changelog header covering the dropped 1.x support and the positional-parameter rejection.

Tests. Added coverage for the mapping/positional parameter paths and the list-sql handling; test_run_with_parameters now uses a mapping, since its previous positional tuple is rejected by design.

closes: #69123 is in the description. One heads-up: I could not stand up the full Airflow toolchain in my environment this round, so I am leaning on CI here for the integrated mypy and provider tests (ruff format/lint are clean locally) and watching them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migrate exasol provider to pyexasol 2.x and remove the <2 cap

2 participants