[v3-3-test] Limit pandas to < 3 for DataFrame XComs (#70791) - #70912
Merged
Conversation
* Limit pandas to < 3 for DataFrame XComs Under pandas 3 a DataFrame does not round trip through an XCom the way it does under pandas 2. The public classes moved to the `pandas` namespace, so the same object qualifies as `pandas.DataFrame` rather than `pandas.core.frame.DataFrame` and the serde registry -- keyed on the latter -- does not dispatch to the pandas serializer at all. The caller gets serde's generic "cannot serialize object of type" with nothing pointing at pandas. Beyond that, the dtypes differ: an `object` column reads back as `str` and missing values as `nan` rather than `None`, so the reader's pandas version, not the writer's, decides what a task receives. Declare `pandas<3` where Airflow declares pandas, and back it with a runtime check, since pandas is an optional dependency and the constraint alone cannot be relied on -- a deployment may install pandas 3 directly or pull it in through another package. The check needs the pandas 3 qualname registered to be reachable at all: without `pandas.DataFrame` in the registry the request never reaches this module. It is registered for that reason only, to refuse the value with a message that names pandas and says what to do, not to support it. Reads are refused as well as writes. A payload written under pandas 2 comes back with pandas 3 dtypes, so accepting it would hand the task different data than was pushed, with no signal. Moving from pandas 2 to pandas 3 is a deliberate migration for users to make. Support for it is not dropped, only deferred. Generated-by: Claude Opus 5 (1M context) following the guidelines at https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions * Sync uv.lock with the pandas < 3 constraint * Add significant newsfragment for the pandas < 3 limit * Apply the pandas < 3 limit to the remaining providers The XCom limit only binds where Airflow declares pandas. Thirteen providers pin pandas independently of common-sql, so a deployment installing any of them could still resolve pandas 3 and reach the runtime refusal rather than being held on a supported version by the constraint. pandas-gbq is deliberately untouched -- a different package, unaffected by this. Provider READMEs and uv.lock are regenerated to match. * Regenerate provider docs index for the pandas < 3 pins The update-providers-build-files hook renders the extras table in each provider's docs/index.rst from its pyproject dependencies. Fourteen providers show the pandas extra, so the pin change widens those tables. * Explain the pandas upper bound at every site and drop a vacuous test contributing-docs/13_airflow_dependencies_and_extras.rst asks for a comment saying why whenever a dependency is upper-bound, and AGENTS.md asks for the tracking URL in the file rather than only in a PR comment. All 45 specifiers across 15 files landed without either. Add the rationale and a pointer to the pandas 3 support PR above each block, so whoever decides when the cap comes off does not have to reconstruct it. test_pandas_2_dataframe_xcom_still_round_trips passed unchanged on main: nothing read pd.__version__ before this PR, so monkeypatching it to a 2.x value while the installed pandas is already 2.x changed nothing observable, and the round trip is already covered by test_pandas_serializers. Dropped. --------- Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com> (cherry picked from commit 415cb70)
potiuk
requested review from
amoghrajesh,
ashb,
kaxil,
o-nikolas and
shahar1
as code owners
August 1, 2026 15:29
boring-cyborg
Bot
added
area:providers
area:task-sdk
kind:documentation
provider:amazon
Contributor
|
Hi maintainer, this PR was merged without a milestone set.
|
1 task
vatsrahul1001
added
the
type:misc/internal
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #70791 to
v3-3-test.Under pandas 3 a DataFrame does not round trip through an XCom the way it does under
pandas 2, so the pandas dependency is capped at
<3everywhere Airflow declares it,backed by a runtime check in the Task SDK serializer.
Conflict resolution
The cherry-pick conflicted in ten files. Every conflict came from unrelated
main-onlychanges rather than from the pandas work itself, so in each case the
v3-3-testside waskept and only the pandas cap was taken:
providers/exasol/pyproject.toml,providers/weaviate/pyproject.toml— kept thebranch pins (
pyexasol>=0.26.0,weaviate-client>=4.4.0); themainside also carrieda
pyexasol<2cap and aweaviate-client>=4.16.0bump that belong to other PRs.providers/{amazon,common/sql,postgres}/docs/index.rst—mainadds an "Optionaldependencies" section generated by tooling that does not exist on
v3-3-test, so thesethree files end up unchanged here.
README.rstanddocs/index.rst— took the pandas rows and tablewidths, restored the branch's client pins.
uv.lock— regenerated with the branch's pinned uv0.11.21rather than hand-merged.The lock diff is larger than the pandas rows alone because capping pandas collapses a
dual-version resolution fork, which in turn simplifies markers on
caio,aiohttp,typing-extensionsandadbc-driver-manager. Regenerating the lock on an otherwisepristine
v3-3-testtree with the same uv produces a zero diff, confirming none of thatchurn is uv-version drift.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Opus 5 (1M context) following the guidelines