refactor(pyiceberg-core): remove DataFusion TableProvider export by kevinjqliu · Pull Request #3143 · apache/iceberg-rust

kevinjqliu · 2026-09-03T18:21:02Z

Which issue does this PR close?

Closes [pyiceberg-core] Remove the DataFusion TableProvider export from the Python bindings #3036.

What changes are included in this PR?

Remove the pyiceberg_core.datafusion module and IcebergDataFusionTable export.
Remove the DataFusion FFI, iceberg-datafusion, OpenDAL, and Tokio dependencies from pyiceberg-core.
Remove DataFusion-specific tests and development dependencies.
Enable Arrow's chrono-tz feature directly to preserve timezone transform behavior previously enabled transitively by DataFusion.
Regenerate the Cargo and Python lockfiles.

Wheel size impact

Release wheels built from latest main and this PR with the same Maturin/Python toolchain on macOS arm64:

Revision	Wheel size
`main`	13.98 MiB
This PR	1.14 MiB

This reduces the wheel by 12.84 MiB (91.8%).

Are these changes tested?

uv 0.9.3 lock --check
Locked dependency installation with uv 0.9.3
Python binding build with Maturin
Python test suite: 12 passed
cargo check -p pyiceberg_core_rust
cargo fmt --all -- --check
cargo deny check license

AI Disclosure

GitHub Copilot assisted with implementation, dependency cleanup, testing, and review.
Kevin reviewed everything again 😄

Copilot

🟢 Approval recommended

The DataFusion binding/export and all in-repo references/tests/dependencies appear consistently removed, with explicit Arrow feature replacement and updated lockfiles/inventory.

Pull request overview

This PR removes the experimental DataFusion integration from the pyiceberg-core Python bindings, decoupling the bindings from DataFusion’s FFI/compat constraints while preserving Arrow timezone behavior by enabling the needed Arrow feature directly.

Changes:

Removes the pyiceberg_core.datafusion module / IcebergDataFusionTable export and associated Rust glue code (including the Tokio runtime helper).
Drops DataFusion/FFI and related dev/test dependencies, and deletes DataFusion- and Hugging Face–specific tests.
Regenerates Rust/Python lockfiles and the Rust dependency/license inventory, and enables Arrow’s chrono-tz feature explicitly for the bindings crate.

File summaries

File	Description
Cargo.lock	Removes DataFusion/FFI-related Rust dependencies from the resolved graph for the Python bindings.
bindings/python/uv.lock	Updates Python lockfile to drop DataFusion and other removed dev/test dependencies.
bindings/python/tests/test_huggingface_and_cdc.py	Deletes Hugging Face + CDC + DataFusion integration tests tied to the removed export.
bindings/python/tests/test_datafusion_table_provider.py	Deletes DataFusion table provider binding tests tied to the removed export.
bindings/python/src/runtime.rs	Removes Tokio runtime helper previously used for DataFusion provider construction.
bindings/python/src/lib.rs	Stops registering the removed `datafusion` submodule in the extension module.
bindings/python/src/datafusion_table_provider.rs	Deletes the Rust implementation of the `IcebergDataFusionTable` PyO3 class and capsule export.
bindings/python/pyproject.toml	Removes DataFusion/Hugging Face dev deps and warning ignore; keeps warnings-as-errors.
bindings/python/DEPENDENCIES.rust.tsv	Regenerates the Rust dependency license inventory after dependency removals.
bindings/python/Cargo.toml	Drops DataFusion/FFI/OpenDAL/Tokio deps; enables `arrow`’s `chrono-tz` feature explicitly.

Review details

Files reviewed: 8/10 changed files
Comments generated: 0
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Remove the DataFusion table provider export, its tests, and its Rust and Python dependency trees from pyiceberg-core. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

kevinjqliu · 2026-09-03T18:32:34Z

 [dependencies]
-arrow = { workspace = true, features = ["pyarrow"] }
-datafusion-ffi = { workspace = true }
+arrow = { workspace = true, features = ["chrono-tz", "pyarrow"] }


DataFusion previously enabled Arrow’s chrono-tz feature transitively, test_year_transform_with_tz requires it

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

kevinjqliu · 2026-09-03T18:36:30Z

-filterwarnings = [
-  "error",
-  # huggingface_hub uses hf_xet.upload_files() internally which is deprecated in hf_xet
-  "ignore::DeprecationWarning:huggingface_hub",


this was needed by the now removed bindings/python/tests/test_huggingface_and_cdc.py file

kevinjqliu · 2026-09-03T18:44:31Z

removing this too since it rely on IcebergDataFusionTable

CTTY

LGTM!

kevinjqliu · 2026-09-04T16:15:18Z

thanks @CTTY

…tale audit ignores (#3149) Follow-up cleanup after the DataFusion TableProvider removal (#3143). - Remove `datafusion-ffi` from workspace deps; nothing inherits it since the Python binding no longer exports a TableProvider. - Remove `aes` and `thrift` workspace deps; no crate inherits them (encryption uses the AES types re-exported by `aes-gcm`). - Drop audit ignores whose advisory no longer matches any locked crate: RUSTSEC-2024-0014 (generational-arena, gone with datafusion-ffi), RUSTSEC-2024-0388 (derivative), RUSTSEC-2025-0134 (rustls-pemfile), RUSTSEC-2026-0097 (rand, all locked versions patched), RUSTSEC-2026-0176/0177 (pyo3, lock is on 0.29.1). - Drop the unmatched MPL-2.0 license exception for generational-arena. Cargo.lock is unchanged. Verified with `cargo check --workspace --all-targets --all-features`, `cargo audit`, and `cargo deny check licenses`. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

Copilot AI lite review requested due to automatic review settings September 3, 2026 18:21

Copilot started reviewing on behalf of kevinjqliu September 3, 2026 18:21 View session

Copilot AI reviewed Sep 3, 2026

View reviewed changes

refactor(python): remove DataFusion integration

40e9e3c

Remove the DataFusion table provider export, its tests, and its Rust and Python dependency trees from pyiceberg-core. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

kevinjqliu force-pushed the kevinjqliu-remove-pyiceberg-datafusion branch from 8e18537 to 40e9e3c Compare September 3, 2026 18:25

chore(python): leave dependency inventory unchanged

4c5f7ab

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

kevinjqliu commented Sep 3, 2026

View reviewed changes

kevinjqliu and others added 2 commits September 3, 2026 11:32

docs(python): clarify PyArrow test dependency

ec9fc2e

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

docs(python): explain PyIceberg core conflict

76b16b3

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

kevinjqliu commented Sep 3, 2026

View reviewed changes

kevinjqliu changed the title ~~refactor(python): remove DataFusion integration~~ refactor(pyiceberg-core): remove DataFusion TableProvider export Sep 3, 2026

kevinjqliu requested review from CTTY and blackmwk September 3, 2026 18:37

kevinjqliu commented Sep 3, 2026

View reviewed changes

kevinjqliu mentioned this pull request Sep 3, 2026

[pyiceberg-core] Remove the DataFusion TableProvider export from the Python bindings #3036

Closed

CTTY approved these changes Sep 3, 2026

View reviewed changes

CTTY merged commit 75c9201 into main Sep 3, 2026
21 checks passed

CTTY deleted the kevinjqliu-remove-pyiceberg-datafusion branch September 3, 2026 23:47

kevinjqliu mentioned this pull request Sep 4, 2026

chore(deps): remove dead datafusion-ffi, unused workspace deps, and stale audit ignores #3149

Merged

comphead mentioned this pull request Sep 28, 2026

ci: check dependency licenses with cargo-deny on every PR #3293

Open

Navigation Menu

refactor(pyiceberg-core): remove DataFusion TableProvider export - #3143

refactor(pyiceberg-core): remove DataFusion TableProvider export#3143
CTTY merged 4 commits into
mainfrom
kevinjqliu-remove-pyiceberg-datafusion

kevinjqliu commented Sep 3, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

kevinjqliu Sep 3, 2026 •

edited

Loading

Uh oh!

kevinjqliu Sep 3, 2026

Uh oh!

kevinjqliu Sep 3, 2026

Uh oh!

CTTY left a comment

Uh oh!

Uh oh!

kevinjqliu commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

kevinjqliu commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

What changes are included in this PR?

Wheel size impact

Are these changes tested?

AI Disclosure

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

kevinjqliu Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinjqliu Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

kevinjqliu Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

CTTY left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kevinjqliu commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kevinjqliu commented Sep 3, 2026 •

edited

Loading

kevinjqliu Sep 3, 2026 •

edited

Loading