Skip to content

DataFusion non-append InsertOp is silently committed as append #2711

Description

@u70b3

Apache Iceberg Rust version

Current main at ac92ec9 (version 0.10.0).

Describe the bug

IcebergTableProvider::insert_into accepts DataFusion's InsertOp, but currently ignores it. The physical commit path always uses Transaction::fast_append.

As a result, InsertOp::Overwrite or InsertOp::Replace can be planned and executed as an append instead of returning an unsupported-operation error. This can produce incorrect table contents because the requested write semantics are silently changed.

To Reproduce

  1. Construct an IcebergTableProvider backed by a catalog.
  2. Call TableProvider::insert_into with a valid input plan and InsertOp::Overwrite or InsertOp::Replace.
  3. Observe that an IcebergCommitExec plan is returned.
  4. The commit executor subsequently commits generated files through fast_append.

Relevant code:

  • crates/integrations/datafusion/src/table/mod.rs ignores the InsertOp argument.
  • crates/integrations/datafusion/src/physical_plan/commit.rs unconditionally uses fast_append.

Expected behavior

Until overwrite and replace semantics are implemented end to end, IcebergTableProvider::insert_into should accept only InsertOp::Append and return a clear unsupported/not-implemented error for Overwrite and Replace before constructing a write plan.

Willingness to contribute

I can contribute a fix for this bug independently.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions