Skip to content

ClassCastException with the DynamicIcebergSink #14251

Closed as not planned
Closed as not planned
ClassCastException with the DynamicIcebergSink#14251
Labels
bugSomething isn't workingstale

Description

@zncleon
zncleon
opened on Oct 4, 2025

Apache Iceberg version

main (development)

Query engine

Flink

Please describe the bug 🐞

We are using the DynamicIcebergSink, DynamicRecordGenerator to collect DynamicRecord(composed by GenericRowData), and we set upsert mode. It works correctly at the beginning, but after running for a time, the following problems may occur. What is the cause of this problem?
StackTrace:

java.lang.ClassCastException: class org.apache.iceberg.GenericDataFile cannot be cast to class org.apache.iceberg.DeleteFile (org.apache.iceberg.GenericDataFile and org.apache.iceberg.DeleteFile are in unnamed module of loader 'app')
        at org.apache.iceberg.util.ScanTaskUtil.contentSizeInBytes(ScanTaskUtil.java:34) ~[iceberg-flink-runtime-1.20-1.11.0-SNAPSHOT.jar:?]
        at org.apache.iceberg.SnapshotSummary$UpdateMetrics.addedFile(SnapshotSummary.java:292) ~[iceberg-flink-runtime-1.20-1.11.0-SNAPSHOT.jar:?]
        at org.apache.iceberg.SnapshotSummary$Builder.addedFile(SnapshotSummary.java:119) ~[iceberg-flink-runtime-1.20-1.11.0-SNAPSHOT.jar:?]
        at org.apache.iceberg.MergingSnapshotProducer.add(MergingSnapshotProducer.java:241) ~[iceberg-flink-runtime-1.20-1.11.0-SNAPSHOT.jar:?]
        at org.apache.iceberg.BaseRowDelta.addRows(BaseRowDelta.java:60) ~[iceberg-flink-runtime-1.20-1.11.0-SNAPSHOT.jar:?]
        at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:?]
        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) ~[?:?]
        at org.apache.iceberg.flink.sink.dynamic.DynamicCommitter.commitDeltaTxn(DynamicCommitter.java:321) ~[iceberg-flink-runtime-1.20-1.11.0-SNAPSHOT.jar:?]
        at org.apache.iceberg.flink.sink.dynamic.DynamicCommitter.commitPendingResult(DynamicCommitter.java:258) ~[iceberg-flink-runtime-1.20-1.11.0-SNAPSHOT.jar:?]
        at org.apache.iceberg.flink.sink.dynamic.DynamicCommitter.commitPendingRequests(DynamicCommitter.java:224) ~[iceberg-flink-runtime-1.20-1.11.0-SNAPSHOT.jar:?]
        at org.apache.iceberg.flink.sink.dynamic.DynamicCommitter.commit(DynamicCommitter.java:152) ~[iceberg-flink-runtime-1.20-1.11.0-SNAPSHOT.jar:?]
        at org.apache.flink.streaming.runtime.operators.sink.committables.CheckpointCommittableManagerImpl.commit(CheckpointCommittableManagerImpl.java:151) ~[flink-dist-1.20.2-mac.jar:1.20.2]
        at org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.commitAndEmit(CommitterOperator.java:175) ~[flink-dist-1.20.2-mac.jar:1.20.2]
        at org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.commitAndEmitCheckpoints(CommitterOperator.java:168) ~[flink-dist-1.20.2-mac.jar:1.20.2]
        at org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.initializeState(CommitterOperator.java:136) ~[flink-dist-1.20.2-mac.jar:1.20.2]
        at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:147) ~[flink-dist-1.20.2-mac.jar:1.20.2]
        at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:294) ~[flink-dist-1.20.2-mac.jar:1.20.2]
        at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106) ~[flink-dist-1.20.2-mac.jar:1.20.2]
        at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreStateAndGates(StreamTask.java:858) ~[flink-dist-1.20.2-mac.jar:1.20.2]
        at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$restoreInternal$5(StreamTask.java:812) ~[flink-dist-1.20.2-mac.jar:1.20.2]
        at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55) ~[flink-dist-1.20.2-mac.jar:1.20.2]
        at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:812) ~[flink-dist-1.20.2-mac.jar:1.20.2]
        at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:771) ~[flink-dist-1.20.2-mac.jar:1.20.2]
        at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:970) ~[flink-sql-jdbc-driver-bundle-1.20.2-mac.jar:1.20.2]
        at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:939) [flink-sql-jdbc-driver-bundle-1.20.2-mac.jar:1.20.2]
        at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763) [flink-sql-jdbc-driver-bundle-1.20.2-mac.jar:1.20.2]
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) [flink-sql-jdbc-driver-bundle-1.20.2-mac.jar:1.20.2]
        at java.lang.Thread.run(Thread.java:829) [?:?]

Willingness to contribute

  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time

Activity

  1. zncleon
    added
    bugSomething isn't working
    on Oct 4, 2025

  2. pvary commented on Oct 6, 2025

    @pvary
    pvary
    on Oct 6, 2025
    Contributor

    @zncleon: Could you please check, if there are any deletes already committed to the table with the Dynamic Sink? Do we have any PositionDeletes, or only EqualityDeletes?

    Thanks, Peter

  3. zncleon commented on Oct 10, 2025

    @zncleon
    zncleon
    on Oct 10, 2025
    Author

    @pvary: Thx for your rsp. My Flink procedure only use DynamicRecord and setUpsert(true), I think it just only use EqualityDeletes. But at the same time, I use Trino to optimize (rewrite datafile), maybe it will introduce this error？

  4. pvary commented on Oct 13, 2025

    @pvary
    pvary
    on Oct 13, 2025
    Contributor

    @pvary: Thx for your rsp. My Flink procedure only use DynamicRecord and setUpsert(true), I think it just only use EqualityDeletes. But at the same time, I use Trino to optimize (rewrite datafile), maybe it will introduce this error？

    This should be some internal Flink stuff. Most probably not related to the Trino optimization.
    Somehow a DataFile is generated, but we try to commit as a DeleteFile.

    That is why I'm asking if the sink has created any delete files already (some edge case issue), or the issue appears every time where there is an update (generic issue).

  5. pvary commented on Oct 13, 2025

    @pvary
    pvary
    on Oct 13, 2025
    Contributor

    CC: @mxm

  6. mxm commented on Oct 17, 2025

    @mxm
    mxm
    on Oct 17, 2025
    Contributor

    Thanks you for reporting this bug @zncleon! I did a lot of digging because I couldn't figure out how a DeleteFile could end up as a DataFile. There isn't any obvious bug where we might mix up the two. However, I've found the issue which isn't exactly obvious.

    We create these so called DeletaManifests which contain the paths to the temporary manifest files which we combine later during commit. What we see in the error you reported is that a "delete manifest" ended up getting used as a regular "data manifest". This is caused accidentally reusing the same manifest paths, due to a caching issue with the code that generates these paths.

    Fix is here: #14358

  7. github-actions commented on Apr 16, 2026

    @github-actions
    github-actionsbot
    on Apr 16, 2026 – with GitHub Actions

    This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

  8. github-actions
    added
    stale
    on Apr 16, 2026

  9. github-actions commented on Apr 30, 2026

    @github-actions
    github-actionsbot
    on Apr 30, 2026 – with GitHub Actions

    This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'

  10. github-actions
    closed this as not plannedon Apr 30, 2026

  11. mxm commented on May 7, 2026

    @mxm
    mxm
    on May 7, 2026
    Contributor

    This has been fixed in #14358.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingstale

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions