Skip to content

Native fanout Iceberg writes fail when their open partitions outgrow the memory pool #6771

Closed

Description

@andygrove

A native fanout write keeps a data file open for every partition a task writes to, and since #6247 its buffers count against the task's memory pool. The writer cannot spill, so a task that writes to enough partitions fails with Additional allocation failed for IcebergWriteExec, where iceberg-java's fanout writer, buffering on the JVM heap, succeeds. Spark's retry takes the same path, so the job fails.

On Iceberg 1.5+ an unsorted partitioned table uses the fanout writer by default, and #6664 makes native writes the default, so this has to be fixed first. Raised in review of #6664.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:Icebergarea:writerNative Parquet writerbugSomething isn't workingpriority:mediumFunctional bugs, performance regressions, broken features

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions