You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native fanout Iceberg writes fail when their open partitions outgrow the memory pool #6771
A native fanout write keeps a data file open for every partition a task writes to, and since #6247 its buffers count against the task's memory pool. The writer cannot spill, so a task that writes to enough partitions fails with Additional allocation failed for IcebergWriteExec, where iceberg-java's fanout writer, buffering on the JVM heap, succeeds. Spark's retry takes the same path, so the job fails.
On Iceberg 1.5+ an unsorted partitioned table uses the fanout writer by default, and #6664 makes native writes the default, so this has to be fixed first. Raised in review of #6664.
Activity