What is the problem the feature request solves?
PR #5038 reuses the Arrow IPC write context across shuffle blocks. On the original Arrow 58.4.0 baseline, CompressionContext owned only an optional zstd compressor; the FlatBufferBuilder was still allocated separately for every batch. Since Comet did not enable Arrow IPC buffer compression, that lifetime change alone did not provide metadata-allocation savings.
Upstream main uses DataFusion 55.0.0 and Arrow/Parquet 59.2.0 through merged #5262. The dependency upgrade is no longer a blocker. The upgrade also migrates the shuffle writer paths to IpcWriteContext. Arrow 59.2.0's write context owns a reusable FlatBufferBuilder, so the deferred optimization can now be measured on main.
This issue tracks validating the shuffle performance benefit, not another dependency upgrade.
Describe the potential solution
Completion checklist:
Comet's outer per-block compression encoders are independent and outside this issue's scope.
Additional context
What is the problem the feature request solves?
PR #5038 reuses the Arrow IPC write context across shuffle blocks. On the original Arrow 58.4.0 baseline,
CompressionContextowned only an optional zstd compressor; theFlatBufferBuilderwas still allocated separately for every batch. Since Comet did not enable Arrow IPC buffer compression, that lifetime change alone did not provide metadata-allocation savings.Upstream
mainuses DataFusion 55.0.0 and Arrow/Parquet 59.2.0 through merged #5262. The dependency upgrade is no longer a blocker. The upgrade also migrates the shuffle writer paths toIpcWriteContext. Arrow 59.2.0's write context owns a reusable FlatBufferBuilder, so the deferred optimization can now be measured on main.This issue tracks validating the shuffle performance benefit, not another dependency upgrade.
Describe the potential solution
Completion checklist:
IpcWriteContext, retaining the reusable context lifetimes introduced by perf: reuse Arrow IPC compression context across shuffle blocks #5038.Comet's outer per-block compression encoders are independent and outside this issue's scope.
Additional context