What is the problem the feature request solves?
Since #5916 the native shuffle writer spills every output partition of a task into one file and records where each partition's blocks are in PartitionedSpill.ranges, a Vec<Vec<Range<u64>>> with one 16-byte range per partition per spill round. The lists live until the task ends: finish_partition reads a partition's ranges but never drops them, and no memory reservation covers them.
That is 16 bytes per spilled block. With 16,000 output partitions and 11 spill rounds it is about 2.8 MB per task; a pool under pressure that spills every batch can reach 16,000 partitions × 88 rounds ≈ 22 MB per task, about 180 MB for an executor running 8 tasks, none of it visible to the memory pool.
Describe the potential solution
- Drop a partition's range list once
finish_partition has copied it into the output.
- Grow the repartitioner's
MemoryReservation by the range size when a range is pushed and shrink it at merge. Use the infallible grow, so a spill cannot fail because of its own bookkeeping.
- Measure retained range memory under heavy spilling (many partitions, many rounds) before and after.
Additional context
Suggested in the review of #5916: #5916 (comment)
What is the problem the feature request solves?
Since #5916 the native shuffle writer spills every output partition of a task into one file and records where each partition's blocks are in
PartitionedSpill.ranges, aVec<Vec<Range<u64>>>with one 16-byte range per partition per spill round. The lists live until the task ends:finish_partitionreads a partition's ranges but never drops them, and no memory reservation covers them.That is 16 bytes per spilled block. With 16,000 output partitions and 11 spill rounds it is about 2.8 MB per task; a pool under pressure that spills every batch can reach 16,000 partitions × 88 rounds ≈ 22 MB per task, about 180 MB for an executor running 8 tasks, none of it visible to the memory pool.
Describe the potential solution
finish_partitionhas copied it into the output.MemoryReservationby the range size when a range is pushed and shrink it at merge. Use the infalliblegrow, so a spill cannot fail because of its own bookkeeping.Additional context
Suggested in the review of #5916: #5916 (comment)