Summary
FixedRangePartitioner::load_partition (graph_construction/partition_shaping.rs:565) returns Vec<[u8; N]> — fully padded fixed-size records — while DetailCodec::bytes trims trailing zeros on the wire. For Graph500-shaped data that is a 5-6x in-memory multiplier over what was written.
Node details are 272 B padded against roughly 18 B plus a name on the wire; edge details are 304 B against roughly 50 B plus a name. The partition is materialized whole before sorting, so the multiplier applies to the largest resident allocation on the ingest path.
Found during a DataFusion capability assessment, verified in source.
Why it matters
At S26 with balanced partitions, edge details alone would load
1,073,741,824 records x 304 B / 256 partitions = ~1.27 GB
in a single partition, against a 4 GiB budget — roughly a third of it, before Vec doubling. Compact storage of the same data is about 230 MB.
This is a separate term from the endpoints skew (#1439) and from the per-partition spill buffers. It does not cause the measured RSS slope; it multiplies whatever the largest partition holds. Fixing the skew without fixing this leaves a 5-6x factor on the dominant remaining allocation.
Suggested direction
Store the wire bytes plus an offset index rather than padded arrays, and sort by the 16-byte key prefix. The records are already length-delimited on disk and the codec already knows how to read them; what is missing is a representation that does not re-inflate them in memory.
This is in-memory only. The wire format does not change, so no digest moves. That is the property to verify first and it makes the change cheap to validate.
Acceptance
- Peak RSS of a details-only partition load measurably reduced at a fixed scale; report the multiplier achieved against the 5-6x predicted.
- The four fixed-width digests are unchanged:
shaped-identities.run 6ea5b846…, node details d7eac2fd…, edge details 4b9d9dd1…, endpoints 0b3c8415…. If any moves, the representation change has leaked into the wire format and the fix is wrong.
partition_count_changes_the_layout_but_not_the_logical_result still passes.
Related
#1439 (the RSS slope and the endpoints skew — a different term), #1387.
Summary
FixedRangePartitioner::load_partition(graph_construction/partition_shaping.rs:565) returnsVec<[u8; N]>— fully padded fixed-size records — whileDetailCodec::bytestrims trailing zeros on the wire. For Graph500-shaped data that is a 5-6x in-memory multiplier over what was written.Node details are 272 B padded against roughly 18 B plus a name on the wire; edge details are 304 B against roughly 50 B plus a name. The partition is materialized whole before sorting, so the multiplier applies to the largest resident allocation on the ingest path.
Found during a DataFusion capability assessment, verified in source.
Why it matters
At S26 with balanced partitions, edge details alone would load
in a single partition, against a 4 GiB budget — roughly a third of it, before
Vecdoubling. Compact storage of the same data is about 230 MB.This is a separate term from the endpoints skew (#1439) and from the per-partition spill buffers. It does not cause the measured RSS slope; it multiplies whatever the largest partition holds. Fixing the skew without fixing this leaves a 5-6x factor on the dominant remaining allocation.
Suggested direction
Store the wire bytes plus an offset index rather than padded arrays, and sort by the 16-byte key prefix. The records are already length-delimited on disk and the codec already knows how to read them; what is missing is a representation that does not re-inflate them in memory.
This is in-memory only. The wire format does not change, so no digest moves. That is the property to verify first and it makes the change cheap to validate.
Acceptance
shaped-identities.run6ea5b846…, node detailsd7eac2fd…, edge details4b9d9dd1…, endpoints0b3c8415…. If any moves, the representation change has leaked into the wire format and the fix is wrong.partition_count_changes_the_layout_but_not_the_logical_resultstill passes.Related
#1439 (the RSS slope and the endpoints skew — a different term), #1387.