perf(storage): the PartitionRun accumulator is unbounded and grows with partition size

[DecisionNerd]

Summary

PR #1440 cleared the S22 rss_bounded_or_plateaued gate and cost ~23% ingest throughput. Recovering it is worth ~2.40 h at S26 — more than the entire nine-item constant-factor plan combined, and the single largest item on the board.

	S18	S19	S20
before #1440	99,921	102,516	101,226 edges/s
after #1440	80,408	78,171	77,412 edges/s
	-19.5%	-23.7%	-23.5%

S20 wall 319s -> 381s. Projected S26 moves 5.53 h -> 7.93 h against a 3.20 h ceiling.

#1440 must not be reverted. It takes projected S26 ingest RSS from 5.57 GB to 2.17 GB against a 3.44 GB limit (slope 5.08 -> 1.94 B/edge). The memory fix and the throughput loss are believed separable.

Mechanism

#1440 removed the per-partition BufWriter::with_capacity(SPILL_BLOCK_BYTES /* 64 KiB */, hashing) and replaced it with PartitionRun (graph_construction/shape.rs:1056), which accumulates a whole contiguous same-partition run into a Vec<u8> and writes it once. It flushes only on partition change.

The routing key is the record's leading 16 bytes, which is also the staged run's sort key, and the range partition is monotone — so runs really are contiguous. (An earlier "scattered writes" hypothesis was checked and killed.)

Which means: with ~256 partitions and ~33.5M endpoint records at S20, a run averages ~131k records ≈ 4.3 MB. The write buffer went from a fixed 64 KiB (L2-resident) to a multi-megabyte Vec that grows by reallocation, and clear() retains capacity so it stays at peak.

This predicts both symptoms at once: throughput down ~23%, and ingest RSS still growing ~1.94 B/edge after the per-partition buffers were removed.

The fix

Flush PartitionRun when bytes.len() crosses a bound as well as on partition change. Writes stay large enough to amortize the syscall, the buffer stays cache-resident, and the accumulator stops scaling with partition size.

Same shape as the #1442 decision: reprice a bounded constant, do not remove the mechanism. #1440's error was deleting the buffer outright when a bounded one was what it needed.

Acceptance — BOTH, or it is not a fix

• Throughput back to ~100k edges/s at S18/S19/S20.
• Ingest RSS still ~91/97/113 MB with slope near 1.9 B/edge. The memory win is what unblocks S22 and must not regress.
• Report the bound curve (64 KiB / 256 KiB / 1 MiB), not a single value chosen by taste.
• Durable artifacts self-consistent via the determinism suite. TCK green.

Verify first

Histogram PartitionRun.bytes.len() at every flush, per family, at S18. If mean flush size is in the megabytes, confirmed. If flushes are already small this hypothesis is dead and the cost is in DurableFileCacheWriter's per-write window arithmetic instead.

Part of #1387. Follows #1439 / PR #1440.

[DecisionNerd]

Verified on a quiet host: the throughput symptom is resolved, the mechanism is not

Three clean-host ladders bracket #1440. All are clean-*-evidence/ runs; ingest wall and RSS are measured.

ladder	S18	S19	S20
pre-#1440 9269362f	99,921/s · 142 MB	102,516/s · 160 MB	101,226/s · 203 MB
on #1440 1955f17d	80,408/s · 91 MB	78,171/s · 97 MB	77,412/s · 114 MB
post-#1440 f80f69fe	101,175/s · 89 MB	105,415/s · 98 MB	105,802/s · 113 MB

This issue's numbers were right. The middle row is exactly the table in the body — the regression was real and was measured cleanly on 1955f17d, not on a contended host.

And it has since been resolved. f80f69fe — the merged form of #1440, which also routes node-keyed families with node-only splitters — recovers throughput to at or above the pre-#1440 baseline while keeping the memory win. Against this issue's own acceptance criteria:

• "Throughput back to ~100k edges/s at S18/S19/S20" → 101,175 / 105,415 / 105,802. Met.
• "Ingest RSS still ~91/97/113 MB" → 89.1 / 97.8 / 113.0 MB. Met, to within a megabyte.

Both, which is what this issue demanded. The projected ~2.40 h at S26 that made this "the single largest item on the board" is therefore already banked, and it should not be scoped as available headroom in any plan.

Not closing it, because the mechanism is still in the tree

The hypothesised cause — an unbounded PartitionRun accumulator — was never fixed. graph_construction/shape.rs:1066:

struct PartitionRun { partition: Option<usize>, bytes: Vec<u8>, records: u64 }

push still flushes only on partition change:

if self.partition.is_some_and(|current| current != partition) {
    self.flush(target, evidence)?;
}

There is no byte bound. SPILL_BLOCK_BYTES (64 KiB) survives only in partition_shaping.rs on a different writer. So the accumulator still grows to a whole contiguous same-partition run — ~131k records ≈ 4.3 MB at S20 — and clear() retains capacity, so it stays at peak.

That matters more now, not less. It scales with partition size, and partition size is exactly what grows with data while the cut caps at 256 partitions (#1439) — 262,144 rows per partition at S22 and rising. #1456 lists it first in its dataflow-defect table. It is currently not costing throughput at these scales, which is why it is no longer urgent; it is still an unbounded buffer on the ingest path.

What changes

Retitled from "recover the ~23% ingest throughput PR #1440 cost, by bounding the PartitionRun buffer" to name the defect that remains rather than the symptom that is gone. Re-scoped as a bounded-buffer fix, and moved under #1456.

Acceptance is now the memory property, not the throughput one:

• PartitionRun flushes on a byte bound as well as on partition change.
• Report the bound curve (64 KiB / 256 KiB / 1 MiB), not a single value chosen by taste.
• Throughput does not regress from the f80f69fe baseline — 101,175 / 105,415 / 105,802 at S18/S19/S20 on a quiet host. This is now a floor to protect, not a target to reach.
• Determinism suite green; TCK green.

Still worth doing before the #1456 engine work adopts framework spill, since an unbounded accumulator interacts badly with a memory pool that does not account for it.

[DecisionNerd]

Evidence relocated from the docs tree (#1625). The 2 file(s) below were committed under docs/development/ as measurement evidence for this issue. They are removed from the tree by #1628 (the #1625 pull request); the repository keeps method and content digests, results live here. Every file remains available at its permalink on commit 29a7b34ebe44, and its SHA-256 is recorded next to it.

Files (2 total, 278 lines)

• docs/development/evidence/partition-run-bound-1445.json sha256:15d99a6965819b21702fc208f78fd470a1fd02b05eadee168023900dc990187d
• docs/development/evidence/partition-run-bound-1445.md sha256:c2917123d314df39dad59dbab5cc9eae4e3459f0af7c048411288e0c45a2375e