perf(storage): retire staged input at shaping boundaries with crash-safe resume (#1418)

[DecisionNerd]

Closes #1418 · part of #1387 · blocks toward #1194 / #900 / #745

What

Shaping held every staged chunk on disk until the whole shape completed, even though nothing reads a chunk again once its records are routed — 186.38 B/edge, 34% of the S22 transient peak, resident for 94% of the run (#1393). The routing loop now works in sealing boundaries:

1. once routed input crosses open_spills × 255 KiB (the perf(storage): one fsync per 255 KB written, because the cache-release window is divided by stream count #1442 bytes-per-fsync budget — one fsync per open spill per seal), every open partition spill seals into a boundary-named segment set (part-<family>-g<boundary>-p<NNNNN>.run/.arrow);
2. a shape-progress-<boundary>.json control is installed — chained by body digest, append-only per boundary;
3. only then are that group's staged inputs unlinked, with the ordinary retire_payload ledger accounting.

Staged residency falls from the whole run to at most one boundary group (saturating at ~4.3 GB for S22+, per the cadence arithmetic).

Crash safety (the acceptance that matters)

• The marker exists only after the group's routed bytes are sealed and durable, and the unlinks only after the marker — an interrupted retirement can neither strand routed data behind missing inputs nor replay onto already-routed spills.
• The head boundary is proven against the chunk receipts' identity-record count (a same-inode corruption that moved the boundary forward would otherwise make resume skip unconsumed chunks); covered boundaries are proven by the digest chain.
• Resume rebuilds partitioners from the claimed sealed segments + the chain's exact routing balance, replays the recorded splitters (the pre-boundary sampling domain is retired), authenticates claimed segment payloads once at the resume boundary, and continues routing at retired_through.
• Unclaimed segments (crash between a group's seal and its marker) are authenticated then discarded; their chunks re-route deterministically from inputs that are still on disk.
• Boundary-less shapes keep the existing behavior byte-for-byte: single finish-time seal, per-family segment unlinks, discard-and-replay recovery.

Measured (same host/inputs/probe as #1393, before = main @ ab1a713e):

scale	staged chunks at the peak instant	rung peak
S17	186.38 → 0.00 B/edge	499.6 → 493.4 B/edge
S18	186.38 → 0.00 B/edge	499.2 → 495.2 B/edge

The rung peak now sets in the later encode/publish phases (per-edge-constant); the admission-relevant staged ceiling saturates at ~4.3 GB for S22+ (projection in the evidence README; S22/S26 confirmation stays with #900/#745). Receipts: docs/development/evidence/staged-retirement-1418/.

Tests

• New crash-mutation tests (subprocess, exit 86) at shape.after_group_seal and shape.after_group_retire: reopen → resume → complete → publish; ledger (storage_current) equals a clean run; current_merge_temporary returns to 0; progress controls are cleaned at supersession.
• Same-facade returned-error retry across boundary retirement on one session object.
• Progress-chain unit tests: head proven against receipts (forward-lying boundary refused), digest-chain tampering refused, grammar round-trips.
• Existing suites green: cargo test --workspace 4,942 passed / 0 failed (durable tests under TMPDIR on ext4), Bazel //:ci_rust_tests 108/108 locally, cargo clippy --workspace -- -D warnings clean, cargo fmt clean, make pre-push-fast green.

Non-goals respected

No S24/S25/S26 execution, no new certification workflow, backward compatibility untouched (pre-v1.0.0 maintainer scope), and no change to the segment spill grammar's classification (Unclassified stays 0 — asserted by the transient-composition test).

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: CurateLabs/graphforge/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: d60373ff-b425-45ed-a189-e3d9010b8614

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}