feat(storage): worker-local partition loads with coordinator-owned evidence (#1456 item 1)

[DecisionNerd]

Summary

Work item 1 of #1456: separate worker results from shared evidence, on the narrowest slice that exercises the real shape — the fixed-width finish_optional loop in graph_construction/partition_shaping.rs, whose evidence.clone() → mutate → *evidence = committed across an fsync is the structural obstacle #1429 and #1448 died on.

• Workers return results plus local counters. load_fixed_partition never sees GraphConstructionEvidence; it returns the sorted partition and a PartitionLoadCounters (plain sums plus per-partition maxima). The merge is explicit and total: PartitionLoadCounters::merge_into writes exactly nine evidence fields, pinned by merge_is_total_and_touches_nothing_else.
• One coordinator owns evidence, ledger and publication. consume_in_partition_order (new graph_construction/partition_load.rs) runs loads on a bounded pool and hands results to the calling thread in canonical partition index order. The ordered critical section is now: merge counters → write the partition. Decoding, sorting and bulk reads run outside it. No worker touches the allocation ledger, so the exact coexistence peak and ordered transition log are byte-for-byte what a sequential pass records.
• Backpressure. At most workers partitions are materialized at any instant, counting loads in flight, loaded-but-unconsumed results, and the one being written — the threads bound perf(storage): make one complete ingest scale with available cores #1448's reorder buffer claimed and did not deliver. A slow first partition holds the pool at the bound (test: slow_first_partition_is_consumed_first_and_holds_the_pool_at_the_bound).
• No public API change for cancellation. The caller's FnMut() -> bool stays on the coordinator; workers poll an AtomicBool every 4096 records.
• Bound vs. observed peak. materialized_records_bound(workers, partitions, peak_partition_records) names the order-independent reservation figure. It is test-only for now: no production budget governs materialized partition records (max_run_records bounds staged runs), so there is no admission gate to hand it to yet.

Default is PARTITION_LOAD_WORKERS = 2 (one partition loading and sorting while the coordinator writes the previous one). Row partitions are unchanged.

Proof

• Schedule-independence: fixed_partition_finish_is_schedule_independent_across_worker_counts routes 4,096 keys into 16 partitions under forced worker counts 1, 2, 3, 5 and asserts equal output SHA-256, equal relabelled evidence (including the in-memory transition log and storage_transient_peak_total_allocated_bytes), and that replaying the ordered ledger reaches the recorded exact peak with ≥ 2 identities coexisting.
• Mutation: an unbounded window fails 2 tests; dropping peak_partition_records from the merge fails 2; a worker forgetting its read counters fails the schedule-independence test.
• Suites (run separately, non-zero counts): partition_load::tests 9, tests::determinism 8, tests::lifecycle_budget 22, partition::tests 12, recovery::tests 5, tests:: 57, shape::tests 10; unfiltered graphforge-storage 1169 passed / 0 failed / 2 ignored.
• cargo test -p graphforge-api --test bdd: TCK 3897 of 3897.
• cargo clippy --workspace -- -D warnings clean; cargo fmt --all -- --check clean; make pre-push-fast passed.

No timings were taken: the host was running the S24–S26 ladder.

Related: #1456, #1429, #1448.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 078ac1a9-7d62-433a-8392-7d1a1e266e11

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}