perf(storage): encode canonical topology and adjacency on admission lanes

[DecisionNerd]

Description

Canonical encoding was the largest remaining single-core construction region after #1599. This change compresses canonical Parquet batches and builds adjacency on admission lanes while preserving artifact bytes, receipt order, and coordinator-owned publication.

Closes #1600. Parent #1448 remains open: whole-ingest scaling still needs further work.

Changes

• Bound the Parquet compression pipeline by admission width and Arrow buffer budget; replay the original write chunks on the coordinator to preserve digests and I/O evidence.
• Pipeline projected adjacency decoding, sort directions concurrently, and build independent CSR files on leased lanes. Keep spill accounting ordered and publish the manifest last.
• Test one/max/reversed schedules, heterogeneous properties, spilled CSR, cancellation after artifact installation, worker joins, cleanup, and successful retry.
• Extend the certifier’s closed diagnostic-region list and schema for the encoding subregions, with a regression test against a real captured receipt.

Acceptance evidence

Measurement report includes all 21 quiet runs and reproducible summaries.

Criterion	Result
Existing encoded digests, 1/max/permuted lanes	Byte-identical to the #1599 merge tree
S18 canonical encoding CPU/wall, 8 usable cores	2.1086 median across three runs; target >= 2.0
S18 whole-ingest median, three alternating pairs	26.93 → 23.66 s (-12.1%)
S20 whole-ingest median, three alternating pairs	110.90 → 94.68 s (-14.6%)
Reopened counts and full scans, S18/S20	All baseline/candidate answer digests match
1/2/4/8/16-core curve	Recorded; 8-core whole-ingest throughput 1.39×, effective cores 1.19

Peak memory increased from 829 to 1,067 MiB at S18 and 2,783 to 2,901 MiB at S20; every measured run stayed below the 4,000 MB cap. Routing and append remain outside this change.

Validation

• cargo fmt --all -- --check
• cargo clippy --workspace -- -D warnings
• cargo test --workspace
• make pre-push-fast
• make pre-push
• Targeted determinism, encoding-lane, and adjacency tests; direct baseline/candidate digest comparison.
• All 42 certifier tests, including a demonstrated failing-then-passing receipt regression; production certifier Clippy.
• Real tiny lifecycle and all three ownership-growth lifecycles through the repaired certifier and measured engine binary.
• python3 docs/development/evidence/encoding-lanes-1600/summarize.py docs/development/evidence/encoding-lanes-1600/candidate

CodeRabbit CLI review was attempted but could not authenticate in this environment; it produced no review. Current code was manually reviewed. The required exact-head CI Gate and merge-queue run remain the merge gates.

The optional certifier --all-targets Clippy check found an existing cloned_ref_to_slice_refs lint at benchmarks/runners/certify/src/lib.rs:3172, verified unchanged on main. Production certifier Clippy and all certifier tests pass.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: CurateLabs/graphforge/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: c3653627-63c9-4060-8f79-a4633c0f7ba0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}