perf(storage,api): remove three per-record hot loops from validate (-33% user CPU)

[DecisionNerd]

Summary

Three per-record hot loops removed from import-session validate — −33 % user CPU at S18 with shaped-output digests byte-identical to main in every run.

main  27.3 user-s    A: 23.6    A+B: 20.9    A+B+C: 18.2   (-9.2 s, -33 %)

This attacks the constraint everything else routes around. validate is a single-threaded, CPU-bound pipeline at 0.85-0.89 cores — 132.2 CPU-seconds at S20 (S18 measured 26.56 user + 6.95 sys = 33.51, x3.945). Every parallelism attempt so far (#1429 at ~1.34x, #1448 measured at 1.46-1.58x against a 2.0 Amdahl ceiling) has been bounded by that floor. Removing work lowers it directly.

The three loops

A. write_bounded_row_groups computed each row's byte contribution with column.slice(i, 1).to_data().get_slice_memory_size() — four heap allocations per row per column, and 52M of the 104M allocations validate made at S18 (heaptrack). RowBytes computes the same number from the column's layout. Anything outside the enumerated fast paths still takes the exact arrow call, and a unit test pins the two equal for every fast path, with and without a null buffer.

B. DetailCodec::bytes scanned the 250 padding bytes of every in-memory 272/304-byte detail record on every routing, sort and encode pass, and read() re-ran it on the record it had just zeroed itself. wire() returns the length-bounded prefix; the UTF-8 and non-empty checks stay on read.

C. Edge normalization built three BTreeSet<Uuid> per batch (candidate endpoints, candidate edges, observed) plus one per node batch — 6.7 % of validate's CPU in BTreeMap::insert. Candidate sets are now sorted, deduplicated Vecs (the same order the BTreeSet iterated, so index probes are unchanged); membership sets are HashSets.

On B, since it removes a validation

Every producer of a padded detail record zero-initialises it, so the padding scan verified an invariant construction already guarantees. Verified at all five construction sites: graph_construction_encoding.rs:2853, intake.rs:456 and :477, shape.rs:978, uuid_membership/rebuild.rs:833 — each [0_u8; N], with read() filling only prefix + 1 + length bytes into a fresh buffer.

The wire path still validates. bytes() is unchanged and still does the full check on wire-bearing bytes; its is_err tests at construction_detail_codec.rs:202-208 still target it.

No coverage regression. A corrupted length byte that truncates a name was not caught by the padding check either — read() only ever wrote length bytes, so the padding stayed zero in that case too. The only thing the scan could catch is memory corruption between construction and use, which ADR 0013's threat model excludes and nothing else in the codebase defends against.

Provenance and what still needs checking

This branch was written by an agent that hit a session limit before reporting. The work was committed and measured; I preserved the branch rather than let it be lost with the worktree, and I did not re-run its gates locally — a timed S24-S26 ladder is running on the host and a local build would contaminate its wall times.

So CI is the verification here. Specifically watch:

• cargo test -p graphforge-api --test bdd — TCK 3,897
• the determinism suite, by self-consistency rather than fixed digests (ADR 0038, docs(adr): determinism belongs at the publication boundary (ADR 0038) #1457)
• permanent_storage_budgets and scale_g500_ladder
• cargo clippy --workspace -- -D warnings

If any of those fail this should not merge on a re-run without diagnosis. The measured numbers above are the agent's, taken at S18 on the #1440 head; they have not been reproduced at S20 on a quiet host and should be treated as measured-at-S18, projected elsewhere.

Part of #1456. Related #1441 (compact records, which B partly addresses), #1387.

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 7b8e0075-87a8-49da-ad6b-d0e53c20a994

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}