observe(ingest): report per-region hashes, writes, and barriers

[DecisionNerd]

S22's phase labelled validation combined construction and sealing, while its stock snapshot omitted hashed bytes, write bytes and actual barrier counts. This made #735's digest-policy priority decision unverifiable.

Stock import receipts now emit graphforge-region-diagnostics/2, including process-wide SHA bytes/elapsed intervals, Linux write-syscall bytes, and attempted file/directory barriers. Worker work contributes to coordinator regions; inclusive and residual rows retain separate meanings, unavailable values remain null, and concurrent captures do not reset totals. The broad SHA changes are digest-preserving adapter substitutions required to observe actual calls. Digest computation and durability behavior remain unchanged.

The measured outcome/phase is stage+seal, with explicit source, node/edge append and manifest regions. Certification schemas and live operator consumers accept it. The documented compatibility consumers retain the CLI validate command, durable Validated state/binding labels, and historical /1 receipts.

S22 acceptance evidence and lossless raw bundle: 473.260492 s for 67,108,864 edges; every command tree reconciles exactly across all seven measurements. Authentication matches 1,088 chunks / 3,290,610,048 bytes with zero difference. Reopened counts and full node/edge scans match all four reference result digests. #735's adjudication moves #1617 to position 1 alongside #1621 and records it as a native #1387 prerequisite. The ingest floor remains unmet; no matched speedup or removable-hash claim is made.

Validation: full make pre-push passed at b8a1c7e (Rust/native, Python 157 tests, Node 118 scenarios/443 steps, all coverage thresholds). The final CI-only integration at 13480ca has an identical runtime/benchmark tree; formatting, make pre-push-fast and changed CI-policy checks passed, and exact-head CI Gate passed. Real tiny and SCALE6–8 growth certification passed on rebuilt stock binaries. Independent code/evidence review found no blockers after correcting the verification helper to read its declared Parquet format. CodeRabbit CLI authentication was unavailable; no CLI pass is claimed.

Closes #1623.
Related: #1632 (measurement-field outcome delivered; literal work-unit criteria remain open), #1617.

Note

Add per-region hash, write, and barrier metrics to ingest diagnostics

• Extends RegionMeasurement in capture.rs with written-byte, hashed-byte, hash-time, fsync-count, and fsync-duration counters, sampled from new process-wide observation wrappers: HashObservation/ObservedSha256 in hash_observation.rs and Observation/ObservedSync in observation.rs
• Snapshots now emit contract version 2 (graphforge-region-diagnostics/2), and the certification schema gains a version-2 branch; version-1 snapshots still validate under the legacy contract in region_diagnostics.rs
• Refactors SHA-256 call sites and file-synchronization call sites across the storage, API, and filesystem crates to route through ObservedSha256 and observed_sync_all; digests and I/O results are unchanged
• Splits import-session attribution into finer regions: stage+seal replaces validate, and new append_nodes, append_edges, source_read, and manifest_persistence regions are added via append_region_name in import_session.rs
• Risk: the validate receipt outcome is now rejected — sanitize_receipt, the tiny lifecycle certification validator, and the CLI tests now require stage+seal first; written-byte attribution only populates on Linux (process_written_bytes returns nothing elsewhere)

^{Macroscope summarized 13480ca.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: CurateLabs/graphforge/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 5a446fa4-d5b6-46e1-8e24-945b3a653dd6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[DecisionNerd]

For whoever is driving this PR: the only failure at the current head is Bazel Bootstrap, a job #1649 (471c16ff0) retired from CI. The Rust gate is now the "Rust Tests" nextest job. Updating the branch from main should drop the stale job and rerun CI Gate against the current workflow.

[DecisionNerd]

#1659 lands #1650, the hub_clone wall-time race found while validating this PR. It pushes the existing local fix/1650-deterministic-clone-stage-clock commits unchanged. Once it merges, updating this branch from main picks it up.