feat(storage): version payload checksums and retire legacy verification

[DecisionNerd]

Description

Published graph payloads previously required repeated SHA-256 reads to refuse same-inode, same-length corruption. Versioned graph/files formats now persist mandatory seed-zero XXH64 checksums and use them with exact length for payload admission. SHA-256 still names CAS objects and authenticates control metadata, receipts, and existing trust boundaries.

The maintainer's pre-v1 policy retires standalone gf verify and legacy graph formats. This removes compatibility readers, SHA-256 admission fallbacks, the publication checksum upgrade path, and the whole-store forensic scanner. Portable-package verification and internal construction rollback snapshots remain required.

Related Issues

Fixes #1637.

The canonical policy owner is #1617. Its broader digest inventory, opt-in diagnostics, identity caching, and complete import/query/export counter requirements remain open.

Changes

• Expanded inventories use versions 5/7 and compact roots 6/8 for raw/mapped routes. Payload checksums are required u64 values encoded as exactly 16 lowercase hex digits. Patricia branches retain version 3; buckets require version 4.
• Capture and installation calculate checksums alongside their existing streamed SHA-256 pass. Current-format payload admission checks length and checksum without payload SHA-256, preserving retained handles, path/link checks, filesystem admission, and publication barriers.
• Graph/files formats 1–4, legacy buckets, and persisted Arrow graph snapshots fail explicitly. Missing/null/malformed checksums, future versions, and descriptor/payload mismatches fail closed. Current expanded-to-compact and raw-to-mapped representation conversions remain supported.
• Remove the standalone CLI command, public verification facade/reports, and forensic scanner; update the public Rust surface ledger. CLI coverage rejects the retired command and preserves portable verify.
• Preserve topology and all-role corruption tests. Replace legacy compatibility tests with deterministic refusal tests, including unchanged CURRENT after rejected snapshot publication. Update fixtures to current checksums and versions.
• ADR 0049 and storage architecture notes record checksum versus authentication responsibilities and the pre-v1 retirement decision. This supersedes ADR 0045's standalone administrative verification requirement, preserving its trust-boundary policy.

Validation

• cargo clippy -p graphforge-storage -p graphforge-api -p graphforge-cli -- -D warnings: passed locally.
• cargo fmt --all -- --check, git diff --check, make pre-push-fast, ADR index and docs-tree checks: passed locally.
• python3 scripts/ci/test-non-cypher-surface-gate.py: 13 passed. The ledger gate passes with 453 public methods, 94 algorithms, and 22 search contracts.
• Rust unit compilation passed. The full local storage suite passed: 1,355 passed, zero failures, seven existing ignored tests. API corruption (four tests), snapshot refusal, retired CLI command rejection, and the current mapped-route guard passed. Commands and results are posted on #1637.
• Completed GitHub failure censuses found stale binding-parity metadata, the deleted-test migration ledger, and the explicit Bazel target/suite entry for the retired verify test. Those references are corrected. Live Cargo/Bazel parity inventory passes; a query of all test source labels finds 121 local sources and zero missing files. Every other lane passed. The final head incorporates current main, whose merged CI policy now runs the full workspace with Cargo/nextest. The final GitHub run passed at d046140e43c566e61635e2771353b54faea95d7c: full workspace nextest, custom-harness tests, doctests, real tiny/ownership-growth lifecycle execution, bindings, contract lanes, native durability, and CI Gate. CodeRabbit completed its GitHub review with no actionable comments; independent final review also found no remaining blocker. Main subsequently gained ADR 0048; its generated-index conflict is resolved by regeneration, retaining both records exactly once. Independent review confirms that the runtime files are identical to the reviewed/tested d046140e4 head. The refreshed 13cc28af27c616174e0ae830773eb3a4421392ee head passed the complete exact-head CI Gate and passed the merge-group CI run. The squash commit af93b16ff55d7b45cfa9ccd448f65864948226b8 is verified on main and feat(storage): version published graph payload checksums for read-time corruption refusal #1637 is closed with all acceptance criteria met.

Format policy

Pre-v1 projects using retired formats must be recreated. No legacy compatibility reader or in-place format migration is supplied. Checksums detect accidental corruption under the existing threat assumptions; they do not establish cryptographic identity. Ordinary payload admission still reads bytes for checksum validation. This PR does not claim zero SHA-256 across the entire facade/query path or change fsync policy.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: CurateLabs/graphforge/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 4d3a20a0-df63-4e13-86d1-c48a1c2c2b2d

📥 Commits

Reviewing files that changed from the base of the PR and between 471c16f and d046140.

⛔ Files ignored due to path filters (5)

• docs/adr/0049-published-payload-checksums.md is excluded by !**/*.md, !**/docs/**
• docs/adr/README.md is excluded by !**/*.md, !**/docs/**
• docs/book/architecture/storage.md is excluded by !**/*.md, !**/docs/**
• docs/development/bazel-migration.md is excluded by !**/*.md, !**/docs/**
• docs/engineering/adrs/README.md is excluded by !**/*.md, !**/docs/**

📒 Files selected for processing (53)

• crates/graphforge-api/src/lib.rs
• crates/graphforge-api/src/resumable_construction.rs
• crates/graphforge-api/src/workspace_hydration.rs
• crates/graphforge-api/src/workspace_hydration/tests.rs
• crates/graphforge-api/tests/file_backed_graph_generation.rs
• crates/graphforge-api/tests/file_backed_scale_evidence.rs
• crates/graphforge-api/tests/support/project_fixture.rs
• crates/graphforge-bindings-node/tests/non-cypher-parity-policy.json
• crates/graphforge-bindings-py/tests/non_cypher_release.py
• crates/graphforge-cli/BUILD.bazel
• crates/graphforge-cli/src/command_retirement_tests.rs
• crates/graphforge-cli/src/lib.rs
• crates/graphforge-cli/src/verify_cli.rs
• crates/graphforge-cli/tests/verify.rs
• crates/graphforge-storage/src/construction_detail_tests.rs
• crates/graphforge-storage/src/corruption_checksum.rs
• crates/graphforge-storage/src/graph_construction.rs
• crates/graphforge-storage/src/graph_construction/encoding_publication.rs
• crates/graphforge-storage/src/graph_delta_journal.rs
• crates/graphforge-storage/src/graph_files.rs
• crates/graphforge-storage/src/graph_manifest.rs
• crates/graphforge-storage/src/graph_object_store.rs
• crates/graphforge-storage/src/graph_object_store/installation.rs
• crates/graphforge-storage/src/graph_object_store/manifest_tree.rs
• crates/graphforge-storage/src/graph_object_store/manifest_tree/tests.rs
• crates/graphforge-storage/src/graph_object_store/materialization.rs
• crates/graphforge-storage/src/graph_object_store/materialization/tests.rs
• crates/graphforge-storage/src/graph_object_store/tests.rs
• crates/graphforge-storage/src/graph_projection.rs
• crates/graphforge-storage/src/lib.rs
• crates/graphforge-storage/src/payload_digest.rs
• crates/graphforge-storage/src/project_generation.rs
• crates/graphforge-storage/src/project_portable_v2_import.rs
• crates/graphforge-storage/src/project_portable_v2_import/adjacency.rs
• crates/graphforge-storage/src/project_publication/participants.rs
• crates/graphforge-storage/src/project_publication/tests.rs
• crates/graphforge-storage/src/property_overlay/inventory/tests.rs
• crates/graphforge-storage/src/property_overlay/projected_reads/tests.rs
• crates/graphforge-storage/src/property_overlay/targeted_reads/tests.rs
• crates/graphforge-storage/src/research_versions/projection.rs
• crates/graphforge-storage/src/research_versions/retained_content.rs
• crates/graphforge-storage/src/research_versions/tests.rs
• crates/graphforge-storage/src/route_component.rs
• crates/graphforge-storage/src/route_component/owned.rs
• crates/graphforge-storage/src/semantic_bindings/legacy_routes/tests.rs
• crates/graphforge-storage/src/storage_attribution.rs
• crates/graphforge-storage/src/verify.rs
• crates/graphforge-storage/tests/graph_delta_journal.rs
• docs-site/astro.config.mjs
• docs-site/scripts/sync-content.mjs
• scripts/ci/test-non-cypher-surface-gate.py
• tests/contracts/non-cypher-rust-surface.json
• tools/bazel/parity/migration_target_map.json

💤 Files with no reviewable changes (6)

• tools/bazel/parity/migration_target_map.json
• crates/graphforge-cli/tests/verify.rs
• crates/graphforge-api/src/lib.rs
• crates/graphforge-cli/BUILD.bazel
• crates/graphforge-cli/src/verify_cli.rs
• crates/graphforge-storage/src/verify.rs

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

---

Walkthrough

Graph payload inventory and compact manifest formats now persist XXH64 checksums. Capture, installation, and admission paths use these checksums with exact-length checks. Legacy graph snapshots and standalone project verification are removed.

Changes

Graph payload integrity

Layer / File(s)	Summary
Versioned checksum wire formats crates/graphforge-storage/src/corruption_checksum.rs, payload_digest.rs, graph_files.rs, graph_manifest.rs, lib.rs	Graph-file inventory and manifest formats add checksum-bearing versions and require an XXH64 value for each payload entry. The Serde adapter accepts only fixed-width lowercase hexadecimal strings.
Checksum capture, installation, and admission crates/graphforge-storage/src/graph_files.rs, graph_object_store*, payload_digest.rs, property_overlay/*, crates/graphforge-api/src/resumable_construction.rs	Capture and installation compute XXH64 alongside SHA-256. Graph-object and retained-file admission validate XXH64 and exact length. Tests cover checksum capture and corruption refusal.
Publication and import integration crates/graphforge-storage/src/graph_construction*, graph_delta_journal.rs, project_generation.rs, project_publication/*, project_portable_v2_import*, route_component*, research_versions/*, graph_projection.rs, crates/graphforge-api/tests/file_backed_scale_evidence.rs, docs-site/*	Graph publication, compaction, route handling, imports, and retained-content paths use the checksum-bearing versions. Compact manifests carry checksums from installation evidence.

Legacy snapshot rejection

Layer / File(s)

Summary

Reject legacy snapshots and validate file-backed generations crates/graphforge-storage/src/project_generation.rs, project_publication/participants.rs, crates/graphforge-storage/src/graph_files.rs, crates/graphforge-api/src/workspace_hydration.rs, crates/graphforge-api/src/workspace_hydration/tests.rs, crates/graphforge-api/tests/file_backed_graph_generation.rs, crates/graphforge-api/tests/support/project_fixture.rs

Staging, hydration, and rematerialization reject legacy graph snapshot participants. Regression tests verify that refusal leaves the current generation unchanged. Graph-generation fixtures publish file-backed graph data.

Standalone verification retirement

Layer / File(s)

Summary

Remove verification API and CLI command crates/graphforge-storage/src/verify.rs, crates/graphforge-storage/src/lib.rs, crates/graphforge-api/src/lib.rs, crates/graphforge-cli/src/lib.rs, crates/graphforge-cli/src/verify_cli.rs, crates/graphforge-cli/src/command_retirement_tests.rs, crates/graphforge-cli/tests/verify.rs, crates/graphforge-cli/BUILD.bazel, tools/bazel/parity/migration_target_map.json, tests/contracts/non-cypher-rust-surface.json, crates/graphforge-bindings-*/*, scripts/ci/test-non-cypher-surface-gate.py

The standalone verification operation, its API facade, and CLI command are removed. Tests and release-surface records are updated; portable verify remains parseable.

Priority: ⬇️ Low

Estimated code review effort: 4 (Complex) | ~50 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant GraphFilesInventory
  participant GraphObjectStore
  participant CompactManifest
  participant PayloadReader
  GraphFilesInventory->>GraphObjectStore: Install payload and capture SHA-256 and XXH64
  GraphObjectStore->>CompactManifest: Provide installed payload checksum
  CompactManifest->>PayloadReader: Supply checksum and exact length
  PayloadReader->>PayloadReader: Validate payload checksum and length

Loading

Merge Risk: ⚪ Minimal · up to d0461

The change introduces checksummed payload admission and intentionally retires legacy formats and standalone verification while retaining portable verification. No concrete blocker remains; merge after normal checks.

Security Architecture Review

Security architecture risk: 🔵 Low · up to d0461

The inspected paths retain SHA-256 checks for object installation and portable-package verification, while using XXH64 for subsequent corruption detection. No attacker-reachable authentication bypass was established. The main design risks are format compatibility and reliance on a trusted local object store; incomplete coverage prevents a minimal-risk assessment.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — The demonstrated integrity exposure concerns payload objects beneath the selected project’s CAS root and graph readers consuming those objects. Project-to-tenant or service mappings are unavailable, so a maximum production tenant or environment blast radius cannot be established.

Trust Boundaries and Controls

• inferred — XXH64 is not protection against deliberately forged payloads. The inspected design relies on SHA-256 checks at installation and package boundaries plus restricted local CAS ownership. An attacker with independent post-install payload-write authority would challenge that assumption, but such authority was not demonstrated for portable-package input.
• observed — Stable CAS directory access retains digest addressing and lifecycle/read locks. Retained construction readers continue checking regular-file status, exact length, read-only permissions and streamed SHA-256 rather than inheriting checksum-only admission.

Resilience and Maintainability Implications

• observed — Internal checkpoint-revert provenance remains journal-owned and bound to generation, manifest and restoration identity. Retirement of standalone verification does not remove this inspected rollback-control path.

Hardening Proposals

• proposed — Document the required CAS write-authority assumption explicitly: XXH64 admission detects corruption in trusted installed content, not malicious rewriting by a storage writer. This would help prevent future callers from treating the checksum as cryptographic authentication.

🚥 Pre-merge checks | ✅ 3 | ❌ 1 | ❓ 1

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 59.48% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 153 functions across 45 files. (2 skipped…	Write docstrings for the functions missing them to satisfy the coverage threshold.
Linked Issues check	❓ Inconclusive	The PR implements the main coding objectives in #1637. It adds versioned seed-zero XXH64 fields, computes XXH64 with the existing SHA-256 capture pass, admits current payloads by XXH64 and exact lengt…	Provide reviewable evidence for the required ADR and architecture-note content, or make those documents available to this assessment. The evidence must cover the format versions, checksum versus authentication responsibilities, trust assump…

✅ Passed checks (3 passed)

Check name	Status	Explanation
Out of Scope Changes check	✅ Passed	The changes remain connected to #1637. Storage changes implement checksum-bearing graph formats, admission, corruption refusal, and legacy-format retirement. CLI, facade, scanner, public-surface, pari…
Title check	✅ Passed	The title clearly summarizes the two primary changes: versioned payload checksums and retirement of legacy verification.
Description check	✅ Passed	The description is detailed and directly aligned with the pull request. It explains the checksum changes, retired formats and commands, compatibility policy, testing, validation results, and migration…

Full details: Linked Issues check

Explanation

The PR implements the main coding objectives in #1637. It adds versioned seed-zero XXH64 fields, computes XXH64 with the existing SHA-256 capture pass, admits current payloads by XXH64 and exact length, rejects legacy graph/snapshot formats, removes the verify command, facade, scanner, and public exports, and retains portable verification and trust-boundary checks. Tests cover checksum capture, admission corruption, malformed wire values, format rejection, and command retirement. The summary also reports the required CI and storage test results. However, the required ADR and architecture-note content is in excluded Markdown files: docs/adr/0049-published-payload-checksums.md, docs/adr/README.md, and docs/book/architecture/storage.md. The available changes show navigation and sync registration, but they do not establish that those documents contain the required format, authentication, trust, and pre-v1 retirement details.

Resolution

Provide reviewable evidence for the required ADR and architecture-note content, or make those documents available to this assessment. The evidence must cover the format versions, checksum versus authentication responsibilities, trust assumptions, and the pre-v1 retirement decision.

Full details: Docstring Coverage

Explanation

Docstring coverage is 59.48% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 153 functions across 45 files. (2 skipped: 2 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[DecisionNerd]

Heads-up from a parallel session: ADR number 0048 is also claimed by PR #1647 (docs/adr/0048-cargo-is-the-ci-build-authority.md, #1618). Whichever PR lands first keeps 0048; the other renumbers to 0049. Also noting, for whoever is driving this: CI at the current head fails Rust Quality on 4 clippy errors in graphforge-storage (too_many_lines ×2, &Option<T> → Option<&T>, an unnecessary !), and the branch is BEHIND main.

[DecisionNerd]

Independent integration review at 36f7906 reproduced the two current CI roots:

• python3 scripts/ci/check-binding-parity-policy.py fails at the Node release surface count after the Rust facade retirement. Update the Node count/digest and Python EXPECTED_RUST_DIGEST against the new authoritative Rust ledger, preserving the equality assertions and release-surface coverage.
• python3 scripts/ci/bazel-migration-ledger-check.py fails with map entry not in cargo metadata: graphforge-cli::verify. Remove the retired CLI test from tools/bazel/parity/migration_target_map.json, its BUILD rule, and the suite reference while Bazel still owns the gate.

The previous four Clippy errors are fixed at this head. The current-only checksum/retirement implementation passes the static acceptance review; remaining safe CI lanes are still running. There is also an integration collision: #1647 adds ADR 0048 for the build authority, so this PR must use the next available ADR number and regenerate references/indices after that prerequisite record lands.

[DecisionNerd]

Reviewed fa56222a8d268e5b5e509af4b97075e2a13d9eae: the binding projection check and migration ledger check now pass locally. The original Bazel finding is only partly fixed: crates/graphforge-cli/BUILD.bazel still declares gf_rust_integration_test(name = "verify", srcs = ["tests/verify.rs"]) at line 214 and includes ":verify" in its test suite at line 327, but that source is absent from this head. Remove both the retired target rule and suite entry, as well as the already-removed migration-map row, before the authoritative Bazel suite can run. The ledger and feature fingerprint checks do not detect this dangling BUILD source.

[DecisionNerd]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[DecisionNerd]

The dangling Bazel target is fixed in 26d23bcf765534e0d4f11d8df97407185be0d12c: both the verify rule and its suite entry are removed. Independent review found no remaining references to the deleted source/target/facade. Live parity inventory passed; querying labels(srcs, tests(//:ci_rust_tests)) found 121 local source labels and zero missing files.

The final head d046140e43c566e61635e2771353b54faea95d7c incorporates main's merged Cargo/nextest CI policy. Its complete CI Gate run passed, including 5,145 nextest tests, 118 BDD scenarios, doctests, tiny/ownership-growth CLI lifecycle execution, bindings, contracts, and native durability. CodeRabbit review is now running.

[DecisionNerd]

ADR #1647 has merged to main737df3dced71de25688f116447731c31241372af and #1648 is closed. This PR now has the expected ADR-index merge conflict. I am resolving only that integration in my isolated /home/ubuntu/gf-1640-codex-ledger checkout and will push an additive main merge; the author worktree is preserved. Prior exact-head CI/reviews passed. This is the next candidate being refreshed, while Binding RC qualification36655614470 has fully passed.

[DecisionNerd]

ADR 0048 is now taken on main by #1647 (737df3dce, docs/adr/0048-cargo-is-the-ci-build-authority.md). This PR's payload-checksum ADR needs to become 0049, including its rows in docs/adr/README.md and docs/engineering/adrs/README.md. Also, since #1649 (471c16ff0) the Rust gate is the "Rust Tests" nextest job, and BUILD.bazel edits are no longer checked in CI.

[DecisionNerd]

The author concurrently pushed the main integration as 13cc28af27c616174e0ae830773eb3a4421392ee. I independently verified its complete tree (d4785602e378c31892d42481bcf6cb4f5be62aea) is byte-identical to my locally reviewed/resolved merge; no duplicate correction is pushed. A second reviewer verifies all four generated files add only ADR0048 versus the old PR and only ADR0049 versus current main, preserving both ADR contents. ADR-index46active/3superseded, diff checks and make pre-push-fast passed against this exact tree. Final exact-head CI is running on13cc28af.