feat(storage): add generation-authenticated ordinal reverse identity authority

[DecisionNerd]

Problem

Issue #966 requires fixed-hop destination identity access whose memory and I/O remain bounded independently of graph cardinality. The current v3 UUID membership LSM cannot provide that guarantee:

• reverse surrogate records are packed and authenticated, but delta/tombstone runs are not ordinal-addressable;
• fence-selected block probes can reread hash-scattered blocks once a bounded cache evicts them, recreating chunks × index I/O;
• per-identity binary probes reduce bytes but create millions of random reads at Graph500 scale;
• loading the complete mapping into RAM violates GraphForge's disk-bound objective.

This is a storage-format prerequisite for projection-aware ExpandExec, not an S20 harness exception.

Objective

Publish a v4 generation-authenticated, ordinal-addressable node_id -> node_uuid/live-state authority whose storage is proportional to live plus retained identities, whose construction and compaction are sequential and bounded, and whose projected reads remain linear constant-factor at SCALE26 without graph-sized RSS.

Debt / regime

• Debt type: architecture, data, test/proof, and observability
• Quality regime: A deterministic compute

Requirements

• Add an authenticated v4 reverse-identity snapshot/delta layout:
  • contiguous newly allocated surrogate ranges are ordinal-addressable;
  • deleted-surrogate overrides are explicit and newest-generation authoritative;
  • retained deltas and compaction preserve generation ordering without sizing files by an arbitrary maximum sparse surrogate;
  • every published file is authenticated by the generation manifest and revalidated on reopen.
• Resolve sorted/batched surrogate requests through sequential/coalesced reads. Do not perform one filesystem seek or one fixed-size block authentication per requested identity.
• Bound resident buffers independently of graph cardinality. File-backed data may scale with retained identities; anonymous graph-sized caches may not.
• Keep construction, delta publication, recovery, and compaction crash-consistent and deterministic.
• Provide an explicit v3 migration/rebuild path. Never interpret a v3 run as v4 authority or silently fall back to full topology materialization.
• Expose aggregate-only work evidence: requested/unique/found identities, runs/ranges/bytes read, sequential read calls, compaction bytes, peak buffered bytes, migration/rebuild disposition, and authentication failures. Never emit UUIDs or paths.

Acceptance Criteria

• A clean build, append, delete, reopen, recovery, and compaction cycle returns the exact newest live node_id -> node_uuid mapping.
• v3 state is rejected or deterministically rebuilt through an explicit migration path; interrupted migration reopens the prior valid generation or the complete v4 generation, never a mixture.
• Corruption, truncation, run substitution, generation mismatch, duplicate surrogate, and tombstone resurrection fail closed before serving identity data.
• 1x/2x/4x and over-memory-budget fixtures prove logical bytes and read calls grow within a documented linear constant factor of retained identity data plus requested output, not request count × index/block size.
• Peak anonymous buffering is capped by a documented fixed budget and does not grow materially with graph cardinality.
• Query consumers can reuse one authenticated generation handle across bounded chunks without reopening or reauthenticating the full index per chunk.
• Cargo/Bazel drift checks and authoritative changed-surface CI pass at the exact head.

BDD Completion Scenarios

Sequential projected lookup at scale

Given authenticated v4 reverse identity state larger than the in-memory budget
When a consumer resolves shuffled and repeated surrogate batches across a query
Then results match the canonical live identities
And aggregate bytes/read calls remain linear constant-factor without per-identity or per-chunk full-index I/O.

Delete, append, compact, and reopen

Given a base snapshot plus appended identities and retained tombstones
When runs compact and the project reopens at the committed generation
Then newest live values and deletions are preserved exactly
And no deleted identity is resurrected from an older run.

Interrupted publication or migration

Given a valid v3 or v4 generation
When migration/publication stops at each durable boundary
Then recovery selects one fully authenticated generation
And incomplete files are never admitted as authority.

Corruption fails closed

Given a committed v4 generation with one altered, truncated, substituted, or generation-mismatched artifact
When the reverse identity handle opens or reads it
Then the operation fails with typed aggregate diagnostics before returning graph identity data.

Implementation Notes

Likely surfaces: graphforge-storage UUID membership manifest/run format, graph construction publication, append/delete deltas, compaction, recovery receipts, reopen validation, bounded lookup metrics, and the storage authority consumed by #966. Prefer packed ordinal range files plus explicit sparse tombstone overrides over a file sized directly by untrusted maximum surrogate ID.

Observability

Add aggregate work counters for generation authentication, ranges selected, sequential read calls, logical bytes, peak buffers, compaction/migration bytes, and typed first failure. Do not log graph UUIDs, filesystem paths, or record contents.

Security And Privacy

Preserve authenticated child-file opens, no-follow handling, immutable generation binding, and fail-closed corruption behavior. Bounds-check every ordinal/range before offset arithmetic. Evidence remains sanitized and aggregate-only.

Testing

• Unit: range encoding/lookup, tombstones, offset bounds, duplicate rejection, aggregate counters.
• Property/adversarial: shuffled/repeated/missing ids, sparse high surrogate rejection, corruption/truncation/substitution, malformed manifests.
• Integration: build -> append -> delete -> reopen -> compact -> reopen; v3 migration and every durable recovery boundary.
• Scale: deterministic 1x/2x/4x and over-budget call/logical-byte/RSS regressions with a fixed request shape.
• BDD mapping: all four scenarios require automated Rust storage integration coverage; crash boundaries use existing subprocess failpoints.

Documentation

Update storage architecture, UUID authority format/migration documentation, recovery invariants, and #966 execution guidance.

Non-Goals

• Projection planning or relationship/node Parquet column pruning owned by fix(exec): make fixed-hop expansion projection-aware #966.
• Changing the certification query, increasing RAM, or increasing timeouts.
• A ladder-only cache, unauthenticated sidecar, graph-sized anonymous map, or per-record random-read workaround.

Related Issues

• Parent consumer blocked by this work: fix(exec): make fixed-hop expansion projection-aware #966
• Canonical fixed-hop scale gate: fix(exec): bound reopen and fixed-hop LIMIT query RSS at scale #904
• Canonical Fly S20 gate: test(scale): complete the S18-S26 lifecycle ladder on OVHC-AGENCY #900