feat(storage): checkpoint durable deltas into immutable Parquet generations

[DecisionNerd]

Problem

An authoritative delta journal prevents small-write amplification but shifts the bound to replay cost and accumulated runs. The immutable model needs a crash-safe checkpoint operation that folds canonical Parquet plus typed deltas into canonical Parquet and then reclaims only unreachable inputs.

“Bounded” is not testable unless the implementation names its controlling budgets, default/maximum values, below/equal/above-bound ladders, cancellation cadence, and typed failure outcomes.

Objective

Implement deterministic, quantitatively bounded compaction as a normal project-generation transaction with safe retention, recovery, and benchmarkable pure merge/replay kernels.

Requirements

• Select one pinned complete generation containing a canonical Parquet base plus a verified contiguous typed-delta prefix.
• Stream/merge into canonical Parquet with explicit memory, spill, disk, input-run, input-byte, output-row, and cancellation-work budgets.
• Declare the configuration fields, defaults, supported maxima, and typed errors for every controlling budget before certification.
• Test each budget below, exactly at, and above its boundary using deterministic small and medium ladders; resource exhaustion must fail before uncontrolled allocation or partial publication.
• Publish the compacted graph as a complete immutable generation through CURRENT.
• Preserve reader leases, named checkpoints, prior retained generations, and concurrent later complete generations.
• Verify counts, identities, schemas, ordering, checksums, and canonical graph fingerprint before publication.
• Reclaim subsumed inputs only through the shared reachability/GC oracle after commit.
• Support preview/status and policy triggers by run count/bytes/replay work; avoid an unbounded background daemon.
• Make crash, cancellation, disk exhaustion, and retry idempotent.
• Expose deterministic merge/replay fixtures and counters for perf(storage): add CodSpeed durability and transaction regression baselines #782; keep correctness assertions outside timed closures.

Acceptance Criteria

• Canonical Parquet plus typed deltas and compacted Parquet produce identical canonical graph state and public query results.
• Peak memory, spill, disk, run count, input bytes, output work, and cancellation cadence are governed by named finite configuration and typed errors.
• Deterministic tests cover below/equal/above each declared budget and show no uncontrolled allocation or partial visibility.
• A crash before CURRENT leaves the old complete generation authoritative; a crash after acknowledgement leaves the compacted generation authoritative.
• Concurrent post-snapshot generations survive and remain visible after compaction.
• Checkpoints and pinned readers retain their exact prior bytes.
• Cleanup removes only inputs proven unreachable after the compacted generation commits.
• Progress/evidence reports input/output runs, rows, bytes, memory high-water mark, spill, elapsed time, and fingerprint.
• perf(storage): add CodSpeed durability and transaction regression baselines #782 can measure the same pure merge/replay kernels and end-to-end compaction path with correctly separated simulation, walltime, and memory evidence.

BDD Completion Scenarios

• Given a long valid typed-delta chain within declared budgets, when compaction commits, then reopen reads the same graph from canonical Parquet with a shorter replay chain.
• Given a budget exactly at its configured boundary, when compaction runs, then it completes within the declared envelope; one unit above returns the typed resource failure before publication.
• Given a reader pinned before compaction, when the new generation publishes and cleanup runs, then the reader continues to see its original snapshot.
• Given disk exhaustion or cancellation during compaction, when the project reopens, then the previous generation remains complete and authoritative.

Implementation Notes

Reuse project publication, resource policy, canonical-Parquet streaming, spill, the #752 typed replay layer, and the shared GC oracle. Derived index compaction must not be confused with authoritative graph compaction.

Observability

Input/output runs/rows/bytes, configured and observed memory/spill/disk/work budgets, phase, fingerprint, cleanup result, cancellation cadence, and structured resource errors.

Security And Privacy

Spill and staging paths remain contained/link-safe and are cleaned or quarantined deterministically. Budget diagnostics contain counts and sizes, never graph values or sensitive host paths.

Testing

Small semantic fixtures, below/equal/above resource ladders, bounded spill tests, concurrent generation barriers, checkpoint/lease tests, #749 crash histories, #776 admission, and deterministic benchmark fixtures for #782.

Documentation

Document configuration fields/defaults/maxima, resource errors, policy triggers, cancellation behavior, recovery, progress fields, and the distinction between CodSpeed diagnostics and correctness evidence.

Non-Goals

In-place Parquet mutation, autonomous unbounded vacuuming, distributed compaction, a universal hardware throughput claim, or using a benchmark to prove correctness.

Related Issues

Canonical tracker #747. Direct prerequisites #752 and #751. Filesystem admission #776. CodSpeed evidence #782.

[DecisionNerd]

Closed by squash merge of #771 (7da35250ce30e2f46cba8e68d8f1d2324e21700d). Acceptance criteria covered by graph_delta_compaction tests and ADR 0019 compaction section.

[DecisionNerd]

Reopened as part of the M6 closure recovery after an exact-tree audit of 6019f8a9b837b1b34615baad220fe6933c540b7d.

The current compactor does not emit canonical Parquet: encode_canonical_nodes writes newline text beginning GFNP into topology/nodes.parquet, while real graph readers use the Parquet reader. Its tests validate the .base_state.json sidecar instead of opening/querying the compacted Parquet. The implementation also materializes and clones the complete reconstructed graph in memory; its spill path serializes another copy but does not release the resident state, so it is not graph-size-independent streaming compaction.

Recovery exit:

• emit schema-correct canonical node, edge, and property Parquet through the real storage writers;
• prove base+deltas and compacted output through real GraphForge queries and reopen, not the sidecar model;
• stream/merge under a hard memory envelope with a fixture larger than that envelope and functional spill/reload;
• retain crash, checkpoint, lease, suffix-delta, fingerprint, and shared-GC evidence.

#753 continues to block #755, #756, and canonical tracker #747.