epic(storage): reduce lifecycle amplification and improve persistent storage/query efficiency

[DecisionNerd]

Closed 2026-09-23 — all epic acceptance criteria satisfied

AC revision (maintainer, 2026-09-23): Terminal named-host capacity proof and
S26 certification are not acceptance criteria for this epic. They are owned
by #900 and
#745. The former
#900/#745 checklist item was removed from Acceptance criteria and recorded
under Non-goals. Every remaining epic criterion is checked.

This epic is closed on its full acceptance-criteria scope. All 51 native
children are closed — the final three in this closeout series: #1473 (rung
peak RSS attributed to page cache, PR #1550), #1384 (ingest authentication
regime rewritten; ADR 0045 records hash-on-write, the SHA-256 naming role, the
XXH64 corruption role and the surviving-boundary ledger, PR #1555), and #1393
(transient peak composed on the integrated tree, PR #1561).

The measured slope this epic leaves for downstream ladder work: 449.4 B/edge
transient peak, flat across S18–S22 (retained archives ed273d2b… and
b6ffb088…), so the S26 projection is 482.5 GB against the recorded 598.0
GB envelope — a 115.5 GB (23.9%) admission margin, up from 9.6 GB (1.6%)
when capacity was first gated on this epic. Ingest application reads at S22
fell 310.1 GB → 68.5 GB with authentication read-backs at 74 B/edge, every
surviving pass named and justified.

Dependency cleanup: #1194 no longer blocks #745. #745 is blocked only by
#900. Ladder execution and S26 certification proceed on those issues without
waiting on this epic.

---

Current capacity after approved full-cache cleanup — 2026-09-14

Complete cleanup observation and native qualification: audited Cargo/coverage/Bazel cache cleanup recovered 43,929,825,280 available bytes while preserving source work, Git history, environments, binaries and measurement evidence. Verification retains the explicit disposition of four regenerated compiler-probe metadata files; all preserved binary/evidence hashes matched.

Current S20/S22 planning decision: admit. Available capacity 777,694,818,304 B; projected demand 588,524,115,286 B; unchanged reserve 141,258,578,535 B; 47,912,124,483 B spare after reserve, zero planning deficit. This supersedes the earlier 3.98 GB margin below. Future builds or other writes can consume this point-in-time headroom.

Remaining close gate: actual final ladder/capacity evidence. No S24/S25/S26 was executed or authorized by this cleanup. The existing ladder’s adjacent admission and final S26 public lifecycle/count/resource requirements remain unchanged.

Current capacity after XYG cache cleanup — 2026-09-14

Verified cleanup and complete native qualification: XYG’s unused dev-profile build cache returned 2,674,348,032 available bytes, preserving both checkouts, dependencies and runtime assets. Native backend and all 647 ABI smoke checks pass.

Current S20/S22 planning decision: admit. Available capacity 733,767,651,328 B; projected demand 588,524,115,286 B; unchanged reserve 141,258,578,535 B; 3,984,957,507 B spare after reserve, zero planning deficit. This supersedes the earlier 1.31 GB margin below; filesystem availability remains point-in-time.

The only remaining close gate is actual final ladder/capacity evidence. No S24/S25/S26 ran; this projection does not authorize or replace their existing admission and execution requirements.

Current capacity after approved cache reclamation — 2026-09-14

The complete reclamation observation and native qualification supersede the earlier 30.10 GB planning deficit below. Audited cleanup recovered 31,305,781,248 available bytes, preserving original executable paths/hashes, frozen binaries, raw evidence and unrelated work.

The existing S20/S22 projection now returns admit: projected demand 588,524,115,286 B, actual available capacity 731,095,973,888 B, unchanged reserve 141,258,578,535 B; 1,313,280,067 B spare after reserve and zero current planning deficit. This is a small, point-in-time margin that subsequent builds/writes can consume.

Remaining acceptance: actual final ladder/capacity evidence. No S24/S25/S26 was executed or authorized by this cleanup. The existing ladder still requires its admitted S24/S25 observations and terminal S26 public lifecycle/count/resource evidence. Lower-rung projection alone does not close #1194/#900/#745. Earlier dated reports remain preserved as historical observations.

Verified capacity follow-up — 2026-09-14

The merged integrated report supersedes earlier capacity estimates as the current evidence ledger. #1269/#1268/#1272/#1274 are verified closed after their focused PRs passed exact-head CI Gate and squash-merged. #1276 records the integrated evidence merge 989579078cc7e71ca4deaa1464afbbed7d222222 and required CI.

On the same OVHC-AGENCY host, merged source 710c6c64f4718c0664d08bdd3aafe21de4a29eba and frozen executables completed only S18 → S19 → S20 → S22, with native admission before each successor and all ten ordinary lifecycle phases passing. Accepted rung workspaces were reclaimed before the next rung. S22 contains 4,194,304 nodes and 67,108,864 live edges. Compared with the preserved 24ca688c run, lifecycle peak falls 47,657,119,744 → 36,760,567,808 bytes. The report preserves storage owners without double-counting, logical and process I/O, CPU, process VmHWM, cgroup/page-cache observations and disjoint construction timings.

Capacity acceptance	Current outcome
Material implementation and deterministic representative budgets	Merged consumed-root retirement, authenticated recovery cleanup and exact packed current-format records; measured storage and I/O improvements, CPU/RSS envelopes and public lifecycle/corruption/recovery tests retained.
Authorized lower-rung integration	S18/S19/S20/S22 passed with unchanged input/host profiles and reserve; native receipts and historical evidence retained.
Product demand projection	S26 projected peak 762,866,827,264 → 588,524,115,286 B; reduction 174,342,711,978 B. At the old available-capacity sample, deficit is 96,428,597,181 B.
Owned artifact reclamation	Actual available-byte increase 75,087,810,560 B, separately measured after freezing. Unrelated work, measured executables and raw evidence preserved.
Current native capacity projection	refuse: projected peak 588,524,115,286 B, actual available capacity 699,686,567,936 B, unchanged admission reserve 141,258,578,535 B. Deficit 30,096,125,885 B; spare after reserve 0 B.
Final capacity/S26	OPEN. No S24/S25/S26 was authorized or executed. The existing ladder still requires S24/S25 admission and actual terminal S26 lifecycle/count/resource evidence; a S20/S22 projection cannot replace it.

All authorized implementation and lower-rung evidence work is complete. #1194/#900/#745 remain open only for the final named-host capacity/S26 outcome in their existing acceptance criteria. No larger-host substitution, reduced reserve, admission waiver or new certification workflow was used. Historical dated notes below remain preserved and are superseded where their capacity numbers differ.

Maintainer scope: before v1.0.0

Backward compatibility is out of scope for this issue and its repair sub-issues before the v1.0.0 release. Do not add or retain legacy readers, old API aliases, compatibility shims, mixed-version support, migration/backfill machinery, or old-version regression tests solely to preserve behavior or data from earlier GraphForge versions. Earlier backward-compatibility requirements in this issue are superseded by this maintainer instruction.

This does not authorize unrelated breaking changes or weaken the issue's current-version acceptance criteria. Exact current-format semantics, supported current-version API/binding and export/import interoperability, authentication, corruption/unsupported-format refusal, crash recovery, retry/idempotency, active snapshots, cancellation and resource budgets remain required where applicable. Never silently reinterpret unsupported old data. Document intentional format/API breaks and the supported current format; a migration implementation is not required. Current-version correctness tests and explicitly behavior-preserving refactors remain in scope. Compatibility guarantees for v1.0.0 and later are a separate release-policy decision.

Verified integrated closeout — 2026-09-11

PR #1260 squash-merged as 1cba0eebb58bd94ed42804bc8d4896b3eaeec4e4 after the required exact-head CI Gate passed (CI run). The integrated acceptance ledger maps the criteria below to merged implementation, direct public tests and source-bound resource evidence. Earlier dated progress notes and open-status statements below are historical and superseded by this closeout.

All authorized implementation is merged. #1257 fixes same-facade construction readers and preserves retained streams, cancellation and failed-publication retry; #1256 supplies disjoint append/seal/resume/publication wall-time receipts. Their required exact-head CI passed before merge, and both issues are verified closed. All native implementation prerequisites are closed.

The existing OVHC-AGENCY S18/S19/S20/S22 ladder passed all ten ordinary lifecycle phases on merged source 24ca688c516a86a68de9cffaab2d5a9215291256 with frozen executables and unchanged input/host profiles. S22 contains 4,194,304 live nodes and 67,108,864 live edges. Retained allocation is 73,638,645,760→17,813,295,104 bytes; actual simultaneous lifecycle peak is 81,197,961,216→47,657,119,744 bytes. Whole-run CPU is 1551.682 seconds, process VmHWM 263,090,176 bytes and cgroup peak 12,474,519,552 bytes. These are different memory authorities; sampled observations are not hard bounds. The report includes normalized storage, overlapping owners, phase I/O/timings, deterministic representative budgets and the remaining shaping/identity/recovery costs.

The existing S20/S22 capacity projection returns refuse: 762,866,827,264 projected bytes against 633,354,096,640 available bytes, with 141,258,578,535 bytes reserved. This is not S26 execution. #901's construction and authorized lower-rung outcomes are complete; #1194/#900/#745 retain the genuinely outstanding final named-host capacity/S26 outcome. No larger-host substitution, admission waiver, S24/S26 run or new certification workflow was used.

All implementation, publishing-contract, opportunity-disposition and integrated-report checklist items below are complete. (Historical note: an earlier draft listed #900/#745 named-host capacity as an epic AC; the maintainer AC revision above removed it.)

Purpose

Aggressively reduce GraphForge's measured lifecycle storage amplification and permanent graph storage by eliminating avoidable representations, coexistence and encoding costs while preserving exact semantics and recovery guarantees. This M5 epic coordinates storage efficiency and evidence-backed query access improvements; it does not replace the existing construction issue or introduce another scale-certification run.

Debt: architecture / data / test-proof. Quality regime: deterministic compute; correctness and recovery take precedence over space savings.

Evidence and problem

The shared S22 ladder at merged 3868e3c751ba0c94928033378d2ddc2b5401cd55 contains 4,194,304 live nodes and 67,108,864 live edges, with no user-property payload:

Measurement	Allocated bytes	Approximate bytes/live edge
One published project	7,578,365,952	113
Retained construction owner alone	44,344,586,240	661
Entire retained lifecycle workspace	73,638,645,760	1,097
Whole-lifecycle peak	81,197,961,216	1,210

The retained workspace is approximately 9.7 times one published project; construction accounts for approximately 60% of that workspace. The canonical edge representation contributes approximately 73 bytes/edge, and UUID/surrogate indexes approximately 38 bytes/edge. Component inventories and lifecycle-owner inventories overlap: never sum both as independent allocations.

The native S20/S22 projection refuses S26 at 1,303,619,917,142 bytes. These are historical measurements, not a current-main baseline or proof that construction bytes are safely reclaimable. Evidence: 900-3868e3c7-evidence/s20-s22-storage-qualification.json on OVHC-AGENCY, recorded in #901. The large workspace has been cleaned; preserve the evidence and use the existing shared ladder for subsequent qualification.

External record-size examples motivate investigation, not a claim that another database is a measured factor smaller. Any comparison must match identities, properties, indexes, durability and lifecycle copies.

Workstreams and ownership

1. Lifecycle amplification — existing feat(storage): stage topology append-only with linear ingest I/O #901. Reuse its construction/retention repair scope and acceptance evidence. Establish per-artifact allocations and consumer lifetimes across append/merge, shape, encode, publication and clean import. Reduce unnecessary coexistence and retained intermediates through authenticated successor ownership and safe retirement. Separate steady retained savings from actual historical-peak reduction. Inspect input/import, export and clean-import coexistence as well; create a bounded child only for a verified distinct cause outside feat(storage): stage topology append-only with linear ingest I/O #901.
2. Permanent representation — this epic's additional scope. Audit canonical topology, UUID membership/surrogate/reverse indexes, adjacency and metadata for redundant identities, excessive widths, duplicated authority, compression opportunities and required access costs. Use representative property-free and property-bearing fixtures, including varied identifier/route cardinality and multi-level merges. Quantify candidate savings and read/write/RSS tradeoffs before selecting repairs. Preserve exact external UUIDs and the distinction between runtime catalog IDs and ontology IDs.
3. Integrated evidence. Reconcile storage budgets and projection on the final integrated tree using the same frozen ladder, host, admission, reserve and first-failure rules. Terminal ladder execution and S26 certification are test(scale): complete the S18-S26 lifecycle ladder on OVHC-AGENCY #900/test(scale): certify the S26 billion-edge round trip on OVHC-AGENCY #745, not epic close gates.

Optimization mandate

Maintainer direction: pursue substantial, evidence-backed disk optimization across both lifecycle and permanent representation. Safe terminal cleanup or an assessment report alone does not complete this epic. Meeting the capacity envelope is required, but does not waive validated permanent-storage opportunities.

Rank material costs by measured allocated bytes and contribution to the actual simultaneous lifecycle peak. Challenge canonical topology, repeated UUID/identity storage, membership/surrogate/reverse indexes, adjacency, metadata, field widths, encoding/compression and input/export/import coexistence. Similar-looking bytes are not proof of redundancy; use source-backed ownership/access analysis and reproducible experiments.

For every material candidate, record baseline bytes, attainable savings, affected phases, implementation scope, current-format obligations and measured or explicitly estimated CPU/I/O/RSS/query tradeoffs. Implement candidates validated to deliver meaningful savings within the existing contracts and resource envelope through focused blocking issues. Reject a candidate only with a specific quantified disposition explaining why savings are immaterial or would violate those contracts or budgets. Implementation inconvenience, passing the capacity gate, or finishing the assessment is not sufficient justification for deferral. Any scope reduction for a validated material opportunity requires an explicit maintainer disposition.

Avoid speculative rewrites and arbitrary percentage promises. Aggressiveness is demonstrated by measured reductions and resolving the material opportunities, not by code churn or weakening correctness.

Query and storage efficiency extension (maintainer-authorized)

Extend this epic to assess and implement practical, material improvements in how GraphForge uses its existing Arrow/Parquet/DataFusion stack. These opportunities belong here because physical layout, indexes and query access determine both permanent storage costs and the I/O/CPU/RSS paid to use that storage. This is a bounded five-candidate assessment, not an open-ended query-engine rewrite or a mandate to enable every available feature.

Candidate	Required investigation	Selection constraint
Filter/projection pushdown and late materialization	Trace representative public queries through GraphForge's custom scans; measure bytes/rows decoded versus returned and when properties are hydrated. Assess page pruning and runtime filters where applicable.	Show an actual gap beyond existing projection, row selection and min/max pruning. Preserve residual filters, nulls, overlays/tombstones and LIMIT/ordering semantics.
Optimizer statistics	Assess cardinality, distinct-count, selectivity and ordering information supplied to the pinned DataFusion optimizer and GraphForge traversal planning; test selective starts, joins and skew.	Statistics must improve an actual supported plan and measured execution. Distinguish estimates from exact facts; stale or absent metadata cannot change answers. Do not assume DataFusion chooses graph traversal starts automatically.
Workload-aware Parquet layout	Compare bounded row-group/page sizes, useful clustering/sorting and per-column encodings using property-free/property-bearing and selective/scan workloads.	Credit gains beyond the existing compression repairs; account for write CPU, sorting/spill, point-read amplification and format/recovery obligations. No global tuning based only on sequential UUIDs.
Selective Bloom filters	Evaluate equality/negative lookups that are not already efficiently answered by existing identity/index authorities; quantify avoided reads and filter overhead.	Add filters only for demonstrated beneficiaries. Include false-positive behavior, absent filters, build/maintenance cost and metadata storage; never allow false-negative pruning.
Bounded fragmentation compaction	Measure repeated append/update/delete workloads for excess fragments, metadata and read amplification after the existing manifest-bucket repair; compare targeted compaction with current maintenance.	Require a material residual problem and bounded rewrite/I/O/temporary-disk/RSS costs. Preserve immutable snapshots, leases, cancellation, recovery and atomic publication; avoid duplicate compaction machinery.

Current-source leads are hypotheses, not verified defects: parquet_scan.rs supplies row counts but unknown column statistics; property_scan.rs correctly distinguishes overlay row upper bounds from exact counts; catalog.rs already implements projection, dense node row selection and conservative row-group pruning. Trace reachable public paths before proposing changes. DataFusion configuration switches alone do not prove GraphForge's custom readers use a feature. Verify behavior against the repository's pinned dependencies and official upstream documentation, including Parquet pruning and configuration semantics.

Assessment and implementation gate

• Freeze the integrated source/toolchain and representative public workloads: selective equality/range predicates, property filtering, joins or supported traversal starts, LIMIT/order cases, scans/aggregations, skewed degrees/selectivity, and repeated updates/deletes. Use deterministic synthetic data with exact UUID/value/schema/null/duplicate semantics and relevant current-format overlay cases. Reuse existing fixtures, counters and harnesses.
• For each of the five candidates, identify existing support, the reachable missing capability (if any), a reproducible baseline and experiment, and an explicit decision: already effective/no change, validated repair, or quantified rejection. “Best practice” or feature availability alone is not evidence of benefit.
• Define practical/material selection budgets from the baseline before accepting a repair: absolute and normalized bytes read/written, rows/pages decoded, allocations/temporary disk, bounded memory, and repeated cold/warm execution measurements as relevant. Report CPU/latency variance and maintenance costs; use deterministic resource assertions rather than noisy timing-only CI gates. No universal percentage target or gains hidden by another regression.
• Implement and merge every validated material candidate that fits the existing contracts/resource envelope through a focused native child that blocks this epic, or record an explicit maintainer disposition. An assessment-only close does not satisfy the validated-repair gate. Preserve all exact-head CI, independent review, recovery and current-format interoperability requirements already stated in this epic.
• Document before/after public behavior and resource evidence, including why each non-selected candidate is already covered, immaterial or impractical. Keep storage savings separate from query-performance gains and avoid double-counting overlap with test(storage): establish permanent topology and identity compaction budgets #1196 and its production repairs.

BDD proof: Given the same supported graph and public query, when a selected scan/planning/layout/index improvement executes, then exact results (including required order, duplicates and nulls) remain unchanged while the declared resource budget improves. Given stale/missing optional optimization metadata, active snapshots or interrupted maintenance, then existing correctness/recovery contracts hold. For unchanged candidates, provide source and execution evidence supporting that disposition.

Sequence this assessment after the existing storage assessment and relevant repair baselines; do not interrupt or duplicate in-flight work. Existing overlapping issues remain owners. Create additional native blocking children only after a bounded gap is verified; respect the three-change WIP limit. This extension changes implementation scope, not the shared #900/#745 qualification process: use the existing admitted lower-rung ladder for integrated projections, and leave terminal S26 certification to #900/#745 (not epic acceptance criteria). It authorizes no final S26 execution, unrelated M3 work, distributed execution, new cache service or replacement storage/query engine.

Acceptance criteria

• The five-candidate query/storage efficiency extension above has a complete evidence-based disposition ledger, and all selected material repairs are merged with direct public-behavior and resource-budget proof.

• fix(storage): simplify construction lifecycle and safely reclaim superseded artifacts #1195 begins with a lifecycle simplification review mapping representations, ownership, consumers, authentication boundaries and documented recovery/resume guarantees. Before/after diagrams identify removable dependencies and explicit supersession boundaries. Prefer fewer redundant representations and recovery states; justify any new journal, receipt or state machine and prove preserved guarantees with direct tests. Adding retirement machinery is not itself the required outcome.

• A fresh, source-identified baseline separates unique physical allocation, logical bytes, retained bytes and simultaneous per-phase peaks, with bytes/node, bytes/edge and amplification ratios. It reconciles shared/CAS objects once and identifies required versus superseded owners.

• feat(storage): stage topology append-only with linear ingest I/O #901 closes with its existing construction and host evidence. Verified additional lifecycle defects are fixed through focused issues; cleanup alone is not accepted as a reduction in an earlier peak.

• Permanent topology and identity/index storage receive a documented, reproducible assessment. Each selected opportunity has a measured before/after budget and regression oracle; implement validated improvements or record a specific evidence-based disposition where savings would violate contracts or yield no meaningful benefit. No arbitrary percentage target or assumed competitor advantage substitutes for this assessment.

• Before/after integrated evidence demonstrates reduced actual simultaneous lifecycle peak as well as reduced retained storage. Phase-boundary cleanup is credited only for later coexistence it actually eliminates; historical peaks remain intact.

• The permanent-storage assessment covers every material category and produces a ranked opportunity ledger. Every validated material opportunity is implemented and merged through a focused blocking issue, or has a quantified contract/resource-based rejection or explicit maintainer disposition. Closing test(storage): establish permanent topology and identity compaction budgets #1196 on assessment/tests does not satisfy this implementation gate.

• Final evidence reports absolute and normalized before/after permanent, retained and peak allocated bytes, the contribution of each landed repair without double counting, resource tradeoffs and the largest remaining costs. Capacity admission alone does not waive this accounting or the opportunity-resolution gate.

• Deterministic payload-dominated 1x/2x/4x fixtures enforce justified retained/peak and per-phase I/O ceilings, including merge-level boundaries. Public construction, reopen, queries, export, verify and clean import preserve exact counts, values, schema and fingerprints.

• Retirement and any format change preserve crash/returned-error retry, idempotent replay, active snapshot/stream leases, cancellation, corruption refusal and atomic publication. Missing or corrupt successor authority fails closed. Allocation removal is recorded only after verified removal.

• Focused implementation issues are merged with exact-head required CI and CI Gate; storage-format/recovery documentation and sanitized evidence explain measured gains and remaining costs.

BDD completion scenarios and proof

• Safe retirement: Given authenticated successor data and superseded session artifacts, when retirement is interrupted and retried, then ordinary resume/reopen returns the same graph, preserves prior authority on failure and removes only owned, no-longer-required artifacts. Prove through Rust-facade crash and returned-error injection tests, including replay after CURRENT advances.
• Compact representation: Given equivalent supported graphs with boundary identifiers and heterogeneous properties, when a selected representation change is written and reopened/exported/imported, then exact semantics and current-format interoperability hold and measured storage improves within its declared resource budget. Prove through physical-allocation and public lifecycle tests, with affected binding parity.
• Honest peak accounting: Given overlapping source, staging, package and imported-project lifetimes, when the lifecycle runs, then ownership reconciliation counts shared files once and records the actual simultaneous maximum. Prove with deterministic allocation/growth tests and existing host evidence.
• Capacity admission (epic evidence): Given the integrated lower-rung ladder evidence this epic produced, when the existing qualifier evaluates admission from that projection, then it records admit/refuse against available capacity and the configured reserve without authorizing a larger run. Prove via the reconciled integrated report and retained ladder archives; terminal S26 certification remains test(scale): complete the S18-S26 lifecycle ladder on OVHC-AGENCY #900/test(scale): certify the S26 billion-edge round trip on OVHC-AGENCY #745 scope.

Implementation and documentation

Likely surfaces: Rust storage construction/encoding/publication, UUID and ordinal indexes, portable import/export cleanup, allocation accounting, and the public resumable-construction facade. Graph data remains Parquet, metadata JSON, and bindings remain thin. Format changes require explicit current-format identification and recovery design, not silent reinterpretation. Backward readers and migration are out of scope before v1.0.0.

Use existing content-free counters for per-owner/current/peak bytes, reads/writes and synchronization. Preserve no-follow identity checks, immutable/shared-object ownership and leases; do not log graph values, UUIDs, secrets or unrestricted paths. Update storage architecture, current-format/breaking-change, recovery/retention and scale-evidence documentation.

Relationships and sequencing

• Parent and M5 close gate: epic(scale): close billion-edge scale and hardened interchange #735.
• Existing construction owner and completion prerequisite: feat(storage): stage topology append-only with linear ingest I/O #901, still under test(scale): complete the S18-S26 lifecycle ladder on OVHC-AGENCY #900; do not reparent or duplicate it.
• Shared execution/certification (out of epic AC): test(scale): complete the S18-S26 lifecycle ladder on OVHC-AGENCY #900 and test(scale): certify the S26 billion-edge round trip on OVHC-AGENCY #745 own the S18–S26 ladder and terminal S26 proof. This epic does not block them; test(scale): certify the S26 billion-edge round trip on OVHC-AGENCY #745 is blocked only by test(scale): complete the S18-S26 lifecycle ladder on OVHC-AGENCY #900.
• Completed foundations: test(scale): attribute physical bytes per node and edge before SCALE26 #951 (accounting), fix(storage): compact authenticated construction detail records #1172 (construction codec), feat(storage): publish compact authenticated graph roots #930 (roots), feat(storage): add generation-authenticated ordinal reverse identity authority #967-feat(storage): define authenticated v4 ordinal identity reads #970 (ordinal identity), feat(storage): expose bounded snapshot retention and orphan garbage collection #751 (retention/GC).
• Coordinate format-sensitive work with fix(storage): encode semantic property routes into portable filenames #976 and portable publication repair fix(storage): prevent Windows portable publication from replacing destinations #1190; finish prerequisites before dependent implementation.

Create native, non-overlapping child issues only for verified bounded repairs or independently reviewable XL concerns; each blocks this epic. Respect the three-change work-in-progress limit and preserve ongoing work.

Non-goals

Terminal S26 billion-edge certification and final named-host capacity proof beyond this epic's integrated lower-rung evidence (#900, #745). A replacement certification tracker, speculative codec rewrites, distributed storage, larger-machine workarounds, changed Graph500 policy, removing required indexes or identities to win a size comparison, weakened authentication/durability, benchmark-only cleanup, or declaring all retained bytes reclaimable.

Focused issue ledger

Snapshot refreshed 2026-09-10; native parent/sub-issue and blocked-by relationships remain authoritative. Closed means the issue is closed, not that integrated efficiency has been established.

Issue	Owned outcome	State at refresh	Recorded prerequisites
#901	feat(storage): stage topology append-only with linear ingest I/O	closed	#930, #931, #932, #951, #971, #979, #1158, #1172, #1195
#1195	fix(storage): simplify construction lifecycle and safely reclaim superseded artifacts	closed	#976
#1196	test(storage): establish permanent topology and identity compaction budgets	closed	#976
#1202	fix(storage): compress permanent construction Parquet with Zstd	closed	#1196
#1203	fix(storage): pack permanent membership records without reserved padding	closed	#1196
#1204	fix(storage): bound manifest allocation with authenticated bucket leaves	closed	#1196
#1205	fix(storage): compress bounded CSR shards with authenticated decode limits	closed	#1196
#1207	test(perf): assess practical query and storage efficiency opportunities	closed	#1196, #1202, #1203, #1204, #1205, #1213
#1213	fix(storage): unify permanent encoding policy across all Parquet publishing paths	closed	#1202, #1216, #1218, #1219, #1222, #1224
#1216	fix(storage): replace authenticated shared payloads during durable mutation	closed	None recorded
#1218	fix(storage): replay valid exploratory edge fragments without losing route semantics	closed	#1219, #1222, #1224
#1219	fix(storage): resolve CAS-backed generations in delta publishing and compaction	closed	#1216
#1221	fix(storage): enforce topology replay boundaries and publishing contracts	closed	#1218, #1219, #1224
#1222	fix(api): preserve CAS graph ownership in workspace publication	closed	None recorded
#1224	fix(api): preserve property ownership in composite publishing routes	closed	None recorded
#1228	fix(storage): preserve authenticated UUID ownership in CAS mutation workspaces	closed	None recorded
#1229	fix(api): publish same-name ontology promotion atomically	closed	#1228
#1230	fix(api): reject ontology removal that strands semantic bindings	closed	None recorded
#1231	fix(api): refresh facade authority after delta compaction	closed	#1228

Every listed issue blocks this epic. #1195 remains under #901; preserve the #901 → #900 completion chain and all existing parents. Update this ledger after merges or verified blocker changes. Creating or recording an issue starts no implementation stream.

Current-format publishing contract — maintainer scope extension

The epic also owns consistent current-format semantics across permanent publishing paths. Encoding uniformity alone is insufficient: construction, mutation, replay, compaction, projections and other verified publishers must preserve their applicable ownership, schema, ordering, routing, identity/index and surrogate high-water-mark contracts. Separate writers and lifecycle implementations may remain where justified.

• Maintain a source-backed publishing/operation matrix, distinguishing permanent graph payloads from private staging, exported artifacts and fixtures. Every supported publisher must obey the shared current-format contract and fix(storage): unify permanent encoding policy across all Parquet publishing paths #1213's encoding policy or an explicit measured exception.
• Resolve topology-overlay gaps through correct admitted behavior or an enforced unsupported-operation boundary. Lack of a current public-facade caller is not a disposition for unsafe admitted storage operations. Test/benchmark-only reachability must be classified and the support decision enforced. Do not add production topology journaling solely to satisfy coverage.
• Prove exact identities, schemas, endpoints, routes, properties, membership/ordinal authorities and ID-allocation continuity across public construction/mutation, replay/compaction, reopen/query, export/full verify and clean import. Preserve authentication, recovery, cancellation and active snapshots.
• Measure storage, CPU, read/write I/O, peak memory and temporary disk; enforce deterministic representative-path budgets. Replay or compaction must not silently undo accepted storage optimizations. Reuse existing evidence rather than create another certification campaign.

Focused ownership is preserved: #1213 owns permanent encoding policy; #1218 owns exploratory edge ordering and relationship schema; #1219 owns CAS-backed delta/compaction ownership. New native blocker #1221 owns the remaining topology-journal support boundary and publishing-contract conformance proof, sequenced after #1218 and #1219. It coordinates with #1213 without adding a new prerequisite to that issue.

This is an explicit maintainer-authorized extension of the epic, not an expansion of #1218's in-flight repair. #1221 must close with merged fixes and direct evidence, or an enforced, tested unsupported-operation disposition; merely recording the gaps is insufficient. Backward compatibility and migration remain out of scope before v1.0.0. Preserve all existing dependencies and the three-change WIP limit; issue creation starts no implementation stream.

Integrated results and closeout map

Maintain one source-bound results table in the existing storage/scale evidence documentation and link it here. Reuse #1196/#1213/#1207 fixtures and #900/#745 host artifacts; this is a reporting requirement, not a new harness or certification issue. The epic coordinator owns reconciliation at integration and closeout.

Final integrated evidence position

The merged integrated report and exact-byte summary replace the earlier provisional comparison table. They map all implemented representation, publishing and query improvements to their own measured fixtures, report the final host lifecycle, and preserve failed or rejected observations. Percentages across different fixtures are not additive.

Terminal capacity/S26 certification is #900/#745 scope (see Non-goals). This epic's integrated-evidence and implementation acceptance criteria are complete.

Required integrated comparison

• Identify frozen baseline and integrated candidate sources, fixture/input identities, toolchain, enabled indexes, format/settings, host/allocation assumptions and exact commands. Compare identical workloads; explicitly mark non-comparable or unmeasured cells.
• Report absolute and normalized permanent allocated bytes and bytes/live edge; retained allocation by owner; actual simultaneous lifecycle peak and temporary-disk peak with the phase/owners responsible; per-phase read/write I/O, CPU/elapsed time and peak RSS. Define amplification denominators and report raw bytes too: a smaller permanent denominator can increase the ratio despite a lower absolute lifecycle peak.
• For representative public queries, report cold/warm repeated latency/CPU observations with variance and deterministic bytes/rows/pages decoded or other applicable work counters. Include maintenance/write costs and state logical-counter versus process/OS measurement boundaries.
• Attribute each landed repair without double counting and show the final combined outcome. Rank remaining costs by absolute allocation, peak contribution and execution work; identify the largest remaining bottleneck and any unresolved regression. Small-fixture evidence and host-scale proof remain separate.
• Before judging each optimization experiment, record workload-specific minimum useful benefit and maximum acceptable resource costs, derived from its measured baseline. Document rationale and threshold revisions; use deterministic CI budgets and repeated timing observations, not arbitrary universal percentages or timing-only gates. test(perf): assess practical query and storage efficiency opportunities #1207 applies this to all five query candidates; fix(storage): unify permanent encoding policy across all Parquet publishing paths #1213 applies it to per-path encoding decisions.

Acceptance ownership and evidence

Acceptance outcome	Owner	Evidence needed at epic close
Construction linearity, safe retirement and peak/resource bounds	#901, completed #1195	Paired lifecycle/fault tests plus shared host evidence
Representation assessment and selected permanent repairs	#1196, #1202–#1205	Merged production results, exact semantic tests and resource dispositions
Permanent publishing encoding policy and exceptions	#1213	Complete writer inventory, actual published encodings and measured per-path budgets
Publishing semantics and support boundaries	#1221, with #1216/#1218/#1219/#1222 and any verified focused blockers	Finite contract census, public conformance and recovery/snapshot evidence; merged independently owned repairs
Five query-efficiency decisions and selected improvements	#1207 and its verified repair owners	Public execution comparisons and justified decisions for every candidate
Integrated gains, remaining costs and lower-rung host projection	Epic coordinator	Reconciled results table and retained S18–S22 ladder evidence on the integrated tree (integrated-storage-1194.md, transient-peak and authentication-regime records)

Before closure, map each acceptance criterion to a merged owner and direct test/result or explicit applicable disposition; a closed issue or green CI alone does not prove a performance outcome. Resolve all validated material opportunities under the existing maintainer-disposition rule. Keep ordinary PR merge gates distinct from #900/#745 terminal scale evidence, and do not run duplicate qualification solely to satisfy multiple trackers.

Execution discipline

Finish live merge candidates and prerequisite repairs first. #1221 supplies a finite publishing-contract census grouped by verified root cause; do not discover and file one issue per symptom serially when the same audit can expose the complete bounded failure set. Preserve focused ownership and the three-change WIP limit. #1207 may prepare source tracing and workload/oracle specifications read-only while blocked, but dependent implementation and accepted measurements wait for its live prerequisites. No dependency is waived by this clarification.

Latest merged encoding outcome

#1213 closed through #1227 (ec7758535ffe7c2a013d5d2808215b19f3008404), with exact-head CI Gate and all required lanes green in run 34452643981. The shared assessment now contains the complete permanent-writer inventory, baseline/candidate paired codec profiles and actual public lifecycle measurements. All 16 permanent sites select Zstd1; replay dictionaries/row groups, staging row groups, restoration metadata and separate lifecycle ownership remain intact. The existing 2 MiB replay ladder remains enforced through independently admitted phases with a bounded private IPC stream when necessary.

At the measured 1,025-node/4,097-edge fixture, compaction Parquet is 43.5% smaller and allocated Parquet is 41.0% smaller. Whole-lifecycle syscall reads/writes fell approximately 9.2%/5.9%, while elapsed time rose 12.48 → 12.99 s and peak RSS 156,284 → 159,492 KiB. These are single-host observations with explicit limitations, not integrated scale percentages or a CPU/RSS guarantee. Deterministic storage, encoding, admission and temporary-stream boundaries have direct tests. All 20 public publishing fixtures and authoritative native CI passed.

The broader publishing-contract census/repairs remain #1221, followed by the live manifest/CSR/query dependencies and the existing admitted lower-rung host comparison. This merge does not close those outcomes or the epic, and introduces no additional implementation stream.

UUID ownership repair merged (2026-09-10)

#1228 completed in #1232 (67412c498580ee60b23171a7bbd0b0617160ea94), exact-head CI run 34460908588 passed, including Bazel and native durability. Mutable v5 controls are private; authenticated immutable runs remain shared. Public mutation/reopen/portable/snapshot/recovery regressions and measured bounded control-copy costs are committed. Canonical #1221 and remaining verified #1229/#1230/#1231 publishing defects remain open; no acceptance criterion is waived.

Composition-clear boundary merged (2026-09-10)

#1230 completed in #1233 (4e397d250ba77c1aaa7e493c857dadd667f638c0), exact-head CI run 34464095172 passed. Bound clear refuses before staging while supported unbound clear retains recovery/retry; exact snapshots, nullable properties, reopen/portable and zero durable-file-change budgets are covered with measured resource evidence. #1229 and #1231 remain the verified native lifecycle blockers for canonical #1221; other #1194 criteria remain open.

Publishing census update: #1228 and #1230 are merged and closed. #1231 is merged and closed through #1234 with exact-head CI green. #1224 was reopened for a separately verified remaining acceptance case: public construction retains exploratory edge properties under _exploratory, while composite edge-property publishing selects the logical relation name as owner, so SET/removal can leave queried values unchanged. The reproducer and source diagnosis remain on the canonical owner-routing issue; no overlapping issue was created. #1229, #1224 and the remaining #1221 contract work stay open.

#1231 completed in #1234 (1aea7a8b823f5e79ba890c80f822cc50ba06f85f), with exact-head Test Suite/CI Gate run 34471082642 green, including Bazel, both bindings, native Windows/macOS lifetime regressions and concurrency. Compaction refreshes complete facade authority, preserves old streams, and recovers its receipt for immediate same-facade retry after a post-CURRENT error. CAS and generation-owned refresh costs have separate deterministic budgets and source-bound CPU/I/O/RSS/allocation evidence. #1224 remains reopened for current construction-versus-ordinary edge-property ownership; #1229 and the remaining #1221 contract work remain open.

Latest merged publishing outcome (2026-09-10)

The publishing repair batch is complete: #1229 merged in #1236 (c481cc1539b94b0dba7e9092b443408a331ae749), the final #1224 named-edge ownership repair merged in #1237 (a04fea86fd64fc3bf650ca3b9634c9291acb323c), and canonical #1221 merged in #1238 (32b9d8f33956c4fb250cd36522b39be545eec85f). All three issues are verified closed. Earlier open/reopened statuses above are historical.

#1238 passed required Test Suite/CI Gate run 34502103107 at exact head 3a45bbd1b532b061bd9695cac53b42af42a51d0e, including authoritative Bazel, both bindings, native Windows/macOS durability and concurrency. Before squash merge, its state was CLEAN, it had no unresolved review threads and its closing references named exactly #1221.

The publishing contract and acceptance ledger document supported publishers and enforce property-only GFDR at admission, persisted decoding, retries and direct replay. Four actual flat/sharded, exploratory/ontology public lifecycles preserve exact identities, endpoints, nullable values and consumed node/edge IDs through mutation, compaction, reopen, export/full verification, clean import and subsequent mutation. Existing merged recovery/snapshot coverage remains in force.

Source-bound measurements separate deterministic logical budgets from process RSS, syscall I/O and sampled overlapping allocation. Local focused checks passed; full local coverage had the documented hardcoded-/tmp filesystem-admission failure and is not claimed green. Required native CI passed without skips or weakened assertions.

Remaining authorized epic work is #1204, #1205 and #1207, followed by integrated evidence on the existing admitted S20/S22 ladder. No S24/S26 run or new certification workflow has been started. The epic remains open for those actual outstanding outcomes; this publishing closure is not a final-capacity claim.

Bounded manifest result merged (2026-09-10)

#1204 closed through #1239 (c74a529ac9c60997c1343b9f36a074bef39173d8), with required exact-head CI Gate and all native lanes green in run 34515324325. The fixed current 352-entry fixture reduces manifest allocation from 1,978,368 to 856,064 bytes (56.7%), meeting the historical 1,536,000-byte acceptance budget. Historical 544-entry counts are separate; no earlier edge-payload savings are credited to buckets. Bounded authenticated insert/replace/delete/split/collapse, corruption refusal, snapshot/recovery and 45 real facade lifecycle cases passed.

Resource evidence separates logical budgets, actual permanent allocation, process RSS, syscall I/O and sampled overlapping file owners. Whole-fixture CPU is not claimed as an isolated manifest or query speedup. Local full pre-push retains its documented hardcoded-/tmp admission failure; exact-head required CI passed without weakening it.

Remaining authorized implementation is #1205 and #1207 (including all material validated query repairs), followed by integrated evidence on the existing admitted S20/S22 ladder. No S24/S26 or new certification workflow has started. Earlier open #1204 status above is historical.

CSR repair merged — #1205 / #1240

PR #1240 squash-merged as 5fc68e568893110cca97b0eb2c797d306b264e27; #1205 closure verified on 2026-09-10 at 21:17:50Z. Exact-head CI Gate and every applicable lane passed in run 34529173041 at 98efdaab5a155484c4634edc214e9d98228ef4d4. The initial benchmark lockfile failure was fixed with an earlier locked-resolution check; failed evidence remains recorded.

Actual eight-route CSR allocation falls 4,972,544→1,363,968 bytes (72.6%); payload 4,920,564→1,324,020. Current-format authenticated decode bounds, all production shard writers, full-width IDs, corruption refusal, snapshots and actual public mutation/reopen/portable/subsequent-mutation tests are merged. Whole-lifecycle RSS rises 220,400→238,252 KiB; CPU/I/O, sampled overlapping disk and cold-query cost observations are disclosed in docs/development/evidence/bounded-csr-1205.json. No query latency improvement is claimed from this storage repair.

Live #1207 prerequisites #1196, #1202–#1205 and #1213 are now all closed. Remaining: complete all five quantified query/storage-efficiency decisions, implement every selected material repair, then integrated evidence on the existing admitted S20/S22 ladder. No S24/S26 run is authorized. Compression and the completed storage repairs do not close this epic.