Repository navigation
Description
Purpose
Complete the fourteen issues below before running S26: the original eleven copying, repeated-reading and I/O amplification findings plus three maintainer-approved import, publication and coverage repairs. This is a finite M5 repair epic under #745, not a new scale certification program. #735 remains the sole M5 close gate; #900/#745 retain ownership of the ladder and billion-edge result.
The maintainer requested this batch as a prerequisite to S26. Findings come from source inspection at 172dee39902ff2afac86232725223dd0bdff155a; they establish reachable extra work, not measured runtime impact or a completed exhaustive audit. Reconfirm each path on current main and add direct regression evidence.
Finite scope
- perf(knowledge): avoid quadratic revalidation of existing ledger rows on append — perf(knowledge): incrementally validate nine ledger merges #1421
- perf(api): remove the whole-workspace snapshot from add_edge — perf(api): remove the whole-workspace snapshot from add_edge #1804
- perf(exec): avoid full node scans when initializing ordinary writes — perf(exec): avoid full node scans when initializing ordinary writes #1805
- perf(storage): bound property mutation inventory refresh to affected files — perf(storage): bound property mutation inventory refresh to affected files #1806
- perf(api): remove per-statement whole-workspace transaction backups — perf(api): remove per-statement whole-workspace transaction backups #1807
- perf(exec): batch relationship MERGE candidate property reads — perf(exec): batch relationship MERGE candidate property reads #1808
- perf(storage): target node label mutations to affected fragments — perf(storage): target node label mutations to affected fragments #1809
- perf(storage): reuse unchanged generation participants during publication — perf(storage): reuse unchanged generation participants during publication #1810
- perf(api): preserve unchanged ledgers when registering a source — perf(api): preserve unchanged ledgers when registering a source #1811
- perf(api): batch artifact derivation input validation by ledger kind — perf(api): batch artifact derivation input validation by ledger kind #1812
- perf(api): avoid unrelated graph scans in inspection helpers — perf(api): avoid unrelated graph scans in inspection helpers #1813
Each child owns one independent cause. The successful write can incur several of these costs; use phase-scoped counters in leaves and an integrated public-operation check here so fixing one cost does not conceal another. Source registration #1811 depends on unchanged-participant reuse #1810; other ordering should follow verified native dependencies and actual integration needs, not issue numbers.
Approved scope expansion (2026-10-05)
The maintainer explicitly requested adding these three backlog issues to this epic so its worker picks them up before S26, accepting the broader scope:
- fix(storage): bulk import refuses uniform-UUID graph500-22 as skewed staged identity partitioning #1731 — fix valid uniform-UUID bulk imports rejected as skewed staged identity partitioning. Retain the deterministic regression and actual patched graph500-22 import, commit and exact reopen count evidence; preserve its blocker relationship to epic(bench): externalize scale certification and add GDC benchmark suites #952.
- perf(api): composite-transaction publish is quadratic, one linear fingerprint scan per row #1420 — remove quadratic per-row fingerprint lookups in composite-transaction publication. Retain deterministic lookup/build counts and identical fingerprint semantics.
- fix(coverage): the Rust coverage ledger rejects a cfg(test) call argument, so Coverage has been red since 2026-10-01 #1762 — repair the Rust coverage ledger scanner, obtain a successful Coverage run, and explain or separately track the earlier release-profile-only test failure as required by the issue.
All three are native children and blockers of #1803 in M5. Workers must include them in this epic's completion work, using their existing acceptance criteria. These additions are not claimed to be measured S26 speedups. Existing WIP limits, focused PRs, CI requirements and explicit scale-run authorization remain in force.
Close criteria
- All fourteen native child issues have merged repairs and direct acceptance evidence, or an explicit maintainer-approved, evidence-backed non-code disposition; no unresolved finding is silently deferred past S26.
- The three approved additions fix(storage): bulk import refuses uniform-UUID graph500-22 as skewed staged identity partitioning #1731, perf(api): composite-transaction publish is quadratic, one linear fingerprint scan per row #1420 and fix(coverage): the Rust coverage ledger rejects a cfg(test) call argument, so Coverage has been red since 2026-10-01 #1762 satisfy their existing issue-specific acceptance criteria, with results linked here.
- A finite representative matrix exercises single-edge/single-node construction, property SET/REMOVE, multi-statement commit/abort, relationship MERGE, label changes, graph publication with large unchanged sibling ledgers, source registration, derivation validation and inspection helpers, and ledger append/merge through the ordinary public Rust facade. For perf(knowledge): incrementally validate nine ledger merges #1421, compare per-existing-row cost at ledger sizes an order of magnitude apart and prove retained invariants by mutation.
- Fixed useful work is compared at two sizes of unrelated graph/ledger data and, where relevant, at multiple statement/input counts. Record actual read/write/checksum bytes, decoded rows, allocations or copy counts appropriate to each claim, plus per-existing-row ledger append cost at two sizes an order of magnitude apart. Remove the identified amplification terms without promising that every operation is constant-time or that necessary affected-fragment rewrites disappear.
- Exact values, identifiers, counters, ordering, missing-subject/conflict errors, retained-generation visibility and every ledger invariant formerly enforced by full validation remain correct. Mutation cases include cancellation/fault recovery and reopen; shared participant reuse includes retention/GC and portable round-trip coverage.
- Corruption refusal, complete generation ownership, existing checksum policy, bounded memory, Rust ownership and thin-binding contracts remain intact. No skips, ignored errors, new fallback engines or weakened checks.
- Each implementation PR has the ordinary exact-head CI Gate and squash-queue merge evidence. Aggregate test commands and results are attached here; methodology belongs in the nearest existing topic document only when needed.
Completion scenarios
- Given a fixed small mutation or metadata request and substantially larger unrelated data, when the final integrated public operation runs, then the specific unwanted scans/copies described by its children are absent and its exact result remains correct.
- Given a later-statement failure, interrupted publication, corrupted required payload or retained reader, when recovery/reopen/GC runs, then existing correctness and durability guarantees hold.
- Given any unresolved child, when S26 launch readiness is assessed, then this prerequisite remains open. Once repairs and their evidence are accepted, the existing adjacent-rung, capacity and authorization requirements still apply.
Scope boundaries and related work
- epic(storage): reduce lifecycle amplification and improve persistent storage/query efficiency #1194 is completed historical lifecycle-amplification work; these are separately identified remaining public-operation costs.
- epic(query): bounded queries should cost what their results cost, not what the graph costs #1388 owns bounded-query/open outcomes and its scale observation; this epic does not duplicate its acceptance gate. Full counts may remain proportional to relevant data; unrelated scans in the named inspection APIs are in scope here.
- perf(knowledge): incrementally validate nine ledger merges #1421 owns repeated ledger row validation as a distinct child of this epic; the knowledge children perf(api): preserve unchanged ledgers when registering a source #1811/perf(api): batch artifact derivation input validation by ledger kind #1812 own repeated ledger loading/encoding and input lookup, not that merge-validation algorithm.
- perf(storage): capture graph publication and CAS install payload identities once #1691/perf(storage): capture publish and export payload identities once per operation #1657 own prior digest-pass reductions; unchanged participant carry-forward is a distinct remaining payload-copy/write cost. Preserve refactor(storage,api): move digests and attribution off the data path; retire forensic verify #1617/feat(storage): version published graph payload checksums for read-time corruption refusal #1637 checksum policy.
- feat(api): unify supported mutations under one transaction lifecycle #754 defines the existing transaction contract; these repairs must preserve it.
- No new CI/workflow gate, benchmark-only path, universal performance ceiling, ingest-throughput commitment, distributed storage design, or unreported scope expansion. Other audit observations are not implicitly added to this fourteen-issue batch.
Native relationships and execution
Native child and blocker of #745. Blocks #900's S26 launch and #745's final certification. All fourteen children are native sub-issues and blockers of this epic, with milestone M5. Preserve existing dependencies and do not reparent unrelated work.
Follow repository WIP limits: at most six unmerged concerns across the coordinated team, integrate the next ready dependency before starting more, and use a focused issue branch/PR per concern. This issue creation does not authorize launching the scale run.
Activity
5 remaining items
DecisionNerd commentedon Oct 5, 2026
#1803 integrated close-gate evidence (local head
b56b15edc)The integrated public Rust API matrix is in PR #1866. Its test
public_facade_workflow_matrix_runs_at_two_unrelated_data_sizesuses one sequential persistentGraphForgescenario per fixture, with 4 and 32 unrelated nodes (8x). It measures each action independently and checks values/rows plus committed state after reopen. It covers node add (twice), edge add, property SET/REMOVE, multi-statement commit/abort, relationship MERGE, label change, assertion appends, graph publication with unchanged assertion and knowledge participants, source registration, artifact registration with derivation validation, and graph/source/artifact inspection.Metrics:
bytes_readandlifecycle_write_bytesare lifecycle I/O deltas;sealed_written_bytesis the region's sealed-output counter when instrumented;checksum_bytesis bytes processed byPayloadDigestCapture;rows_decodedis reported only when the region records that counter;instrumented_rowsis the separate inspectionrowscounter and must not be read as decoded rows.n/ameans no such operation counter was emitted, not zero.The assertion participant measured 41,389 bytes at the 4-node fixture and 80,768 bytes at the 32-node fixture; exact participant bytes were asserted unchanged across both publication steps. The integrated fixture grows unrelated nodes (4→32) and assertion claim bytes (64→128 KiB) together, so this combined run is not an independent graph-size-versus-ledger-size experiment; use the scoped child tests below and the #1421 10x ledger comparison for isolated scaling evidence. The integration fixture appends assertions to an existing ledger, and sources/artifacts likewise append existing rows. #1421's distinct per-existing-row cost comparison at ledger sizes at least 10x apart, plus mutation proof, is recorded in the nine-child evidence index; it is not inferred from this 1–2-row integration fixture.
Scoped counters and correctness coverage
The integrated matrix intentionally leaves unsupported decoded-row counters as
n/a. Existing focused facade tests supply scoped values for those operation claims: edge rollback/fault evidence #1815, ordinary writes #1822, SET/REMOVE #1838, transaction checkpoint and later-failure recovery #1834 and #1807 result comment, MERGE #1821, label decoded rows/reopen #1843, participant reuse and unchanged write/hash bytes #1816, source reuse #1841, derivation per-kind bytes/decoded rows #1832, and inspection row scope #1819. These links retain each test's own fixture and measured quantities; none are presented as integrated-matrix counters.Public cancellation/reopen, injected fault recovery/reopen, and reuse correctness were rerun at this head. The named tests also verify commit/abort semantics, exact reopened values, participant retention/GC, and portable round-trip:
CARGO_TARGET_DIR=/home/ubuntu/.cache/graphforge-target-1803-facade CARGO_BUILD_JOBS=1 python3 scripts/test_environment.py -- cargo nextest run -p graphforge-api --lib --locked -E 'test(public_facade_workflow_matrix_runs_at_two_unrelated_data_sizes)' --no-capture— passed, 1 passed, 898 filtered; emitted the table above.CARGO_TARGET_DIR=/home/ubuntu/.cache/graphforge-target-1803-facade CARGO_BUILD_JOBS=1 python3 scripts/test_environment.py -- cargo nextest run -p graphforge-api --lib --locked -E 'test(/production_history_certifies_cancel_drop_and_idempotent_retry/) | test(/composite_kill_reopen_matrix_never_exposes_mixed_state/) | test(/graph_mutation_reuses_large_unchanged_knowledge_participant/)'— passed, 3 passed, 896 filtered.CARGO_TARGET_DIR=/home/ubuntu/.cache/graphforge-target-1803-facade CARGO_BUILD_JOBS=1 python3 scripts/test_environment.py -- cargo nextest run -p graphforge-storage --lib --locked -E 'test(/reused_parent_participants_are_hard_linked_without_payload_hash_or_write/)'— passed, 1 passed, 1600 filtered.make check— passed (fmt, clippy, bench compile, ruff, mypy, workflow/repository checks).The other approved child results are linked directly: #1731 actual graph500-22 import/reopen result, #1420 deterministic fingerprint-work result, and #1762 successful coverage run. Their implementations and CI gates are merged on
main. PR #1866 passed exact-head CI Gate at head94d2708625cebc0521034c703f6a7ce44d8deef6(run 37392121376), then passed the merge-queue CI Gate (merge-group run 37393634695) and merged as973583ba3ee687b36b64727279966aa25dd995eb. The squash commit is onmain; #1803 closed automatically on merge.