Skip to content

epic(perf): remove public-operation copying and I/O amplification before S26 #1803

Closed
14 of 14 issues completed

Description

@DecisionNerd

Purpose

Complete the fourteen issues below before running S26: the original eleven copying, repeated-reading and I/O amplification findings plus three maintainer-approved import, publication and coverage repairs. This is a finite M5 repair epic under #745, not a new scale certification program. #735 remains the sole M5 close gate; #900/#745 retain ownership of the ladder and billion-edge result.

The maintainer requested this batch as a prerequisite to S26. Findings come from source inspection at 172dee39902ff2afac86232725223dd0bdff155a; they establish reachable extra work, not measured runtime impact or a completed exhaustive audit. Reconfirm each path on current main and add direct regression evidence.

Finite scope

Each child owns one independent cause. The successful write can incur several of these costs; use phase-scoped counters in leaves and an integrated public-operation check here so fixing one cost does not conceal another. Source registration #1811 depends on unchanged-participant reuse #1810; other ordering should follow verified native dependencies and actual integration needs, not issue numbers.

Approved scope expansion (2026-10-05)

The maintainer explicitly requested adding these three backlog issues to this epic so its worker picks them up before S26, accepting the broader scope:

All three are native children and blockers of #1803 in M5. Workers must include them in this epic's completion work, using their existing acceptance criteria. These additions are not claimed to be measured S26 speedups. Existing WIP limits, focused PRs, CI requirements and explicit scale-run authorization remain in force.

Close criteria

  • All fourteen native child issues have merged repairs and direct acceptance evidence, or an explicit maintainer-approved, evidence-backed non-code disposition; no unresolved finding is silently deferred past S26.
  • The three approved additions fix(storage): bulk import refuses uniform-UUID graph500-22 as skewed staged identity partitioning #1731, perf(api): composite-transaction publish is quadratic, one linear fingerprint scan per row #1420 and fix(coverage): the Rust coverage ledger rejects a cfg(test) call argument, so Coverage has been red since 2026-10-01 #1762 satisfy their existing issue-specific acceptance criteria, with results linked here.
  • A finite representative matrix exercises single-edge/single-node construction, property SET/REMOVE, multi-statement commit/abort, relationship MERGE, label changes, graph publication with large unchanged sibling ledgers, source registration, derivation validation and inspection helpers, and ledger append/merge through the ordinary public Rust facade. For perf(knowledge): incrementally validate nine ledger merges #1421, compare per-existing-row cost at ledger sizes an order of magnitude apart and prove retained invariants by mutation.
  • Fixed useful work is compared at two sizes of unrelated graph/ledger data and, where relevant, at multiple statement/input counts. Record actual read/write/checksum bytes, decoded rows, allocations or copy counts appropriate to each claim, plus per-existing-row ledger append cost at two sizes an order of magnitude apart. Remove the identified amplification terms without promising that every operation is constant-time or that necessary affected-fragment rewrites disappear.
  • Exact values, identifiers, counters, ordering, missing-subject/conflict errors, retained-generation visibility and every ledger invariant formerly enforced by full validation remain correct. Mutation cases include cancellation/fault recovery and reopen; shared participant reuse includes retention/GC and portable round-trip coverage.
  • Corruption refusal, complete generation ownership, existing checksum policy, bounded memory, Rust ownership and thin-binding contracts remain intact. No skips, ignored errors, new fallback engines or weakened checks.
  • Each implementation PR has the ordinary exact-head CI Gate and squash-queue merge evidence. Aggregate test commands and results are attached here; methodology belongs in the nearest existing topic document only when needed.

Completion scenarios

  • Given a fixed small mutation or metadata request and substantially larger unrelated data, when the final integrated public operation runs, then the specific unwanted scans/copies described by its children are absent and its exact result remains correct.
  • Given a later-statement failure, interrupted publication, corrupted required payload or retained reader, when recovery/reopen/GC runs, then existing correctness and durability guarantees hold.
  • Given any unresolved child, when S26 launch readiness is assessed, then this prerequisite remains open. Once repairs and their evidence are accepted, the existing adjacent-rung, capacity and authorization requirements still apply.

Scope boundaries and related work

Native relationships and execution

Native child and blocker of #745. Blocks #900's S26 launch and #745's final certification. All fourteen children are native sub-issues and blockers of this epic, with milestone M5. Preserve existing dependencies and do not reparent unrelated work.

Follow repository WIP limits: at most six unmerged concerns across the coordinated team, integrate the next ready dependency before starting more, and use a focused issue branch/PR per concern. This issue creation does not authorize launching the scale run.

Activity

  1. added
    bugSomething isn't working
    coreCore source code changes
    on Oct 4, 2026
  2. 5 remaining items

  3. DecisionNerd commented on Oct 5, 2026

    @DecisionNerd
    ContributorAuthor

    #1803 integrated close-gate evidence (local head b56b15edc)

    The integrated public Rust API matrix is in PR #1866. Its test public_facade_workflow_matrix_runs_at_two_unrelated_data_sizes uses one sequential persistent GraphForge scenario per fixture, with 4 and 32 unrelated nodes (8x). It measures each action independently and checks values/rows plus committed state after reopen. It covers node add (twice), edge add, property SET/REMOVE, multi-statement commit/abort, relationship MERGE, label change, assertion appends, graph publication with unchanged assertion and knowledge participants, source registration, artifact registration with derivation validation, and graph/source/artifact inspection.

    Metrics: bytes_read and lifecycle_write_bytes are lifecycle I/O deltas; sealed_written_bytes is the region's sealed-output counter when instrumented; checksum_bytes is bytes processed by PayloadDigestCapture; rows_decoded is reported only when the region records that counter; instrumented_rows is the separate inspection rows counter and must not be read as decoded rows. n/a means no such operation counter was emitted, not zero.

    Unrelated nodes Action Read B Lifecycle write B Sealed write B Checksum B Decoded rows Inspection rows
    4 add_node #1 164098 0 27018 165667 n/a 12
    4 add_node #2 201638 0 30031 200178 n/a 16
    4 add_edge 169614 0 24152 157006 n/a 22
    4 property_set 227363 0 14361 181137 n/a 28
    4 property_remove 244660 0 14747 195390 n/a 32
    4 relationship_merge 267340 0 35040 260058 n/a 38
    4 label_change 244459 0 27147 223920 5 46
    4 multi_statement_commit 528334 31217 49171 797221 n/a 50
    4 multi_statement_abort 0 0 n/a 0 n/a n/a
    4 ledger_subject_node 280005 0 31954 257596 n/a 52
    4 ledger_append_assertion 253564 0 60817 459583 n/a 61
    4 assertion_append_existing_row 337408 0 60989 544278 n/a 73
    4 graph_publish_with_assertion_ledger 332507 0 33946 312532 n/a 68
    4 source_ledger_append 243076 0 7983 194784 n/a 74
    4 source_ledger_second_append 244667 0 8139 197152 n/a 81
    4 artifact_append_and_derivation_validation 259843 0 67490 509083 n/a 89
    4 artifact_append_existing_row 269393 0 67754 519953 n/a 102
    4 graph_publish_with_knowledge_siblings 348943 0 34660 334057 n/a 96
    4 inspection_helpers 482043 0 n/a 316461 n/a 59
    4 source_and_artifact_inspection 23645 0 n/a 6775 n/a 4
    32 add_node #1 173787 0 29175 197953 n/a 68
    32 add_node #2 210324 0 30382 221116 n/a 72
    32 add_edge 175949 0 26723 192249 n/a 78
    32 property_set 233760 0 14931 200918 n/a 84
    32 property_remove 250176 0 14842 214463 n/a 88
    32 relationship_merge 276126 0 35288 285902 n/a 94
    32 label_change 248572 0 26165 243811 33 102
    32 multi_statement_commit 555640 35881 51870 890518 n/a 106
    32 multi_statement_abort 0 0 n/a 0 n/a n/a
    32 ledger_subject_node 290447 0 33022 283322 n/a 108
    32 ledger_append_assertion 305142 0 100874 712699 n/a 117
    32 assertion_append_existing_row 467716 0 101063 876209 n/a 129
    32 graph_publish_with_assertion_ledger 381726 0 33138 375720 n/a 124
    32 source_ledger_append 295410 0 8680 250405 n/a 130
    32 source_ledger_second_append 297003 0 8834 252765 n/a 137
    32 artifact_append_and_derivation_validation 312872 0 107557 762249 n/a 145
    32 artifact_append_existing_row 322408 0 107828 773140 n/a 158
    32 graph_publish_with_knowledge_siblings 407005 0 38088 401481 n/a 152
    32 inspection_helpers 496272 0 n/a 322281 n/a 199
    32 source_and_artifact_inspection 23645 0 n/a 6775 n/a 4

    The assertion participant measured 41,389 bytes at the 4-node fixture and 80,768 bytes at the 32-node fixture; exact participant bytes were asserted unchanged across both publication steps. The integrated fixture grows unrelated nodes (4→32) and assertion claim bytes (64→128 KiB) together, so this combined run is not an independent graph-size-versus-ledger-size experiment; use the scoped child tests below and the #1421 10x ledger comparison for isolated scaling evidence. The integration fixture appends assertions to an existing ledger, and sources/artifacts likewise append existing rows. #1421's distinct per-existing-row cost comparison at ledger sizes at least 10x apart, plus mutation proof, is recorded in the nine-child evidence index; it is not inferred from this 1–2-row integration fixture.

    Scoped counters and correctness coverage

    The integrated matrix intentionally leaves unsupported decoded-row counters as n/a. Existing focused facade tests supply scoped values for those operation claims: edge rollback/fault evidence #1815, ordinary writes #1822, SET/REMOVE #1838, transaction checkpoint and later-failure recovery #1834 and #1807 result comment, MERGE #1821, label decoded rows/reopen #1843, participant reuse and unchanged write/hash bytes #1816, source reuse #1841, derivation per-kind bytes/decoded rows #1832, and inspection row scope #1819. These links retain each test's own fixture and measured quantities; none are presented as integrated-matrix counters.

    Public cancellation/reopen, injected fault recovery/reopen, and reuse correctness were rerun at this head. The named tests also verify commit/abort semantics, exact reopened values, participant retention/GC, and portable round-trip:

    • CARGO_TARGET_DIR=/home/ubuntu/.cache/graphforge-target-1803-facade CARGO_BUILD_JOBS=1 python3 scripts/test_environment.py -- cargo nextest run -p graphforge-api --lib --locked -E 'test(public_facade_workflow_matrix_runs_at_two_unrelated_data_sizes)' --no-capture — passed, 1 passed, 898 filtered; emitted the table above.
    • CARGO_TARGET_DIR=/home/ubuntu/.cache/graphforge-target-1803-facade CARGO_BUILD_JOBS=1 python3 scripts/test_environment.py -- cargo nextest run -p graphforge-api --lib --locked -E 'test(/production_history_certifies_cancel_drop_and_idempotent_retry/) | test(/composite_kill_reopen_matrix_never_exposes_mixed_state/) | test(/graph_mutation_reuses_large_unchanged_knowledge_participant/)' — passed, 3 passed, 896 filtered.
    • CARGO_TARGET_DIR=/home/ubuntu/.cache/graphforge-target-1803-facade CARGO_BUILD_JOBS=1 python3 scripts/test_environment.py -- cargo nextest run -p graphforge-storage --lib --locked -E 'test(/reused_parent_participants_are_hard_linked_without_payload_hash_or_write/)' — passed, 1 passed, 1600 filtered.
    • make check — passed (fmt, clippy, bench compile, ruff, mypy, workflow/repository checks).

    The other approved child results are linked directly: #1731 actual graph500-22 import/reopen result, #1420 deterministic fingerprint-work result, and #1762 successful coverage run. Their implementations and CI gates are merged on main. PR #1866 passed exact-head CI Gate at head 94d2708625cebc0521034c703f6a7ce44d8deef6 (run 37392121376), then passed the merge-queue CI Gate (merge-group run 37393634695) and merged as 973583ba3ee687b36b64727279966aa25dd995eb. The squash commit is on main; #1803 closed automatically on merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcoreCore source code changes

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions