Skip to content

perf(storage): retire consumed shaping roots before resolved merge peak #1268

Description

@DecisionNerd

Problem and verified cause

The final #1194 ladder still refuses S26 capacity. A frozen-source S18 diagnostic on 24ca688c516a86a68de9cffaab2d5a9215291256 found two obsolete derived authorities coexisting with the final resolved-endpoint merge: merge-identities (142,610,432 allocated bytes) and merge-endpoints (402,657,280 bytes). The successful validation receipt records a 2,561,531,904-byte construction staging peak. A pathname sampler observed a 2,765,774,848-byte project peak including the import owner; this sampled observation is not a hard bound.

Source tracing and independent review show that the unassigned identity root has no consumer after assign_surrogates. The node-ordered endpoint root has no consumer after all resolved windows have been durably installed and its reader is closed, before inserting the final run into the online merger or calling resolved.finish_optional. Neither obsolete source belongs to the completed-shape inventory. Original accepted chunks remain the authority for restarting an incomplete shape. Existing cleanup tolerates absent derived runs.

The diagnostic commit hit an environment error after durable publication because the driver omitted the native TMPDIR; its failed result is preserved. Reopening with the correct environment returned the exact 4,194,304-edge count. This diagnostic is allocation/source evidence, not a successful full lifecycle or a before/after performance claim.

The first candidate demonstrated only the identity-root saving at 1x/2x/4x because the final resolved.push can execute the large online carry merge before finish_optional. The refinement durably installs the last run, closes and retires the consumed endpoint root, then performs that final insertion. No new recovery authority is needed.

Scope

Reduce this verified coexistence using existing authenticated unlink/accounting and incomplete-shape recovery. Do not add a cleanup journal, compatibility reader, fallback engine, generic writer abstraction, or new certification workflow. Preserve full UUIDs, properties, current-format corruption refusal, cancellation, retries and active snapshots. This is a focused new capacity repair under canonical #1194; the completed #901 outcomes remain historical evidence.

Acceptance criteria

  • Retire the derived unassigned identity root after successful surrogate assignment and the derived node-ordered endpoint root after readers close, before the final resolved merge. Retain shaped identities and all still-needed original inputs.
  • Prove real multi-level 1x/2x/4x construction peak reduction against frozen current-source baseline with deterministic allocated-byte budgets; do not substitute final cleanup for an earlier peak reduction.
  • Directly test crashes and returned errors at the shortened lifetime boundaries, cancellation/reopen, exact reconstruction, replaced-inode/extra-link refusal, and accounting only after verified unlink. Preserve completed-shape authentication and existing snapshot/retry behavior.
  • Prove actual public construction/query/reopen/export/full verification/clean import/subsequent mutation with exact identities/properties, reusing existing regression fixtures where appropriate.
  • Freeze executables before paired resource measurements. Report allocated peak/retained bytes, CPU, read/write I/O, process memory and sampling limitations, including failed/superseded observations. Set representative budgets from baseline evidence before judging candidate results.
  • Pass changed-surface repository gates and exact-head CI Gate, squash merge, verify closure. Update the existing integrated report with the measured outcome; final capacity remains epic(storage): reduce lifecycle amplification and improve persistent storage/query efficiency #1194/test(scale): complete the S18-S26 lifecycle ladder on OVHC-AGENCY #900/test(scale): certify the S26 billion-edge round trip on OVHC-AGENCY #745. Any subsequent host evidence uses only the existing admitted ladder through S22; no S24/S26 execution.

Completion scenario

Given a sealed multi-level construction with retained original chunks, when obsolete derived roots are removed before subsequent merge work, then peak allocation decreases and interruption reconstructs exactly from original authority without publishing a partial generation or double-counting removal.

Verified completion: #1271 squash-merged as cd3f863e65ef8de5c4185cf7e5f4ead1886dd747 after exact-head CI34875341039 and CI Gate passed on28ab552c. The failed prior run and separately merged #1272 gate correction remain documented. Frozen 1x/2x/4x shaping peaks fell10.87–11.30%, with unchanged I/O and CPU/RSS within preselected diagnostic envelopes. Five public construction and four publishing-contract tests passed. Final integrated host capacity remains #1194/#900/#745.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    coreCore source code changesenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions