perf(storage): prepare construction readers before CURRENT and account publication in the ingest budget (#1481) - #1543
Conversation
…date before CURRENT (#1481) Move the construction reader refresh (hydration and read_authority) from after the CURRENT replacement to against the durable, lease-verified candidate immediately before it, through the publisher's existing pre-CURRENT preparation seam with cancellation still pollable to the commit point. A candidate that cannot be hydrated or authorized now fails closed: CURRENT stays on the parent generation instead of committing a generation whose reader workspace cannot be prepared. Receipts nest hydration and read_authority under generation_commit for fresh publications; replay keeps the sibling shape. ABeforeInstall keeps the committed-authority recovery contract; an interrupted attempt still recovers through the standard transaction-recovery entrypoint.
…repair A/B (#1481) Current integrated-tree baseline (main moved: #1502, #1531, #1533, #1534, #1538): S18 base n=3 median 31.818s (7.586 us/edge, publish 2.394s); S22 base n=3 median 547.803s (8.163 us/edge, publish 42.014s with quiet-region decomposition). The reader-preparation repair A/B under the predeclared criterion shows publish wall unchanged within noise at S18 and S22 and whole-ingest medians below the declared benefit thresholds, so no throughput gain is claimed: the repair is kept for fail-closed publication (preparation failure leaves CURRENT unchanged). Includes the rustfmt application to the repair commit.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: CurateLabs/graphforge/.coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Warning Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
core
… for the split refresh-failure tests (#1481)
1 participant
Closes #1481.
What
Two commits:
Reader preparation before CURRENT (
868a0a8f). The construction reader refresh (hydration+read_authority) moves from after theCURRENTreplacement to against the durable, lease-verified candidate immediately before it, through the publisher's existing pre-CURRENTpreparation seam; cancellation remains pollable to the commit point. A candidate that cannot be hydrated or authorized now fails closed —CURRENTstays on the parent generation — instead of committing a generation whose reader workspace must be recovered afterward. Receipts nesthydration/read_authorityundergeneration_commitfor fresh publications; replay keeps the sibling shape; the certification region allowlist is segment-based, so no schema change is needed.BeforeInstallkeeps the committed-authority recovery contract; interrupted attempts still recover through the standard transaction-recovery entrypoint.Evidence (
1ac59a97). Current integrated-tree baseline (main moved: feat(knowledge): add Source and Artifact lifecycle with bidirectional lineage (#1349) #1502, fix(storage): keep the shape-end controls independent of the sealed-segment count (#1526) #1531, test(benchmarks): keep host-run fixtures off the retained ladder evidence root #1533, docs(evidence): retain clean-ladder rung evidence behind a verified post-run step (#1530) #1534, feat(storage): publish immutable research Versions and scoped restoration #1538 since the S18-attribution observations), clean S22 baseline with quiet publication decomposition, the repair A/B under the predeclared criterion, and updatedingest-region-diagnostics.md.Baseline (quiet, BenchExec
--cores 0-15 --memlimit 4GB, recorded Graph500 inputs SHA-verified)Clean S22 publish decomposition:
cas_install16.3 s /generation_commit11.6 s /hydration7.3 s /publication_receipt5.7 s /read_authority0.7 s — same structure the contended runs indicated, now on quiet timings.A/B and disposition
Publish wall is unchanged within noise (S18 2.394 → 2.398 s; S22 42.0 → 41.7 s median): the repair moves work inside the same publish region and does not remove it. Whole-ingest median deltas (−0.77 s S18, −13.2 s S22) are below the predeclared benefit thresholds (9.2 s / 91.3 s =
max(3% of baseline median, 2× baseline spread)), so no whole-ingest throughput gain is demonstrated or claimed. The repair is kept for its fail-closed semantics (#1481 completion scenario 2): corruption or failure during reader preparation leaves the acknowledged generation unchanged. Publication's remaining deficit against the 1M edges/s floor stays owned by the parallel-ingest workstreams under #1387.Tests
CURRENTis unchanged, and that preparation failure leavesCURRENTunchanged with the session recoverable; the split API refresh-failure tests cover both the fail-closed pre-commit boundary and the retainedBeforeInstallcommitted-authority contract.cargo fmt --all -- --check,cargo clippy --workspace -- -D warnings,make pre-push-fast, graphforge-storage lib (1264 passed), graphforge-api lib construction/publication suites (724 passed, excluding the 37repository::tests that fail identically on pristinemainon this host for environment reasons), CLI receipt test asserting the new nesting, certify-runner suite (41 passed).Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.