plan: streaming construction — the #1387 floor, per resource

[DecisionNerd]

Ingest floor plan — the #1387 floor, per resource

Plan of record and close gate for the ingest-throughput workstream. #1387 owns the floor; #1456 owned the construction foundation beneath it. This issue closes only after #1387's acceptance outcomes are met. #735 remains M5's canonical closing tracker; this plan does not close the milestone.

History. Revisions 10–14 (measured rung tables, the rev 12–14 corrections, research routing, evidence index) are archived verbatim in docs/development/evidence/ingest-floor-plan-1478-rev14.md. GitHub keeps this body's edit history. Nothing in the archive is maintained; the decisions and cautions below are what carried forward.

How this plan stays current

This body owns the strategy, decisions and ordering rationale. GitHub owns execution state: the task list below renders each issue's live open/closed state, and each issue's native sub-issue and Blocked by / Blocking panels carry the prerequisites. Do not mirror status, counts, current main or queue contents here. Edit this body only when the order of operations changes or a finding changes a decision, budget, contract or closure criterion; record the change as a dated line under Revision log.

The floor

#1387: 1,000,000 edges per second, sustained at every supported scale, maintainer-confirmed 2026-09-18. It is one falsifiable number: any eligible complete ingest below it fails. The tested deadline is

T_complete_ingest(E) <= E / 1,000,000

which allows 4.19 s at S18 and 67.1 s at S22. Complete ingest includes registration, normalization, validation, identity resolution, construction, authentication, publication and acknowledgement; moving work between stages or behind verify does not change the boundary. S26 admission is a release claim about a billion-edge round trip and does not close the epic.

Decisions in force

#	Decision (maintainer, 2026-09-19 unless noted)	What it forbids
D1	The direct floor is the acceptance gate. Resource figures (≤1.6 KB device I/O per edge, ingest-scoped; S18 begin + resume + publish ≤0.84 s; wall exponent ≤1.02) are policy allocations and diagnostics, judged by matched-boundary measurement W/E = (C/E)/(C/W).	Any CPU-per-edge ceiling derived from an N-process probe (the ≤3.54 µs derivation is withdrawn); treating effective cores below one as a measured serial fraction; achieved rates as capacities; logical bytes as device bytes.
D2	Bytes and work removal run ahead of and alongside structural experiments; overlap is tried before decomposition.	Building a pipeline to hide measured preparation work (F5 no-go: ≤1.06× bound).
D3	The encode seam was viable and a performance no-go (#1496).	Inferring engine migration or a shaping experiment from the seam result.
D4	Reusable phase/region attribution in stock release output (#1477, landed).	Test-only scopes as the measurement instrument.
D5	Regression ratchets coexist with the unmet 1M gate (#1476).	Retiring the requirement by rewording; ratchets that stop tracking as performance improves.
R14	Continuation rule (2026-09-21). After each landed repair, recompute the remaining gap from a complete-ingest measurement on the integrated tree before starting or continuing structural work. Structural A/Bs run against a frozen baseline SHA under the owning issue's predeclared gate. Shaping alone is bounded below 1.92× (48% of validate).	Scoping #1448 as a floor-closing change; adopting the explicit-exchange, owned-artifact pipeline (a #1509 hypothesis row) as a second roadmap.
R14	Retention rule (2026-09-21). Ladder evidence is archived under #1530 before it is reported; issue text carries every number a decision rests on.	Citing scratch paths as evidence (the ladder root was deleted on 2026-09-21; the only S24 rung is gone).

Baseline of record

Retained evidence (#1530 / PR #1534). The archive index distinguishes complete rung evidence from surviving controller summaries. The b6ffb088 baseline retains S18–S22 rung/result JSON, receipts, plan, projection and controller summary; its MANIFEST.sha256 digest is 807609f97477e91c3cb5321261a4e9274f51573c150da88b320baf3b55c8342d. These links pin the merged archive commit from #1534. Historical raw evidence for 93c041df, f80f69fe (including S24), 1955f17d, ab1a713e, 4fbfe84c and 403fc02a was lost on 2026-09-21. Retained controller summaries support only their reported values; other historical numbers rest on the copied issue tables and comments, not missing per-operation receipts or publication-region data. The archived rev14 §12 is historical; this retained index is the current evidence location.

The integrated baseline is b6ffb088 (2026-09-21, quiet host, stock release, one S18–S22 ladder on a tree containing #1452, #1519 and the #1526 fix). S18 32.3 s / 129,975 edges/s; S19 63.1 s / 132,986; S20 128.8 s / 130,240; S22 532.2 s / 126,099 edges/s, 0.87 authority cores. The floor gap at S22 is 7.93×. Per-operation cpu_ns and publication regions were archived before this number was reported. The prior ab1a713e summary (S22 432.7 s, 155,093 edges/s, 0.99 cores) has no receipts. Historical rows (93c041df S22 593.1 s; f80f69fe S24 2,754.0 s) stay in the archive.

Work order

Each item is a native issue reference; GitHub shows its state. Prerequisites are the issues' own Blocked by panels. Ordering rationale is in the right column and changes only by editing this body.

• Step 0 · Integrated baseline — evidence on epic(storage): scale complete ingest across cores and reach 1M edges/s #1387: one clean quiet-host S18–S22 ladder on a recorded main SHA containing perf(storage): sealing one spill costs eight durability barriers, serialised on the directory inode #1452, perf(storage): retire staged input at shaping boundaries with crash-safe resume (#1418) #1519 and the fix(storage): S22 construction fails "control record exceeds bound" because the shape-end checkpoint ledger holds one entry per sealed segment (#1519) #1526 fix, stock release, per-operation cpu_ns and observe(storage): decompose publication into named regions in stock receipts (#1481) #1512 publication regions, ingest-scoped device bytes and sync counts where the instruments exist. Archived before it is reported. Every later step scopes from this, not from the archive.
• docs(evidence): retain clean-ladder rung evidence the plan cites; the ladder root was deleted on 2026-09-21 #1530 — retention rule for ladder evidence. Precondition for step 0 being reportable.
• test(benchmarks): host-run tests write fixture executables into the live ladder evidence root via a hardcoded path #1529 — the benchmarks test suite hardcodes the ladder root as scratch. Land before the next local test run on the ladder host.
• perf(storage): attribute construction publication and fit it into the ingest floor budget #1481 — publication attribution, allocation and justified repair; S22 pair owed. Publication was ~42% of the allowance at every observed scale; it is not fixed overhead.
• perf(bench): make the ingest gates two-sided so improvements ratchet automatically (#1387 D1/D5) #1476 — scope-aware regression gates that track improvement (D5).
• epic(storage): reduce lifecycle amplification and improve persistent storage/query efficiency #1194 — bytes, ranked by attribution and judged in device units (D2). Follow its live children.
• spike(storage): evaluate sorting and partitioning reuse for construction (#1504) #1506 — sorting/partitioning reuse spike (gates perf(storage): make one complete ingest scale with available cores #1448).
• spike(storage): evaluate Tokio/DataFusion scheduling and cancellation for construction (#1504) #1508 — scheduling/cancellation reuse spike (gates perf(storage): make one complete ingest scale with available cores #1448).
• spike(storage): evaluate spill, buffering, and memory-pool reuse under construction pressure (#1504) #1507 — spill/buffering/memory-pool reuse spike.
• perf(storage): make one complete ingest scale with available cores #1448 — shaping over partitions: one hypothesis, frozen baseline SHA, ≥10% whole-ingest median at S18 and S20 with identical digests, no verification or contract changes. The largest lever that is not byte removal; cannot meet the floor alone.
• docs(storage): integrated construction reuse experiments, tradeoff matrix, and ADR (#1504) #1509 — reuse ADR and decision matrix (adopt / retain / hybrid), including the explicit-exchange candidate as a hypothesis row. Not an M5 gate.
• Hardware last — one run on a defined reference host only after the per-resource budgets pass on OVHC-AGENCY. A 32-core box changes neither bytes per edge nor the S24 drift.
• epic(storage): scale complete ingest across cores and reach 1M edges/s #1387 — floor acceptance: the direct deadline at every supported rung, archived evidence, no-degradation shape criterion. Closes this issue.

Conditional: any need for a process-wide admission manager, a manifest format or a publication-protocol change stops the experiment that found it and routes to #1509 for a separate decision. Reopen #1448's adoption before production if incremental import would need full adjacency reconstruction, a query-admission contract change (ADR 0026), or substantial discarded preparation after publication conflicts.

Cautions that must survive into any design

1. Worker count, execution partitions and durable partition layout are three different numbers; hash or round-robin repartitioning does not replace recorded UUID splitters.
2. Streaming does not eliminate blocking stages; sorts accumulate or spill.
3. Keep a separate total-memory admission budget; a framework pool accounts for registered consumers, not every flowing batch or writer buffer.
4. CPU work in async needs explicit admission and cooperative cancellation.
5. Retain GraphForge's filesystem authority; tokio::fs removes no fsync cost.
6. No two schedulers may each claim all cores; cap build jobs the same way.
7. Delivered speedup at N workers is not a pool size; size pools by measuring the stage.
8. Do not move work across the ingest boundary to move the metric; the lifecycle budget beside the rate prevents it.
9. N independent processes are not one parallel ingest.
10. Achieved rates are not capacities, and logical bytes are not device bytes; state demand and capacity in the same units or it is not an argument.
11. Do not take ratios from a small fixture; use rung-scale attribution.

Standards that do not move

TCK green (cargo test -p graphforge-api --test bdd) · G500 S26 admits and completes · GDC suites green · every feature retained · fail-closed publication · no silently accepted corruption · the open-time content sweep is not tradeable. Spendable: recoverability-without-restart, eager verification whose refusal is duplicated at a consuming boundary, in-memory representations and buffer sizes, process boundaries and phase counts, constants priced for other hardware, wire formats with a version bump and the determinism suite green, byte-identical intermediates (ADR 0038).

Reversibility

Byte cuts, resource-policy unpinning, the encode seam and the contention characterization are two-way doors. Replacing sort/spill/memory accounting with DataFusion in shaping is one-way in practice and carries the heaviest gate (#1509).

Live views

#1387 floor · #1456 foundation · #1194 storage bytes · #1504 reuse evaluation · #1388 query workstream · #735 M5 · open PRs

Revision log

• rev 17, 2026-09-22 — docs(evidence): retain clean-ladder rung evidence the plan cites; the ladder root was deleted on 2026-09-21 #1530 merged (docs(evidence): retain clean-ladder rung evidence behind a verified post-run step (#1530) #1534): the retained-evidence paragraph above and the docs(evidence): retain clean-ladder rung evidence the plan cites; the ladder root was deleted on 2026-09-21 #1530 work-order check landed with it; the baseline of record points at docs/development/evidence/ladder/ instead of the deleted ladder root. No D1–D5 change.
• rev 16, 2026-09-22 — step 0 checked against the b6ffb088 S18–S22 ladder (S22 126,099 edges/s, 7.93× gap). perf(bench): make the ingest gates two-sided so improvements ratchet automatically (#1387 D1/D5) #1476 checked on close. No D1–D5 change.
• rev 15, 2026-09-21 — focused rewrite; revisions 10–14 archived to the repo; work order converted to native issue references; R14 continuation and retention rules carried forward. No D1–D5 change.
• rev 14, 2026-09-21 — step 0 prerequisite (fix(storage): S22 construction fails "control record exceeds bound" because the shape-end checkpoint ledger holds one entry per sealed segment (#1519) #1526 fix), evidence loss, continuation rule, explicit-exchange design routed to docs(storage): integrated construction reuse experiments, tradeoff matrix, and ADR (#1504) #1509.
• rev 13, 2026-09-19 — withdrew the ≤3.54 µs/edge derivation; matched-boundary accounting; perf(storage): attribute construction publication and fit it into the ingest floor budget #1481 owns publication.
• rev 12, 2026-09-19 — F1 and perf stat refuted rev 11's byte and scaling premises; D1–D5 ruled.

[DecisionNerd]

Rev 13, 2026-09-19 — maintainer-authorized follow-up to the independent review.

• D3 is resolved on encode in spike(storage): prove the Arrow/DataFusion seam on encode, with a go/no-go decision gate #1465. Seam viability permits a bounded shaping experiment, not the migration. The ≥2× parallelism criterion is removed rather than moved to another surface.
• Withdraw CPU ≤3.54 µs/edge and the mandatory 2.30× cut: N-process throughput speedup is not CPU/wall or a single-ingest ceiling. perf(bench): make the ingest gates two-sided so improvements ratchet automatically (#1387 D1/D5) #1476 now preserves valid regression gates and the separate direct 1M acceptance requirement.
• Replace the circular +1.1% “validation” with matched-boundary accounting and the actual whole-ingest deadline. Publication/setup must be included exactly once.
• Publication has its own bounded owner, perf(storage): attribute construction publication and fit it into the ingest floor budget #1481, in M5, as a native sub-issue and blocker of epic(storage): scale complete ingest across cores and reach 1M edges/s #1387. It re-measures the integrated publication path including CSR, attributes costs, and implements a justified repair or documents the budget disposition. The historical ~42% is a conditional budget warning, not a current-tree forecast.
• observe(storage): wire the region instrument to a caller, and fix the spec that would break it (#1387 D4) #1477 separates CPU concurrency, scheduler states, throughput speedup and overlapping-scope attribution. observe(storage): separate fsync contention from core contention in the N-way ingest probe (#1387) #1480 no longer treats loop devices/logical volumes on one array as storage isolation or elapsed-minus-CPU as fsync wait.
• Updated the current summaries of epic(storage): scale complete ingest across cores and reach 1M edges/s #1387/epic(storage): stop maintaining a hand-rolled dataflow engine; reuse Arrow/DataFusion where determinism and durability allow #1456. Historical observations remain attributed to their builds; the 1M floor, correctness contracts and epic(scale): close billion-edge scale and hardened interchange #735's canonical M5 role are unchanged.

This revision changes the plan and issue specifications only. No new benchmark, product-code change, or throughput gain is claimed.

[DecisionNerd]

Dependency follow-up to rev 13: execution prerequisites and parent closure gates are now recorded in native GitHub Blocked by / Blocking relationships, with the exact mapping in §5. #1480 is now a native child and blocker of #1387. #1477 precedes the new metric gates, encode seam and normalization/overlap work; #1480 precedes barrier work; both feed publication attribution. Partition/shaping decomposition (#1448) waits for skew/buffer bounds, the encode seam decision, overlap and barrier disposition. The separate #1194 byte-work closure relationships are restored for its open children. No dependency was added from #1480 to the permanent instrument or unconditionally from encode to #1439. #1464 remains a parent closure/disposition item, not an execution prerequisite for the refuted prefix-sum proposal. #735 and the 1M floor are unchanged.

[DecisionNerd]

Appended §13 with a Mermaid flowchart of the current native issue dependencies. Solid arrows show implementation prerequisites; dashed arrows show parent closure dependencies. No relationships or plan decisions changed.

[DecisionNerd]

#1480 measurement completed; report and reproducible evidence are in PR #1482 (report, machine-readable evidence).

Measured on OVHC-AGENCY using the preserved stock durable fa4dec51 binary and matched S18 inputs, cache preparation and resource limits:

Mean seconds per import	N=1 traced	N=16 traced b	N=16 traced c
Main scheduled running	18.058	29.674	30.853
Main runnable delay	0.084	2.410	2.438
Main synchronization blocking	2.500	52.105	54.697
Main scheduler-state unknown	0	0.078	0.070
Synchronization blocking inside shaping	1.775	40.687	41.164

Synchronization blocking is the largest observed contributor to this N-process contention loss. CPU duration also rises: untraced wait4 CPU is 19.852–20.204 s/import alone versus 36.157–36.167 s at N=16. The trace does not establish an increased instruction count or the mechanism behind that CPU increase.

Bounded intervention supported: continue #1452 with durability-preserving barrier amortization in shaping/spill publication. Shaping executes 6,554 directory and 2,739 file synchronization calls per process. Preserve crash/refusal contracts and measure the same durable path before claiming improvement. This is evidence for where to investigate, not a prediction of savings from removing barriers.

Controls pass for CPU, known sleeping delay, pinned scheduler contention, and exact file/directory sync counts. The report retains the EMFILE attempt, phase-incomplete pilots, and failed initial analysis. Unresolved wakeup gaps are explicitly unknown rather than silently counted as blocked; independent review verified the corrected accounting. Traced/untraced differences include run variability. All 258 archived artifact hashes matched the retained files.

No independent-storage isolation or durability ablation was used. Shared-array device service, ext4 journal behavior and directory locking remain unseparated. This measures historical-build validate, not current-main complete ingest or publication (#1481). It supplies no CPU/edge ceiling, single-parallel-ingest claim, or full-ladder qualification. #1477 should preserve explicit scheduler uncertainty in the reusable instrument.

[DecisionNerd]

Rev 14 — incorporated the completed #1480 work into the plan body, beyond the earlier evidence comment.

• Marked step 1 complete: docs(perf): attribute N-way ingest contention (#1480) #1482 merged as 52f29ad0, with green PR and merge-queue CI; observe(storage): separate fsync contention from core contention in the N-way ingest probe (#1387) #1480 is closed.
• Added §2.3.1 with measured synchronization/scheduler results, CPU scope, explicit unknown intervals and the historical-build/validate-only limitations. Linked the merged report and machine-readable evidence in §12.
• Recorded the supported next intervention: bounded durability-preserving barrier amortization in shaping/spill publication (perf(storage): sealing one spill costs eight durability barriers, serialised on the directory inode #1452), with no predicted speedup or fsync-removal authorization.
• Updated the open-unknowns table, dependency summary and Mermaid diagram. perf(storage): sealing one spill costs eight durability barriers, serialised on the directory inode #1452's observe(storage): separate fsync contention from core contention in the N-way ingest probe (#1387) #1480 prerequisite is satisfied; perf(storage): attribute construction publication and fit it into the ingest floor budget #1481 still waits on observe(storage): wire the region instrument to a caller, and fix the spec that would break it (#1387 D4) #1477. Existing native relationships were preserved.

The integrated baseline remains outstanding. The 1M floor, D1–D5 decisions, publication scope and #735 close gate are unchanged.

[DecisionNerd]

Dependency update: #1452 is complete — PR #1494 squash-merged as 9614ffb0 (PR CI and merge-queue CI green), issue closed with per-syscall duration evidence (comment). §5's table row for #1452 ("None; #1480 completed") is satisfied and done; the measured result supersedes the §0 "pursue" directive: directory syncs −41.5% (S18) / −48.1% (S18-half) per validate, file barriers untouched, no durability contract changed. This clears one of #1448's open execution blockers (#1439, #1445, #1465, #1472 remain). Retirement-side unlink batching was deliberately left out of #1452's scope and is available as a small follow-up on the same seam if the rev 15 balance wants it.

[DecisionNerd]

Construction foundation #1456 is now closed, with all ten native children and all listed blockers closed. Acceptance mapping and dispositions. Final child #1439 landed in #1503 as eeda146 with green exact-head PR CI and required merge-queue CI; the CodeRabbit finding was verified, repaired, and confirmed resolved. This does not close #1387 or this plan: the integrated S18–S22 baseline, direct throughput floor, and remaining performance work retain their existing owners. The explicit F5 measured-bound no-go is already recorded in §5.

[DecisionNerd]

Plan edit — #1448 re-scope and the encode-seam disposition (three local edits, no revision bump):

• §5 table, perf(storage): make one complete ingest scale with available cores #1448 row: the five historical prerequisites are recorded as satisfied and spike(storage): evaluate sorting and partitioning reuse for construction (#1504) #1506, spike(storage): evaluate Tokio/DataFusion scheduling and cancellation for construction (#1504) #1508 are added as execution prerequisites. The seam pass no longer "permits a bounded shaping experiment": spike(storage): prove the Arrow/DataFusion seam on encode, with a go/no-go decision gate #1465 landed in spike(storage): qualify the bounded DataFusion encode seam #1496 as viable but a performance no-go (about 7.6% slower encode; baseline retained), so this step authorized none.
• §5 step 6, new closing paragraph: records that disposition, the opening of epic(storage): evaluate DataFusion, Tokio, and Arrow replacements for custom construction machinery #1504 (2026-09-20; children docs/research(storage): inventory construction machinery and record comparison protocol (#1504) #1505–docs(storage): integrated construction reuse experiments, tradeoff matrix, and ADR (#1504) #1509 under the docs/research(storage): inventory construction machinery and record comparison protocol (#1504) #1505 protocol), and the routing: perf(storage): make one complete ingest scale with available cores #1448 becomes "shaping over partitions", takes its mechanism from the spike(storage): evaluate sorting and partitioning reuse for construction (#1504) #1506/spike(storage): evaluate Tokio/DataFusion scheduling and cancellation for construction (#1504) #1508 dispositions, and is natively blocked by them, because a hand-rolled partition-lane scheduler is exactly what those spikes evaluate. perf(storage): make one complete ingest scale with available cores #1448 stays the epic(storage): scale complete ingest across cores and reach 1M edges/s #1387 child that owns shaping parallelism. epic(storage): evaluate DataFusion, Tokio, and Arrow replacements for custom construction machinery #1504 is not an M5 issue and gates neither epic(storage): scale complete ingest across cores and reach 1M edges/s #1387 nor this plan; only its two mechanism spikes gate perf(storage): make one complete ingest scale with available cores #1448.
• §13: epic(storage): evaluate DataFusion, Tokio, and Arrow replacements for custom construction machinery #1504 added as an entry point with that scope.

Evidence for the "largest lever" claim is the current-tree region tree from the #1456 close evidence (quiet S19, 6d6b7f17): shaping 23.88 s wall / 21.39 s CPU (0.90 cores, 48% of validate); canonical encoding 15.51 s / 14.52 s (0.94 cores, 31%). The integrated S18–S22 baseline (step 0) remains outstanding and is the before-measurement for #1448 and #1481.

Order change only. D1–D5, the 1M floor, publication scope (#1481) and the #735 close gate are unchanged. Reversible: if the maintainer prefers hand-rolled partition lanes to proceed in parallel with #1504, remove the two blockers and the row reverts to "no open prerequisites".

[DecisionNerd]

#1481 disposition: publication attributed at S18+S22; fail-closed reader-preparation repair landed; no throughput gain claimed

PR #1543 (branch perf/1481-publication-attribution) carries the final two pieces: the bounded repair and the evidence.

Current integrated-tree baseline (main aaf20d83 era, re-baselined because #1502/#1531/#1533/#1534/#1538 landed after the 097e763 S18 observations; recorded inputs SHA-verified; BenchExec CPUs 0–15 / 4 GiB; quiet guard before/during/after):

Rung	n	Ingest wall median	µs/edge	÷ 1M/s allowance	Publish median	Publish µs/edge
S18	3	31.818 s	7.586	7.59×	2.394 s	0.571
S22	3	547.803 s	8.163	8.16×	42.014 s	0.626

Clean S22 publish decomposition: cas_install 16.3 s, generation_commit 11.6 s, hydration 7.3 s, publication_receipt 5.7 s, read_authority 0.7 s — same structure the contended run indicated, now on accepted quiet timings. Every S22 run reaches the enforced 4 GiB RSS ceiling; block I/O ≈ 190 GB read / 180 GB write per run.

Repair. Reader preparation (hydration + read_authority) now runs against the durable, lease-verified candidate immediately before CURRENT via the publisher's pre-CURRENT preparation seam; cancellation stays pollable to the commit point. A candidate that cannot be hydrated or authorized fails closed (CURRENT unchanged) instead of committing a generation needing post-commit recovery. Receipts nest the two regions under generation_commit for fresh publications; replay keeps the sibling shape.

A/B under the predeclared criterion (benefit > max(3% of baseline median, 2× baseline spread), alternating quiet pairs, binaries differing only by the repair commit): publish wall unchanged within noise (S18 2.394→2.398 s; S22 42.0→41.7 s median). Whole-ingest medians: S18 −0.77 s (threshold 9.2 s), S22 −13.2 s (threshold 91.3 s) — not demonstrated; no throughput gain is claimed. The repair is justified and kept for the fail-closed publication behavior (#1481 scenario 2).

Remaining budget. Complete ingest is 7.4–8.2 µs/edge against the 1.00 µs/edge floor; publication alone is 0.57–0.63 µs/edge. No publication-side change closes that gap; the deficit remains owned by the parallel-ingest workstreams under #1387, which stays the floor close gate. #1481's remaining budget arithmetic and the S22 reconciliation are recorded in docs/development/evidence/publication-attribution-1481.{md,json}.