Skip to content

docs(development): streaming construction plan rev 11 — the #1387 floor, per resource (#1387, #1456) - #1475

Merged
DecisionNerd merged 1 commit into
mainfrom
docs/streaming-construction-plan-rev11
Sep 18, 2026
Merged

DecisionNerd merged 1 commit into
mainfrom
docs/streaming-construction-plan-rev11

Conversation

@DecisionNerd

@DecisionNerd DecisionNerd commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

What

Writes the Streaming Construction Plan into the repository as docs/development/streaming-construction-plan.md (rev 11) and indexes it from docs/engineering/README.md. Until now the plan of record for #1387 and #1456 existed only as a Claude artifact plus a pointer file under ~/.claude, unreadable by anyone outside a Claude session.

Location: docs/development/, the established home for evidence-bearing plans (perf-g500-ladder.md, m4-exit-evidence.md, the bazel-migration-* family). Not docs/adr/ — it records no single decision and would trip the ADR frontmatter/index policy (ADR 0038, #1390). Not a docs-site page — docs-site/scripts/sync-content.mjs publishes an explicit allowlist of reader-facing development pages and this is an internal engineering plan like its siblings, so no sidebar entry.

Why rev 11

Rev 10 was red-teamed on 2026-09-18 and scored 10/24. This revision fixes the five P0 and five P1 findings; the map is §0 of the document. The load-bearing changes:

  • Bytes bind first. 1.2 KB read + 0.7 KB written per edge, logical and constant across S22–S24 (measured). At 1M edges/s that is 1.21 + 0.71 GB/s in the ideal-cache case against a harness-measured ~360 + ~300 MB/s device: 2.4–3.4× over before any amplification, ~18× at S24's physical ratios. Sequenced first.
  • "CPU banked" held only single-threaded. F2 probe on a quiet host (method in §2.3): SHA-256 scales 15.07× on 16 threads, zstd decode 7.96×, GNU sort 7.27×. The sort/decode-shaped majority of the path has a ceiling below the 8.13 single-thread-equivalent cores the floor needs at today's CPU/edge. The CPU cut is not optional.
  • "71% serial" is seal's share of CPU, not an Amdahl fraction; the serial fraction has never been measured and CPU/wall cannot measure it (the landed concurrency_attribution module says so itself). §5 step 1 specifies the three numbers the instrument must report.
  • The refuted prefix-sum "critical path" is removed and the sequencing question is recorded as a maintainer decision (§9 D2), not decided here.
  • Merge-queue state is part of the plan (§4): eight of twelve PRs landed; five open, perf(api,bench): derive concurrency from the machine; drop the adjacent-RSS growth gate #1466 held behind perf(storage): publish the adjacency CSR with the generation instead of rebuilding it per process #1453.
  • 14–18% of ingest wall is outside every timed operation on all nine measured rungs (§3.3); publish (28.2 s at S22 at 0.58 cores) is on the "nothing sacred" list.

What this does not do

  • Does not change epic(storage): scale complete ingest across cores and reach 1M edges/s #1387's floor. Five per-resource budgets are a proposal in §9 D1; the decision is the maintainer's.
  • Does not run a ladder. Five PRs are still landing; a ladder now would measure a baseline about to change. The only new measurement is the host probe, reported as measured with its method and a known-positive check.
  • Does not edit M5's description beyond a pointer line (it is a scope statement, not a plan).

Gates run

make pre-push-fast green; scripts/ci/adr-index.py check (34 active / 3 superseded agree); scripts/ci/test-text-checkout-policy.py (3,592 paths LF); scripts/source_size_policy.py (0 violations).

Refs #1387, #1456, #1462, #1465, #1194.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…o as rev 11 (#1387, #1456)

The plan of record for the #1387 ingest floor and the #1456 foundation lived
only as a Claude artifact (rev 10) plus a pointer file in ~/.claude. Neither is
readable outside a Claude session. This lands it as
docs/development/streaming-construction-plan.md and indexes it from
docs/engineering/README.md beside the bazel-migration family.

Rev 11 is the refactor after the 2026-09-18 red-team critique of rev 10:

- bytes per edge (1.2 KB read + 0.7 KB written, logical, constant across
  S22-S24) bind the floor before cores do; sequenced first, not last
- the CPU budget is met single-threaded only: an F2 probe on the quiet host
  measures 15.07x SMT scaling for SHA-256 but 7.27x (sort) and 7.96x (zstd
  decode) on 16 threads, so the sort/decode-shaped majority of the path has a
  ceiling below the 8.13 cores the floor needs at today's CPU/edge
- "71% serial" relabelled as seal's share of CPU; the instrument must report
  CPU-busy, off-CPU wait and achieved parallelism separately before any A/B
- the refuted prefix-sum critical path removed; sequencing recorded as a
  maintainer decision, not decided here
- merge-queue state recorded (eight of twelve PRs landed; five open)
- 14-18% of ingest wall is outside every timed operation, measured on all
  nine rungs; publish (28.2 s at S22, 0.58 cores) added to "nothing sacred"
- the floor is unchanged; five per-resource budgets are a marked proposal

Every number is labelled measured or estimated and the two are not mixed in
one table. No ladder was run; the only new measurement is the host probe.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 18, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: CurateLabs/graphforge/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 96aa9f98-ca5f-4c7a-886a-58b2a863a164

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core source code changes documentation Improvements or additions to documentation testing Test coverage and testing infrastructure tooling Developer tooling and automation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant