Define streaming indexed recursive WARP architecture

[flyingrobots]

Summary

• Defines the v20 architecture that eliminates production full-state materialization.
• Uses Roaring ordinals as the selection plane over bounded CAS address pages.
• Specifies stream-only attachments and cycle-safe recursive WARP graph traversal.

Issue

Refs #824

Test plan

• npm run lint:md -- --ignore CHANGELOG.md
• npm run lint:md:code
• npm run lint:docs-topology
• Complete static gates and stable unit-test shards passed in pre-push: 506 files; 7,079 tests passed; 2 skipped.

ADR checks

• This PR does not implement ADR 2 without satisfying ADR 3
• If this PR touches persisted op formats, I linked the ADR 3 readiness issue
• If this PR touches wire compatibility, I confirmed canonical-only ops are still rejected on the wire pre-cutover
• If this PR touches schema constants, I confirmed patch and checkpoint namespaces remain distinct

This is a design-only change and does not modify persisted op formats, wire compatibility, or schema constants.

[coderabbitai]

Warning

Review limit reached

@flyingrobots, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 46 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 6f74bc8e-275e-456f-90d0-e2e518d3e8fb

📥 Commits

Reviewing files that changed from the base of the PR and between 59c97a2 and 8b5e546.

📒 Files selected for processing (1)

• docs/plans/streaming-indexed-recursive-warp.md

📝 Walkthrough

Walkthrough

The PR adds a 943-line architecture plan for v20.0.0. It defines bounded streaming and retained-graph designs, recursive traversal, migration constraints, typed failures, enforcement tests, implementation slices, acceptance criteria, and validation commands.

Changes

Streaming Indexed Recursive WARP

Layer / File(s)	Summary
Invariants and architectural constraints docs/plans/streaming-indexed-recursive-warp.md	Defines the no-full-materialization invariant, release posture, current whole-state contracts, and bounded-operation classifications.
Retained graph and streaming execution model docs/plans/streaming-indexed-recursive-warp.md	Specifies retained manifests, ordinal and selection structures, stream-first payload APIs, recursive traversal, and bounded global operations.
Migration, failure, and behavioral enforcement docs/plans/streaming-indexed-recursive-warp.md	Describes bounded migration, typed obstructions, source ratchets, required behavioral proofs, and fixture policies.
Implementation slices and release acceptance docs/plans/streaming-indexed-recursive-warp.md	Lists implementation slices, issue dispositions, acceptance criteria, open design questions, and validation commands.

Estimated code review effort: 2 (Simple) | ~10 minutes

Poem

I’m a rabbit with a streaming plan,
No giant state heap where burdens span.
Roots stay indexed, queues hop light,
Recursive paths remain in sight.
Tests guard each bounded way—
V20 blooms from docs today!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise and accurately summarizes the main change: defining the streaming indexed recursive WARP architecture.
Description check	✅ Passed	The description matches the template with Summary, Issue, Test plan, and ADR checks, and includes a same-repo issue reference.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/plans/streaming-indexed-recursive-warp.md`:
- Around line 511-533: Specify a bounded capacity for the explicit async work
queue used by recursive traversal, including admission backpressure when full
and external spill/resume behavior for overflow. Update the traversal budget
sections associated with the queue and the referenced sections around the queue
implementation so pending child work cannot grow with graph fan-out or violate
the stated heap bound.
- Around line 646-652: The migration plan must specify a bounded decoder for
oversized legacy replay values, rather than merely chunking the CAS stream
before synchronous decoding. Update the legacy component streaming steps to use
an incremental parser or external-memory conversion path that never materializes
the full CBOR value, and revise the associated test to exercise the actual
23,995,927-byte oversized shape while enforcing page/shard bounds.
- Around line 600-604: Update the Export design to preserve deterministic scan
order despite bounded-concurrent payload reads: assign sequence numbers and
buffer completed payloads until the next expected sequence is available, or use
a single ordered writer that applies backpressure before opening more payloads.
Ensure the streaming sink receives records strictly in scan order while
retaining the concurrency bound.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 30dfaaab-3446-4140-8ca0-03b7c86b1bda

📥 Commits

Reviewing files that changed from the base of the PR and between 2fc5b7d and 59c97a2.

📒 Files selected for processing (1)

• docs/plans/streaming-indexed-recursive-warp.md

📜 Review details

⏰ Context from checks skipped due to timeout. (8)

• GitHub Check: coverage-threshold
• GitHub Check: test-bun
• GitHub Check: test-node (22)
• GitHub Check: test-deno
• GitHub Check: type-firewall-lint
• GitHub Check: type-firewall-generated-sdk
• GitHub Check: preflight
• GitHub Check: v19 base/head performance

🧰 Additional context used

🪛 LanguageTool

docs/plans/streaming-indexed-recursive-warp.md

[style] ~118-~118: The double modal “required bounded” is nonstandard (only accepted in certain dialects). Consider “to be bounded”.
Context: ...streaming. - #632 through #634 required bounded node, property, neighborhood, and trave...

(NEEDS_FIXED)

---

[style] ~681-~681: The double modal “Required retained” is nonstandard (only accepted in certain dialects). Consider “to be retained”.
Context: ... retained-root-unavailable | Required retained root is absent | Rebuild or migrate | |...

(NEEDS_FIXED)

---

[grammar] ~725-~725: Use a hyphen to join words.
Context: ... to the CBOR decoder. - A multi-gigabyte generated attachment streams through a b...

(QB_NEW_EN_HYPHEN)

🔇 Additional comments (1)

docs/plans/streaming-indexed-recursive-warp.md (1)

504-507: 🎯 Functional Correctness

Specify root-pinned continuation semantics.

When a budget ends, “continue with cursor” must preserve the resolved live roots, visited-basis set, pending queue, and ordering state. Otherwise resumption can read a newer live root or duplicate/skip records. Make the cursor/receipt binding explicit and test resume across live-locator changes.

Also applies to: 566-576, 675-689

[github-actions]

Release Preflight

• package version: 19.0.0
• prerelease: false
• npm dist-tag on release: latest
• npm pack dry-run: passed
• jsr publish dry-run: passed

If this PR is from a release/* branch and merges to main, Main Push Release Branch Check will run final preflight and create v19.0.0. A maintainer who is a JSR @git-stunts scope member must then dispatch the Release workflow manually.

[github-actions]

Release Preflight

• package version: 19.0.0
• prerelease: false
• npm dist-tag on release: latest
• npm pack dry-run: passed
• jsr publish dry-run: passed

If this PR is from a release/* branch and merges to main, Main Push Release Branch Check will run final preflight and create v19.0.0. A maintainer who is a JSR @git-stunts scope member must then dispatch the Release workflow manually.

[github-actions]

Release Preflight

• package version: 19.0.0
• prerelease: false
• npm dist-tag on release: latest
• npm pack dry-run: passed
• jsr publish dry-run: passed

If this PR is from a release/* branch and merges to main, Main Push Release Branch Check will run final preflight and create v19.0.0. A maintainer who is a JSR @git-stunts scope member must then dispatch the Release workflow manually.