Repository navigation
perf(orchestration): parallel multi-query fan-out and wave context sliding window - #869
2001anshkaushik wants to merge 7 commits into
Conversation
…iding window Gap-2: Parallel multi-query fan-out in DocumentStoreBase._queryDocuments - Replace sequential for-loop over question.questions with ThreadPoolExecutor dispatch for SEMANTIC/PROMPT/QUESTION and KEYWORD question types - Single-query path bypasses executor entirely (zero overhead for 1 sub-query) - Concurrent writes to the shared documents dict serialised via threading.Lock held only during the merge step, not during the blocking I/O call itself - Embedding validation runs before any futures are submitted (fail-fast) - _MAX_PARALLEL_QUERIES = 8 caps thread count for large fan-out pipelines - Expected latency: N x p99 -> ~p99 (e.g. 3x200ms: 600ms -> ~210ms) Gap-1: Server-side sliding-window context budget in RocketRideDriver - _prune_wave_context() with two complementary eviction strategies: 1. Sliding window (context_window_waves, default 5) 2. Character budget (wave_context_budget_chars, default 12_000) - Minimum 1 wave always retained to prevent empty-context LLM calls - Both configurable via node config fields; budget=0 disables char limit - Called after every execute_wave append Tests: 20 new tests in test_document_store_parallel.py (20/20 pass) - Correctness, parallelism timing, thread safety stress test, edge cases
Defect 1 (Critical): Empty SEMANTIC/PROMPT/QUESTION query list crashed with
ValueError: max_workers must be greater than 0.
The 'else' branch (intended for N>1 queries) was entered when len(queries)==0,
computing min(8, 0)=0 and passing that to ThreadPoolExecutor.
Fix: add explicit 'if not queries: pass' guard before the == 1 / else split,
making the three paths: empty → no-op, single → direct call, N>1 → executor.
Defect 2 (Critical): Same max_workers=0 crash for KEYWORD type with empty
question list. Same fix applied to the KEYWORD branch.
Defect 3 (Logic): The KEYWORD branch was written as an independent 'if' in the
original code, making KEYWORD and GET true peers. The refactor preserved this
structure correctly for the KEYWORD 'if', but the relationship to the GET
'elif' was ambiguous and would have caused GET to silently no-op if a future
refactor collapsed the two 'if' blocks into an 'elif' chain. Added explicit
structural comment explaining WHY KEYWORD must remain an independent 'if' and
must never be made 'elif' off the SEMANTIC block.
Confirmed: GET is currently reachable because the KEYWORD 'if' is False when
type==GET, so the elif GET arm fires correctly. But the zero-query KEYWORD
path fell to else:pass instead of elif GET — this silent no-op was the latent
bug. After fix: KEYWORD(queries=[]) → returns {} and never touches GET branch.
Tests: 6 regression tests added to TestPreFlightAuditRegressions (26/26 pass)
- test_semantic_empty_questions_no_crash
- test_prompt_empty_questions_no_crash
- test_keyword_empty_questions_no_crash
- test_keyword_empty_does_not_call_search
- test_get_type_is_reachable
- test_get_type_not_affected_by_keyword_zero_queries
|
No description provided. |
📝 WalkthroughWalkthroughAdds configurable wave-history pruning to RocketRideDriver (sliding-window then character-budget eviction, returned as a shallow copy used only for planning prompts) and refactors DocumentStoreBase._queryDocuments to parallelize multi-sub-query SEMANTIC/KEYWORD execution with a max-worker cap and pre-dispatch embedding validation. Adds tests and a pytest rocketlib stub. ChangesPerformance and Context Management
Sequence Diagram(s)sequenceDiagram
participant Caller
participant DocumentStoreBase
participant ThreadPoolExecutor
participant SearchBackend as searchSemantic/searchKeyword
Caller->>DocumentStoreBase: _queryDocuments(question)
DocumentStoreBase->>DocumentStoreBase: validate embedding_model (pre-flight)
DocumentStoreBase->>ThreadPoolExecutor: submit sub-query tasks
ThreadPoolExecutor->>SearchBackend: execute sub-query
SearchBackend-->>DocumentStoreBase: sub-query results
DocumentStoreBase->>DocumentStoreBase: merge results under lock
DocumentStoreBase-->>Caller: aggregated documents
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@nodes/src/nodes/agent_rocketride/rocketride_agent.py`:
- Around line 281-286: The current code prunes the in-place `waves` list via
`_prune_wave_context(waves)`, which loses data needed for `trace['waves']` and
the fallback `_synthesize(...)`; change the flow to keep an unmodified full
history (e.g., `full_waves = waves` or `history_waves`) and only pass a pruned
copy to planning: call something like `planning_waves =
self._prune_wave_context(list(full_waves))` (or have `_prune_wave_context`
return a pruned copy), then use `planning_waves` for `plan_wave()` while leaving
`full_waves` intact for `trace['waves']` and `_synthesize(...)`; ensure all
references that produce the trace or call `_synthesize` use the unpruned
`full_waves`.
In `@packages/ai/tests/ai/common/test_document_store_parallel.py`:
- Around line 339-345: The test currently validates a local reimplementation
(_DriverStub._prune_wave_context) instead of the real node, so update the tests
to import and exercise the actual RocketRideDriver from
nodes/src/nodes/agent_rocketride/rocketride_agent.py and exercise its
integration path (e.g., call RocketRideDriver._run or public run entry) instead
of using _DriverStub; if the suite cannot import nodes, either move these test
cases to a test target/config that has nodes on PYTHONPATH or make the nodes
package importable from this test suite, then delete the stubbed
_prune_wave_context usage and assert behavior via the real RocketRideDriver.
- Around line 202-220: The test's rigid bound (elapsed < 2 *
SIMULATED_QUERY_LATENCY_S) is too tight and causes CI flakes; change the
assertion to compare against a sequential baseline measured in the same test
run: measure sequential_time by calling store._queryDocuments(question)
n_queries times in a loop (using the same question and store) to get a realistic
sequential_elapsed, then assert that the parallel elapsed measured in
test_parallel_execution_is_faster_than_sequential is significantly less than
that sequential_elapsed (e.g., elapsed < sequential_elapsed * 0.9) instead of
comparing to 2 * SIMULATED_QUERY_LATENCY_S.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: c5106df0-8675-45bd-a039-f5c65adcbaec
📒 Files selected for processing (3)
nodes/src/nodes/agent_rocketride/rocketride_agent.pypackages/ai/src/ai/common/store.pypackages/ai/tests/ai/common/test_document_store_parallel.py
…ting patterns
Fix 1 — State mutation: _prune_wave_context now returns a pruned copy
- _prune_wave_context(waves) -> List: returns a shallow copy with window
and char-budget eviction applied; the master waves list is never touched
- _run() computes context_waves = _prune_wave_context(waves) at the top of
each planning iteration and passes context_waves to plan_wave()
- master waves (= trace['waves']) grows unbounded as the canonical audit
trail; _synthesize() at max-wave fallback still receives the full trace
- Removed the post-execute_wave prune call which was the mutation site
Fix 2 — CI flake: replace wall-clock timing assertion with Barrier proof
- Deleted test_parallel_execution_is_faster_than_sequential (wall-clock
comparisons are non-deterministic on loaded CI runners / Windows scheduler)
- Added test_parallel_execution_is_structurally_concurrent: uses
threading.Barrier(n=3, timeout=5s) to prove all 3 futures execute
concurrently — a serial executor would deadlock and raise BrokenBarrierError
- This proof is deterministic and immune to OS scheduler jitter
Fix 3 — Stub anti-pattern: replace _DriverStub with real RocketRideDriver test
- Deleted TestWaveSlidingWindow from test_document_store_parallel.py (it
tested a verbatim stub copy of the algorithm, not the production class)
- Created nodes/test/agent_rocketride/test_prune_wave_context.py following
the established project pattern (cf. nodes/test/astra_db/test_convert_filter.py)
— uses RocketRideDriver.__new__ to bypass iGlobal init, sets only
_context_window_waves and _wave_context_budget_chars, then calls the
real production method
- 4 new test classes: TestReturnsCopy (copy semantics), TestSlidingWindow,
TestCharacterBudget, TestCombinedStrategies
Results:
packages/ai tests: 16/16 passed (0.67s)
nodes/test/agent_rocketride: 19/19 passed (2.17s)
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@nodes/test/agent_rocketride/test_prune_wave_context.py`:
- Around line 77-115: The test file contains Unicode multiplication signs (×,
U+00D7) in comments (e.g., in
TestReturnsCopy::test_master_list_never_mutated_by_budget_eviction and other
similar tests) which Ruff flags; replace those characters with the ASCII
asterisk (*) (or "x") in the comment text so the comments read e.g. "3 waves *
100 chars = 300 chars → budget eviction will fire" to avoid encoding/ambiguity
issues while leaving all test logic and identifiers like _prune_wave_context,
TestReturnsCopy, and the named test methods unchanged.
- Line 25: Replace the typing imports and usages with Python 3.10+ native
generics: remove "from typing import Any, Dict, List" and switch all type
annotations that use Dict and List to lowercase forms (e.g., change Dict[str,
Any] to dict[str, Any] and List[Dict[str, Any]] to list[dict[str, Any]]); update
any function signatures, variable annotations or test fixtures in this file that
reference Dict, List, or Any accordingly so only "Any" usage remains either
imported or replaced with concrete types if possible.
In `@packages/ai/tests/ai/common/test_document_store_parallel.py`:
- Line 24: Replace the legacy typing imports and usages: remove "from typing
import Any, Callable, Dict, List, Optional, Tuple" and instead import Any and
Optional from typing only if needed, import Callable from collections.abc, then
update all type annotations to use native generics (e.g., change Dict[str, Any]
→ dict[str, Any], List[Doc] → list[Doc], Tuple[str, int] → tuple[str, int]) and
replace typing.Callable usages with collections.abc.Callable; update any
occurrences of those symbols in the test file (references like Dict, List,
Tuple, Callable) to the new forms so the test uses Python 3.10+ native generics.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 8d9dbca8-66a1-4d53-a078-955737a22c60
📒 Files selected for processing (3)
nodes/src/nodes/agent_rocketride/rocketride_agent.pynodes/test/agent_rocketride/test_prune_wave_context.pypackages/ai/tests/ai/common/test_document_store_parallel.py
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/ai/tests/ai/common/test_document_store_parallel.py`:
- Around line 38-47: The current stub unconditionally inserts a minimal
_rocketlib module using sys.modules.setdefault which can leave tests with an
incomplete module if another test later expects additional attributes; instead,
fetch any existing module = sys.modules.get('rocketlib') and if none assign the
new ModuleType, otherwise merge missing attributes: ensure _IInstanceBase is
added only when getattr(existing, 'IInstanceBase', None) is None (use setattr
for 'IInstanceBase'), so the stub becomes order-independent (update the code
around _rocketlib, _IInstanceBase, and the sys.modules.setdefault call to
perform this safe-merge logic).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 9c4903d0-2d76-40c8-93d7-1b2e9813c138
📒 Files selected for processing (2)
nodes/test/agent_rocketride/test_prune_wave_context.pypackages/ai/tests/ai/common/test_document_store_parallel.py
…st modules
Root cause
----------
Both test files injected rocketlib stubs inline in their module bodies using
sys.modules.setdefault(). The first file collected by pytest wins; the second
file's setdefault() is a no-op. If test_document_store_parallel.py loaded
first, it registered only IInstanceBase — missing tool_function and
rocketlib.types — causing an ImportError when test_prune_wave_context.py
subsequently imported RocketRideDriver.
The plan's proposed fix (copy the comprehensive stub into both files) was
rejected because it creates two copies of the same module stub. Any new
symbol added to the rocketlib interface would require updating two places —
exactly the drift that leads to 'works on my machine' CI failures.
Fix
---
Move the stub to conftest.py at the repository root. pytest guarantees that
conftest.py files are imported before any test module body executes,
regardless of collection order or parallelism. conftest.py is the canonical
location for session-scoped setup code in pytest projects.
conftest.py (NEW) — single source of truth for all rocketlib stubs:
- rocketlib.IInstanceBase — no-op base class
- rocketlib.tool_function — no-op decorator (bare and called forms)
- rocketlib.types — sub-module with IInvokeOp/IInvokeTool/
IInvokeMemory stubs
test_document_store_parallel.py — inline stub removed; replaced with a
comment pointing to conftest.py explaining the pattern prohibition.
test_prune_wave_context.py — same removal; same explanatory comment.
Verification
------------
Both orderings confirmed green:
store first: 35/35 passed (0.85s)
prune first: 35/35 passed (0.78s)
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/ai/tests/ai/common/test_document_store_parallel.py`:
- Line 36: Remove the unused import of the sys module in the test file: delete
the line importing sys from
packages/ai/tests/ai/common/test_document_store_parallel.py (the standalone
"import sys" at top of the file) since sys is not referenced elsewhere in the
file.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: c3a7244d-8c5e-4956-a8cd-7c1fd12c76e5
📒 Files selected for processing (3)
conftest.pynodes/test/agent_rocketride/test_prune_wave_context.pypackages/ai/tests/ai/common/test_document_store_parallel.py
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/ai/tests/ai/common/test_document_store_parallel.py`:
- Around line 197-201: Test assumes 3 concurrent threads; change n_queries to
honor the configured worker cap by setting n_queries = min(3,
_MAX_PARALLEL_QUERIES) (or read the executor's max workers if available) before
creating threading.Barrier, and use that variable when instantiating barrier so
the rendezvous count never exceeds the executor's worker limit (adjust imports
or access to _MAX_PARALLEL_QUERIES from the module where it's defined if
necessary).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: ebce3529-2bf1-490f-afbf-1a4d20baddc5
📒 Files selected for processing (1)
packages/ai/tests/ai/common/test_document_store_parallel.py
|
@coderabbitai resume |
✅ Actions performedReviews resumed. |
There was a problem hiding this comment.
♻️ Duplicate comments (1)
packages/ai/tests/ai/common/test_document_store_parallel.py (1)
197-201:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winGuard against vacuous concurrency proof when worker cap is 1.
At Line 197,
Barrier(1)can pass immediately, so the test may succeed without proving parallel dispatch. Skip when max workers are below 2.♻️ Suggested fix
- n_queries = min(3, DocumentStoreBase._MAX_PARALLEL_QUERIES) + max_workers = DocumentStoreBase._MAX_PARALLEL_QUERIES + if max_workers < 2: + pytest.skip('Structural parallelism is not expected when max workers < 2') + n_queries = min(3, max_workers)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/ai/tests/ai/common/test_document_store_parallel.py` around lines 197 - 201, The test creates a threading.Barrier sized by n_queries derived from DocumentStoreBase._MAX_PARALLEL_QUERIES which is vacuous when that cap is 1; guard the test by checking DocumentStoreBase._MAX_PARALLEL_QUERIES and skip the test early if it's less than 2 (e.g. call pytest.skip with a brief message) before computing n_queries and constructing the barrier so the barrier check actually validates parallel dispatch.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@packages/ai/tests/ai/common/test_document_store_parallel.py`:
- Around line 197-201: The test creates a threading.Barrier sized by n_queries
derived from DocumentStoreBase._MAX_PARALLEL_QUERIES which is vacuous when that
cap is 1; guard the test by checking DocumentStoreBase._MAX_PARALLEL_QUERIES and
skip the test early if it's less than 2 (e.g. call pytest.skip with a brief
message) before computing n_queries and constructing the barrier so the barrier
check actually validates parallel dispatch.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 6feafd7d-48fe-4c19-8182-f5c51475ca34
📒 Files selected for processing (1)
packages/ai/tests/ai/common/test_document_store_parallel.py
|
Wave was intended to operate exactly how you implemented it, running multiple subnodes concurrently. However, a problem came up in testing that needs to be fixed in the c++ engine before we can merge. It is a corner case, but it is pretty serious.. Most python nodes assume the sequential architecture. However, this makes them parallel which has the capability to corrupt "temporary" data that they are holding. The solution is, in the C++ engine, we need to ensure that no node is shared between wave partners, or lock a mutex so a node runs to completion prior to returning. Either way, it requires a backend change. Neither is too terfibly difficuly. |
dsapandora
left a comment
There was a problem hiding this comment.
This PR should not merge yet. The wave feature is implemented and scoped correctly at the store layer — fan-out via ThreadPoolExecutor in _queryDocuments with a properly-scoped merge lock, and _prune_wave_context() returning a fresh copy per planning iteration instead of mutating waves in place (fixing CodeRabbit's line-286 flag while keeping trace['waves'] and the synthesis fallback intact). The Barrier-based concurrency test is a solid replacement for a flaky timing check, and the invariant tests around 'master list never mutated' are the right shape for this kind of change.
However, testing surfaced a serious corner case that blocks this PR: most Python nodes were written assuming the engine executes sequentially, and several hold temporary/mutable state during execution. Running wave partners concurrently — which is exactly what this PR correctly implements — exposes those nodes to being shared across concurrent wave partners, which can corrupt that temporary state. This is a real regression introduced by turning previously-sequential execution parallel, and it isn't something the current Python-side tests exercise: the concurrency proof and invariant tests here cover the document store's merge safety, not node-level state isolation.
The fix has to land in the C++ engine, not in this PR's Python code: either (a) ensure no node instance is shared between wave partners (give each wave its own node instances), or (b) hold a per-node mutex so a node runs to completion before it can be re-entered by another wave. Either approach looks workable and shouldn't be a heavy lift, but it's a backend change and a prerequisite for merging wave-parallel execution safely — without it, this corner case can produce corrupted results in production under the exact concurrency this PR enables. New tests exercising node-level state isolation (or mutex serialization) should land alongside whichever fix is chosen, since the current suite doesn't cover this failure mode on any platform.
Once that's addressed, the remaining items are minor and non-blocking:
- The SEMANTIC and KEYWORD fan-out blocks in
store.pyare near-identical; worth a shared helper if this file is touched again. - The 'Defect 3' framing in
test_document_store_parallel.py(GET allegedly unreachable via the KEYWORDelif) isn't an actual bug before or after this change —typeis a single enum value, so the branch was always correctly reachable regardless of query count. Worth tightening that docstring. _prune_wave_contextonly sumsresults[].summarylengths, not thecallspayload, so a wave with large tool-call args could still blow past the intended context budget. Worth watching if context-overflow issues resurface.
Given this has apparently been sitting since around May 26, prioritizing the C++ side fix quickly would help — otherwise this is a candidate to close and reopen once the engine change is ready, rather than left open indefinitely.
|
Closing per the close-and-reopen suggestion in review rather than leaving this parked. The design was endorsed by both reviewers and the feature is still wanted, but it is hard-blocked on engine-side wave-partner node isolation (per-wave instances or a per-node mutex), which needs to land first and is not yet tracked anywhere. The store refactor in #1663 has also since replaced store.py, so the branch no longer applies. Plan: track the engine prerequisite separately, split off the wave-context pruning half (which does not depend on the concurrency fix) as a small standalone PR, and remake the multi-query fan-out against the new store/ package once the engine work lands. |
Summary
This PR hardens the orchestration layer by addressing two critical bottlenecks in agent workflows: I/O latency and context bloat.
1. Parallel Multi-Query Fan-Out (
DocumentStoreBase)forloop with aThreadPoolExecutorfan-out. Semantic/Keyword queries now execute concurrently, with athreading.Lockserializing the final dictionary merge. This drops latency fromN * p99to~p99.if not queries: pass) to preventmax_workers=0crashes, and guarantees in-flight thread cleanup via context managers.2. Server-Side Sliding Window (
RocketRideDriver)wavescontext previously grew infinitely unless the LLM cooperatively emitted aremovesignal, risking context-window overflows._prune_wave_context()guard enforcing a sliding window (context_window_waves) and a character budget (wave_context_budget_chars) before every prompt compilation, ensuring context remains mathematically bounded.Type
performance, enhancement
Testing
test_document_store_parallel.py, covering concurrency stress tests and zero-query edge cases)./builder testpasses cleanlyChecklist
Summary by CodeRabbit
New Features
Tests