Repository navigation
perf: add a streaming contains join to engine v2 - #778
Merged
azimafroozeh merged 2 commits intoSep 27, 2026
Merged
azimafroozeh merged 2 commits into
azimafroozeh merged 2 commits into
Conversation
5 of 7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Addresses the cross-binding filter failure in engine v2. Previously, the query built a Cartesian product before filtering
$p.text contains $m.number, which could exhaust the query-memory limit.containsjoins and discard nonmatching passages as the right scan streams.Recorded debug-CLI measurements on local storage: median of three warm runs and maximum warm process RSS. All workloads have 266 Matter nodes and 2,000 SourceRevision nodes. Base
24da4f54and the streaming implementation were measured in separate rounds.Speedup is base time divided by streaming time, rounded to a whole multiple. The memory-refusal case has no successful baseline to divide by. The last workload was re-measured on the final tree at 0.53 s and 130 MiB peak RSS, with the same 30 rows.
Backing issue / RFC
Checklist
Local verification
cargo check --workspace --tests -q --message-format=short: passed.cargo test -p omnigraph-gqt --locked -q: 380 passed.cargo test -p omnigraph-planner: 75 passed.cargo test -p omnigraph-engine --lib engine::: 74 passed;engine_v211,engine_v2_memory23,engine_v2_plan_replay13,literal_filters14.cargo clippy --workspace --all-targets -q --message-format=short -- -D warnings -W clippy::dbg_macro: passed.cargo fmt --all --check: passed.cargo test --workspace --exclude omnigraph-gqt --exclude omnigraph-dst --locked --no-fail-fast --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints: 3,402 passed, 4 failed, 29 ignored. The four failures come from local environment restrictions (three CLI session-lock tests and one Unix-socket test) and fail the same way without this change.openapisurface passes; therust-stringsurface fails with 1,075 entries, all reported while evaluating the merge base, none from this change.Notes for reviewers
set engine = v2; engine v1 retains the reported overflow behavior.ContainsJoinforms only when the text binding's scan is on the join's right: written second, or moved there when both row counts are known and the needles are no more than the texts. Otherwise the conjunct runs on a filteredCrossJoin, bounded in memory but testing every pair.limitabove the join waits for one output batch of kept pairs (the batch size or its byte budget) rather than the first right batch.ContainsJoin,CrossJoin.filtersand the scan'sruntime_filter, advancingEXPLAIN_VERSIONto 3. Passage text is still read from storage; no text index is added.