feat(fts): add composable scorer core for multimatch - #8092
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
BubbleCal
marked this pull request as ready for review
July 30, 2026 10:02
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
Claude Code Review is paused for this repository. To reconnect it, an admin of this repository's GitHub organization (or the account owner, for personal repositories) who can also manage your Claude organization's Code Review settings needs to re-link GitHub in Code Review settings. This is a one-time step.
Tip: disable this comment in your organization's Code Review settings.
Xuanwo
requested review from
lance-community
and removed request for
lance-community
July 30, 2026 14:22
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
The revised collector now preserves exact score/row-ID ordering across reversed modern segments, so the previous top-k correctness blocker is resolved. The posting-backed composition direction is acceptable; I did not find another correctness issue that should block it.
Xuanwo
approved these changes
Jul 31, 2026
BubbleCal
added a commit
that referenced
this pull request
Jul 31, 2026
## Feature Review this PR as the Boolean/Phrase increment only. ### What is the new feature? Extend the composable FTS scorer tree with same-column `BooleanQuery` and `PhraseQuery` execution. ### Why do we need this feature? Boolean clauses need exact composition of required, optional, and prohibited matches without materializing join intermediates. Phrase clauses additionally require an approximation/confirmation split so posting iteration remains cheap while positional checks stay exact. ### How does it work? - Boolean SHOULD clauses use score sums; MUST clauses intersect membership while preserving the existing first-MUST scoring contract; MUST NOT clauses filter confirmed matches. - Phrase leaves use the scorer's two-phase `matches()` hook for positional confirmation. - Sum and Boolean score bounds remain conservative before propagating competitive thresholds. - Phrase-containing compound queries require an index with positions. - Cross-column, incomplete-index, stale-overlay, and Boost-containing shapes continue to use the exact fallback in this layer. ## Stack 1. #8092: scorer core + same-column MultiMatch 2. This PR: Boolean + Phrase 3. #8094: Boost + nested composition Related: https://linear.app/lancedb/issue/OSS-1598/introduce-a-composable-scorer-interface-for-compound-fts ## Validation - `cargo check -p lance-index -p lance --lib` - `cargo test -p lance-index scalar::inverted::compound::tests --lib` - `cargo test -p lance dataset::tests::dataset_index::test_same_column_compound_scorer_is_exact_and_bounded --lib` - `cargo test -p lance dataset::tests::dataset_index::test_nested_multimatch_limit_propagation --lib` - `cargo test -p lance io::exec::fts::tests::test_boolean_query_parts_searched_metrics --lib` - `cargo test -p lance dataset::scanner::test::test_fts_query_contract_rejects_invalid_values --lib` - `cargo clippy -p lance-index -p lance --lib --tests -- -D warnings` - `cargo fmt --all` Co-authored-by: Yang Cen <yang@lancedb.com>
BubbleCal
added a commit
that referenced
this pull request
Aug 1, 2026
## Feature Stacked on #8093 (which is stacked on #8092). Review this PR as the Boost increment only. ### What is the new feature? Add posting-backed `BoostQuery` composition, including nested Boost nodes inside same-column Boolean/Phrase scorer trees. ### Why do we need this feature? Boost scoring subtracts a weighted negative score from the positive score. This can produce signed results, so ordinary non-negative upper-bound propagation is insufficient for exact bounded top-k execution. ### How does it work? - `BoostScorer` is driven by the positive scorer and lazily confirms/scores the negative side on matching documents. - Signed score bounds conservatively subtract the negative range without underestimating the parent upper bound. - Competitive thresholds propagate to the positive side only when the negative scorer is known to be non-negative. - Match and Boost multipliers are rejected unless finite and non-negative. - Nested Boolean + Phrase + Boost execution is covered by the same-column end-to-end test. - Cross-column, incomplete-index, and stale-overlay shapes retain the exact fallback. No performance improvement is claimed here; this PR validates exact bounded execution semantics. Related: https://linear.app/lancedb/issue/OSS-1598/introduce-a-composable-scorer-interface-for-compound-fts ## Validation - `cargo check -p lance-index -p lance --lib` - `cargo test -p lance-index scalar::inverted::compound::tests --lib` - `cargo test -p lance dataset::tests::dataset_index::test_same_column_compound_scorer_is_exact_and_bounded --lib` - `cargo test -p lance dataset::tests::dataset_index::test_nested_multimatch_limit_propagation --lib` - `cargo test -p lance io::exec::fts::tests::test_boolean_query_parts_searched_metrics --lib` - `cargo test -p lance dataset::scanner::test::test_fts_query_contract_rejects_invalid_values --lib` - `cargo clippy -p lance-index -p lance --lib --tests -- -D warnings` - `cargo fmt --all` --------- Co-authored-by: Yang Cen <yang@lancedb.com>
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 2, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 5, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 6, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 10, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 10, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 12, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 14, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 14, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 16, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Feature
What is the new feature?
Introduce the posting-backed compound FTS scoring core and use it for bounded same-column
MultiMatchQueryexecution. The change adds the internalComposableScorerprotocol, a cross-partitionTopKCollector, incrementalWandCursorsupport, andCompoundQueryExecintegration.Why do we need this feature?
Compound FTS currently falls back to DataFusion plans that remove the query limit and materialize intermediate results. Leaf FTS already has WAND/MAXSCORE bounds; this PR establishes the composable scorer and collector layer needed to preserve exact top-k semantics while applying the limit inside posting-backed execution.
How does it work?
MultiMatchQueryshapes use DisMax over posting-backed leaves.Stack
Related: https://linear.app/lancedb/issue/OSS-1598/introduce-a-composable-scorer-interface-for-compound-fts
Validation
cargo check -p lance-index -p lance --libcargo test -p lance-index scalar::inverted::compound::tests --libcargo test -p lance dataset::tests::dataset_index::test_same_column_multimatch_uses_compound_scorer --libcargo test -p lance dataset::tests::dataset_index::test_nested_multimatch_limit_propagation --libcargo clippy -p lance-index -p lance --lib --tests -- -D warningscargo fmt --all