feat(fts): compose boolean and phrase scorers - #8093
Merged
Merged
Conversation
github-actions
Bot
added
A-index
This was referenced Jul 30, 2026
BubbleCal
force-pushed
the
yang/oss-1598-composable-fts-boolean-phrase
branch
from
July 30, 2026 16:43
4d9f7d1 to
5d11d3a
Compare
BubbleCal
added a commit
that referenced
this pull request
Jul 31, 2026
## Feature ### What is the new feature? Introduce the posting-backed compound FTS scoring core and use it for bounded same-column `MultiMatchQuery` execution. The change adds the internal `ComposableScorer` protocol, a cross-partition `TopKCollector`, incremental `WandCursor` support, and `CompoundQueryExec` integration. ### Why do we need this feature? Compound FTS currently falls back to DataFusion plans that remove the query limit and materialize intermediate results. Leaf FTS already has WAND/MAXSCORE bounds; this PR establishes the composable scorer and collector layer needed to preserve exact top-k semantics while applying the limit inside posting-backed execution. ### How does it work? - Scorers expose ordered document iteration, exact scores, shallow score bounds, competitive-score feedback, and two-phase confirmation. - One collector owns the competitive threshold across all index partitions and applies deterministic score/row-id tie ordering. - Same-column, fully indexed, fresh `MultiMatchQuery` shapes use DisMax over posting-backed leaves. - Cross-column, incomplete-index, stale-overlay, and unsupported query shapes retain the existing exact fallback. ## Stack 1. This PR: scorer core + same-column MultiMatch 2. #8093: Boolean + Phrase 3. #8094: Boost + nested composition Related: https://linear.app/lancedb/issue/OSS-1598/introduce-a-composable-scorer-interface-for-compound-fts ## Validation - `cargo check -p lance-index -p lance --lib` - `cargo test -p lance-index scalar::inverted::compound::tests --lib` - `cargo test -p lance dataset::tests::dataset_index::test_same_column_multimatch_uses_compound_scorer --lib` - `cargo test -p lance dataset::tests::dataset_index::test_nested_multimatch_limit_propagation --lib` - `cargo clippy -p lance-index -p lance --lib --tests -- -D warnings` - `cargo fmt --all` --------- Co-authored-by: Yang Cen <yang@lancedb.com>
BubbleCal
marked this pull request as ready for review
July 31, 2026 09:16
BubbleCal
force-pushed
the
yang/oss-1598-composable-fts-boolean-phrase
branch
from
July 31, 2026 09:25
5d11d3a to
bea5caa
Compare
BubbleCal
force-pushed
the
yang/oss-1598-composable-fts-boolean-phrase
branch
from
July 31, 2026 10:05
bea5caa to
952f7db
Compare
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
Claude Code Review is paused for this repository. To reconnect it, an admin of this repository's GitHub organization (or the account owner, for personal repositories) who can also manage your Claude organization's Code Review settings needs to re-link GitHub in Code Review settings. This is a one-time step.
Tip: disable this comment in your organization's Code Review settings.
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Xuanwo
approved these changes
Jul 31, 2026
BubbleCal
added a commit
that referenced
this pull request
Aug 1, 2026
## Feature Stacked on #8093 (which is stacked on #8092). Review this PR as the Boost increment only. ### What is the new feature? Add posting-backed `BoostQuery` composition, including nested Boost nodes inside same-column Boolean/Phrase scorer trees. ### Why do we need this feature? Boost scoring subtracts a weighted negative score from the positive score. This can produce signed results, so ordinary non-negative upper-bound propagation is insufficient for exact bounded top-k execution. ### How does it work? - `BoostScorer` is driven by the positive scorer and lazily confirms/scores the negative side on matching documents. - Signed score bounds conservatively subtract the negative range without underestimating the parent upper bound. - Competitive thresholds propagate to the positive side only when the negative scorer is known to be non-negative. - Match and Boost multipliers are rejected unless finite and non-negative. - Nested Boolean + Phrase + Boost execution is covered by the same-column end-to-end test. - Cross-column, incomplete-index, and stale-overlay shapes retain the exact fallback. No performance improvement is claimed here; this PR validates exact bounded execution semantics. Related: https://linear.app/lancedb/issue/OSS-1598/introduce-a-composable-scorer-interface-for-compound-fts ## Validation - `cargo check -p lance-index -p lance --lib` - `cargo test -p lance-index scalar::inverted::compound::tests --lib` - `cargo test -p lance dataset::tests::dataset_index::test_same_column_compound_scorer_is_exact_and_bounded --lib` - `cargo test -p lance dataset::tests::dataset_index::test_nested_multimatch_limit_propagation --lib` - `cargo test -p lance io::exec::fts::tests::test_boolean_query_parts_searched_metrics --lib` - `cargo test -p lance dataset::scanner::test::test_fts_query_contract_rejects_invalid_values --lib` - `cargo clippy -p lance-index -p lance --lib --tests -- -D warnings` - `cargo fmt --all` --------- Co-authored-by: Yang Cen <yang@lancedb.com>
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 2, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 5, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 6, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 10, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 10, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 12, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 14, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 14, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 16, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Feature
Review this PR as the Boolean/Phrase increment only.
What is the new feature?
Extend the composable FTS scorer tree with same-column
BooleanQueryandPhraseQueryexecution.Why do we need this feature?
Boolean clauses need exact composition of required, optional, and prohibited matches without materializing join intermediates. Phrase clauses additionally require an approximation/confirmation split so posting iteration remains cheap while positional checks stay exact.
How does it work?
matches()hook for positional confirmation.Stack
Related: https://linear.app/lancedb/issue/OSS-1598/introduce-a-composable-scorer-interface-for-compound-fts
Validation
cargo check -p lance-index -p lance --libcargo test -p lance-index scalar::inverted::compound::tests --libcargo test -p lance dataset::tests::dataset_index::test_same_column_compound_scorer_is_exact_and_bounded --libcargo test -p lance dataset::tests::dataset_index::test_nested_multimatch_limit_propagation --libcargo test -p lance io::exec::fts::tests::test_boolean_query_parts_searched_metrics --libcargo test -p lance dataset::scanner::test::test_fts_query_contract_rejects_invalid_values --libcargo clippy -p lance-index -p lance --lib --tests -- -D warningscargo fmt --all