feat(fts): compose boosted compound scorers - #8094
Merged
Merged
Conversation
github-actions
Bot
added
A-index
This was referenced Jul 30, 2026
BubbleCal
force-pushed
the
yang/oss-1598-composable-fts-scorer
branch
from
July 30, 2026 16:43
a1062d5 to
341d0ea
Compare
BubbleCal
added a commit
that referenced
this pull request
Jul 31, 2026
## Feature ### What is the new feature? Introduce the posting-backed compound FTS scoring core and use it for bounded same-column `MultiMatchQuery` execution. The change adds the internal `ComposableScorer` protocol, a cross-partition `TopKCollector`, incremental `WandCursor` support, and `CompoundQueryExec` integration. ### Why do we need this feature? Compound FTS currently falls back to DataFusion plans that remove the query limit and materialize intermediate results. Leaf FTS already has WAND/MAXSCORE bounds; this PR establishes the composable scorer and collector layer needed to preserve exact top-k semantics while applying the limit inside posting-backed execution. ### How does it work? - Scorers expose ordered document iteration, exact scores, shallow score bounds, competitive-score feedback, and two-phase confirmation. - One collector owns the competitive threshold across all index partitions and applies deterministic score/row-id tie ordering. - Same-column, fully indexed, fresh `MultiMatchQuery` shapes use DisMax over posting-backed leaves. - Cross-column, incomplete-index, stale-overlay, and unsupported query shapes retain the existing exact fallback. ## Stack 1. This PR: scorer core + same-column MultiMatch 2. #8093: Boolean + Phrase 3. #8094: Boost + nested composition Related: https://linear.app/lancedb/issue/OSS-1598/introduce-a-composable-scorer-interface-for-compound-fts ## Validation - `cargo check -p lance-index -p lance --lib` - `cargo test -p lance-index scalar::inverted::compound::tests --lib` - `cargo test -p lance dataset::tests::dataset_index::test_same_column_multimatch_uses_compound_scorer --lib` - `cargo test -p lance dataset::tests::dataset_index::test_nested_multimatch_limit_propagation --lib` - `cargo clippy -p lance-index -p lance --lib --tests -- -D warnings` - `cargo fmt --all` --------- Co-authored-by: Yang Cen <yang@lancedb.com>
BubbleCal
force-pushed
the
yang/oss-1598-composable-fts-scorer
branch
from
July 31, 2026 09:25
341d0ea to
1650f6b
Compare
BubbleCal
force-pushed
the
yang/oss-1598-composable-fts-scorer
branch
from
July 31, 2026 10:05
1650f6b to
664ba62
Compare
Base automatically changed from
yang/oss-1598-composable-fts-boolean-phrase
to
main
July 31, 2026 11:19
BubbleCal
added a commit
that referenced
this pull request
Jul 31, 2026
## Feature Review this PR as the Boolean/Phrase increment only. ### What is the new feature? Extend the composable FTS scorer tree with same-column `BooleanQuery` and `PhraseQuery` execution. ### Why do we need this feature? Boolean clauses need exact composition of required, optional, and prohibited matches without materializing join intermediates. Phrase clauses additionally require an approximation/confirmation split so posting iteration remains cheap while positional checks stay exact. ### How does it work? - Boolean SHOULD clauses use score sums; MUST clauses intersect membership while preserving the existing first-MUST scoring contract; MUST NOT clauses filter confirmed matches. - Phrase leaves use the scorer's two-phase `matches()` hook for positional confirmation. - Sum and Boolean score bounds remain conservative before propagating competitive thresholds. - Phrase-containing compound queries require an index with positions. - Cross-column, incomplete-index, stale-overlay, and Boost-containing shapes continue to use the exact fallback in this layer. ## Stack 1. #8092: scorer core + same-column MultiMatch 2. This PR: Boolean + Phrase 3. #8094: Boost + nested composition Related: https://linear.app/lancedb/issue/OSS-1598/introduce-a-composable-scorer-interface-for-compound-fts ## Validation - `cargo check -p lance-index -p lance --lib` - `cargo test -p lance-index scalar::inverted::compound::tests --lib` - `cargo test -p lance dataset::tests::dataset_index::test_same_column_compound_scorer_is_exact_and_bounded --lib` - `cargo test -p lance dataset::tests::dataset_index::test_nested_multimatch_limit_propagation --lib` - `cargo test -p lance io::exec::fts::tests::test_boolean_query_parts_searched_metrics --lib` - `cargo test -p lance dataset::scanner::test::test_fts_query_contract_rejects_invalid_values --lib` - `cargo clippy -p lance-index -p lance --lib --tests -- -D warnings` - `cargo fmt --all` Co-authored-by: Yang Cen <yang@lancedb.com>
BubbleCal
force-pushed
the
yang/oss-1598-composable-fts-scorer
branch
from
July 31, 2026 11:19
664ba62 to
43d0edf
Compare
BubbleCal
marked this pull request as ready for review
July 31, 2026 11:19
Xuanwo
approved these changes
Aug 1, 2026
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 2, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 5, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 6, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 10, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 10, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 12, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 14, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 14, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
sbrunk
added a commit
to sbrunk/lance
that referenced
this pull request
Aug 16, 2026
The composable scorer stack (lance-format#8092, lance-format#8093, lance-format#8094, lance-format#8131) added `compound.rs`, a single-pass `ComposableScorer` that runs a whole compound tree against one column's index with `Match`/`Phrase` leaves scored by a `MemBM25Scorer`. It matches `FtsQuery` exhaustively, so the `CombinedFields` variant has to be accounted for in multiple places. BM25F blends `tf'`, `dl'` and `docFreq'` across columns before scoring, so it cannot be a leaf that draws its postings from a single index; making it one needs a multi-index leaf protocol and block-max bounds out of the MAXSCORE scan. Gate it out instead, leaving those trees on the union/sort plan `plan_combined_fields_query` already builds: - `supports_compound_scorer` rejects any tree containing a `CombinedFields` node, at the top level or nested. - The two `compound.rs` entry points return `Error::not_supported` rather than `unreachable!()`, so a future planner change fails loudly instead of silently misscoring. - `collect_all_fts_columns` reports the real target columns, `contains_phrase_query` is false (BM25F needs no positions), `validate_fts_query_contract` has nothing to add (the weights are private and every constructor, `Deserialize` included, goes through `try_with_boosts`), and `count_fts_leaves` counts the per-column posting scans its partition estimate is approximating. `test_nested_combined_fields_limit_propagation` covers the gate: it nests a combined_fields node under MUST, SHOULD and BoostQuery, so a leak surfaces the `not_supported` error. Also apply lance-format#8102 to `plan_combined_fields_query`, which still deep-copied the fragment descriptor list on every plan. Borrow the slice like `plan_match_query` now does and materialize only the uncovered subset the flat plan takes ownership of.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Feature
Stacked on #8093 (which is stacked on #8092). Review this PR as the Boost increment only.
What is the new feature?
Add posting-backed
BoostQuerycomposition, including nested Boost nodes inside same-column Boolean/Phrase scorer trees.Why do we need this feature?
Boost scoring subtracts a weighted negative score from the positive score. This can produce signed results, so ordinary non-negative upper-bound propagation is insufficient for exact bounded top-k execution.
How does it work?
BoostScoreris driven by the positive scorer and lazily confirms/scores the negative side on matching documents.No performance improvement is claimed here; this PR validates exact bounded execution semantics.
Related: https://linear.app/lancedb/issue/OSS-1598/introduce-a-composable-scorer-interface-for-compound-fts
Performance
Validation
cargo check -p lance-index -p lance --libcargo test -p lance-index scalar::inverted::compound::tests --libcargo test -p lance dataset::tests::dataset_index::test_same_column_compound_scorer_is_exact_and_bounded --libcargo test -p lance dataset::tests::dataset_index::test_nested_multimatch_limit_propagation --libcargo test -p lance io::exec::fts::tests::test_boolean_query_parts_searched_metrics --libcargo test -p lance dataset::scanner::test::test_fts_query_contract_rejects_invalid_values --libcargo clippy -p lance-index -p lance --lib --tests -- -D warningscargo fmt --all