BM25 scores change when neither the data nor the query changes, because the corpus the scores are measured against is derived partly from the rows a particular query happens to read.
Two triggers with the same root cause.
1. The user filter moves the corpus
The flat side of a partially indexed plan is read through filtered_read(filter_plan, ...), and initialize_scorer folds counted_input.num_rows() from that already-filtered scan into num_docs. Filtered-out tail rows are therefore missing from the corpus, while the index statistics still count filtered-out indexed rows.
2. Index coverage moves the corpus
A fully covered fragment selection is scored against index-only statistics, even when unindexed fragments exist outside the selection. Once optimize_indices() folds those fragments in, the same rows score differently.
Reproduce
Two indexed rows (title = alpha, beta), ten alpha rows appended unindexed, limit=1. Run the query, then optimize_indices(), then run it again. No searchable data changed:
| scenario |
before |
after |
prefilter(true), filter id < 2 |
[(1, 0.6931472)] |
[(1, 2.1594841)] |
with_fragments([0]) |
[(1, 0.6931472)] |
[(1, 2.1594841)] |
Both hold for single-column match too, where the returned row changes as well, not just its score: [(0, 0.6931472)] → [(1, 2.1594841)].
Scope
Affects single-column match and combined_fields alike — plan_match_query / plan_flat_match_query / initialize_scorer and the FlatFieldStats / SharedFtsScorer path have the same shape.
The overlay-triggered form of this invariant is already fixed for combined_fields in #7905 (FlatScanFilter::AtEmission, pinned by test_fts_combined_fields_overlay_preserves_prefilter_corpus), since a same-value overlay must not move a ranking. The filter- and coverage-triggered forms remain, for both query types.
Fix
Build one corpus that is independent of the filter and of index coverage, before either child runs, and apply the user filter only to emitted candidates. Candidate selection can stay fragment-scoped; only corpus construction has to be global.
Found in review of #7905:
#7905 (comment) and
#7905 (comment)
BM25 scores change when neither the data nor the query changes, because the corpus the scores are measured against is derived partly from the rows a particular query happens to read.
Two triggers with the same root cause.
1. The user filter moves the corpus
The flat side of a partially indexed plan is read through
filtered_read(filter_plan, ...), andinitialize_scorerfoldscounted_input.num_rows()from that already-filtered scan intonum_docs. Filtered-out tail rows are therefore missing from the corpus, while the index statistics still count filtered-out indexed rows.2. Index coverage moves the corpus
A fully covered fragment selection is scored against index-only statistics, even when unindexed fragments exist outside the selection. Once
optimize_indices()folds those fragments in, the same rows score differently.Reproduce
Two indexed rows (
title=alpha,beta), tenalpharows appended unindexed,limit=1. Run the query, thenoptimize_indices(), then run it again. No searchable data changed:prefilter(true), filterid < 2[(1, 0.6931472)][(1, 2.1594841)]with_fragments([0])[(1, 0.6931472)][(1, 2.1594841)]Both hold for single-column
matchtoo, where the returned row changes as well, not just its score:[(0, 0.6931472)]→[(1, 2.1594841)].Scope
Affects single-column
matchandcombined_fieldsalike —plan_match_query/plan_flat_match_query/initialize_scorerand theFlatFieldStats/SharedFtsScorerpath have the same shape.The overlay-triggered form of this invariant is already fixed for
combined_fieldsin #7905 (FlatScanFilter::AtEmission, pinned bytest_fts_combined_fields_overlay_preserves_prefilter_corpus), since a same-value overlay must not move a ranking. The filter- and coverage-triggered forms remain, for both query types.Fix
Build one corpus that is independent of the filter and of index coverage, before either child runs, and apply the user filter only to emitted candidates. Candidate selection can stay fragment-scoped; only corpus construction has to be global.
Found in review of #7905:
#7905 (comment) and
#7905 (comment)