Conversation
github-actions
Bot
added
A-index
sbrunk
force-pushed
the
combined-fields-java
branch
4 times, most recently
from
August 22, 2026 12:21
773c473 to
723c46b
Compare
sbrunk
force-pushed
the
combined-fields-java
branch
2 times, most recently
from
September 8, 2026 12:19
36461ed to
2a0042e
Compare
Score several text columns as one virtual field (Lucene's `CombinedFieldQuery` / BM25F blend) instead of the per-field max fusion `MultiMatch` does. Adds the query type with serde and JSON parsing, the `CombinedFieldsBM25Scorer`, the indexed scan and the planner and execution nodes that drive it. BM25F blends per-column term frequencies and document lengths into one `tf'`/`dl'` per row, so the scan is row-granular by construction. Two consequences shape the design: Row granularity, not document granularity. An inverted index may hold one document per list element and report `_doc_index` coordinates. BM25F cannot use such an index: it joins the target columns on the row address, and element coordinates of different columns have no correspondence to pair them on. `combined_fields` therefore declares itself row-granular everywhere the granularity plumbing asks, and rejects a target column that can only supply element documents. Corpus statistics must match that granularity. Releases before lance-format#7656 indexed each `List<String>` element as its own document, so those files report element-scoped `docCount`/`docFreq` while the scan accumulates by row. Mixing the two domains corrupts `idf'` and `avgdl'`, shifting an old index's top-k relative to the same data reindexed on a current build. Hence `bm25_row_stats_for_terms`, which counts distinct rows, delegating to the document-granular path on V3 where one row owns one document. A cross-field score is complete only when every target column's index holds the row, because `dl'` sums each column's length and a row absent from a column's `DocSet` contributes 0. This commit therefore requires every target column to cover every scanned fragment and refuses the query otherwise, naming the uncovered fragments and the columns to reindex. Scoring the rows no index covers is the next commit. The indexed scan reads every posting up front, then scores the union of the query terms' postings and keeps a bounded top-k. Every candidate is scored, so the result is exact by construction, and candidates are visited in ascending row-id order, which makes the top-k deterministic under ties. MAXSCORE pruning and read pruning are both follow-ups.
Dataset-level coverage for BM25F, checked against an independent brute-force BM25F reference (`lance_index::scalar::inverted::oracle`) that re-derives every statistic from the raw text, so it shares no code with the scan it checks. Each case asserts exact scores rather than just a hit set, because a wrong corpus size still returns the right rows in almost the right order. That is what pins down the parts easy to get subtly wrong: the per-column `w_f` factors, which are invisible at unit weights; ties, where the score-then-row ordering has to be deterministic across runs; and top-k across every k, where the pruning must agree with an exhaustive scan. Also covers the released-format fixtures (V1 and V2) so the row-granularity statistics path runs against real files rather than synthetic ones, nulls and empty strings, and the refusal paths: no index on any target column, and `fast_search` without full coverage.
…d_fields The previous commit refuses a `combined_fields` query whose target columns do not all cover every scanned fragment, so a default full-text search fails on any dataset with rows appended since the indexes were built. `MatchQuery` already unions in a flat scan for its unindexed fragments; this does the same for BM25F. Coverage is per column here, which makes it more than a copy of the single-column path. `dl'` sums each column's document length and a row absent from a column's `DocSet` contributes 0, so a fragment indexed for `title` but not `body` cannot be scored from the index at all. The indexed scan is therefore restricted to the intersection of per-column coverage and everything else goes to the flat scan, rather than splitting on the union. Both sides then score against one shared corpus. The flat side alone sees the rows no index covers, so it measures their contribution and publishes the blend; the indexed side waits for it instead of folding only its own `docCount'`/`docFreq'`/`avgdl'`. Without that, a row reached through either path would rank differently depending on which side happened to score it. Data overlays are handled by measuring rather than patching. When a target column carries an overlay-stale index entry, folding the flat row into the index statistics would double count it against the entry it replaces, and the flat scan cannot subtract what it replaced. So the corpus is measured from current data instead: every target fragment is scanned, every row folded into every column, and the index statistics left out. That costs a full scan of the target columns, so it stays confined to the stale case. `fast_search` is unchanged, being index-only by contract.
…tistics The cases that matter here are the ones where a shared corpus is easy to lose, since both scan sides must agree on `docCount'`/`docFreq'`/`avgdl'`: unindexed and partially indexed fragments, per-column index skew over both row-id schemes, a mixed indexed/flat plan, deletions followed by optimize, and overlay-stale fragments. Each asserts exact scores against the brute-force reference, because scoring the two sides against different corpora still returns the right rows in almost the right order. Also covers what only the flat path reaches: nulls and empty strings read from the scan rather than an index, list and nested columns, a column under a list, filters, and the plan shape itself, so a query that should union does not silently answer from the index alone. The external row-address prefilter is covered for the same reason, over both plan shapes: the fully covered plan ANDs the mask into the indexed scan's prefilter, while the mixed plan's flat child never reaches an index-side prefilter and is masked on its output instead. Scores are asserted there too, because the mask picks what is emitted and not the corpus it is measured against, so a surviving row must keep the score the unmasked query gave it.
sbrunk
force-pushed
the
combined-fields-java
branch
from
September 8, 2026 21:24
2a0042e to
02af44d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Exposes the Rust
combined_fields(BM25F) query through the Java scanner APIFullTextQuery.combinedFields(queryText, columns)plus an overload taking per-column boosts and an operator.Unlike
multiMatch, which scores each column separately and keeps the best field, this treats the target columns as one virtual field and blends term statistics across them, so a single query can match terms spread over severalcolumns.
Validation (unique columns, boost count matching column count, finite boosts >= 1) stays in the Rust core and surfaces as an exception when the query runs. The Java class only rejects null or empty text and columns, and defensively copies the column and boost lists.
Depends on the core combined_fields PR #7905