Repository navigation
performance: bm25/rrf ranked read + a join materializes the joined column corpus-wide → Arrow 2 GB offset overflow at ~700k+ rows #563
Description
Summary
A ranked read (bm25(...) or rrf(...)) that also traverses/joins to another node type makes the planner materialize the joined variable-width column for the whole corpus before it prunes to the top-limit. Once that column exceeds Arrow's 2 GB per-column offset limit (i32 offsets on a Utf8/binary array), the query fails with an Offset overflow-class error. In practice this kicks in on a large text corpus (~700k+ rows) and forces dropping the join from otherwise-natural recall queries.
Environment
- Engine 0.8.1 (downstream build; the relevant planner/scan paths are unmodified from upstream).
- Single-node
omnigraph-server, S3-compatible storage. - Corpus: a
Chunknode type, ~1.05M rows,text: String @index(BM25) +embedding: Vector(3072) @index(ANN), with aChunkOfArtifact: Chunk -> Artifactedge.
Repro (shape)
// Fails at corpus scale — the rrf/bm25 leg + the join materialize Chunk.text corpus-wide:
query recall($q: String) {
match {
$c: Chunk
$c chunkOfArtifact $a // <-- the join is what tips it over
search($c.text, $q)
}
return { $c.slug, $c.text, $a.slug, $a.name }
order { rrf(bm25($c.text, $q), nearest($c.embedding, $q)) }
limit 20
}
The same query without the chunkOfArtifact join (rank the chunks alone, recover the artifact from the chunk slug afterward) runs fine — because a nearest()-only path prunes to top-k first, but the bm25/rrf leg + join does not.
Impact
We had to drop the chunkOfArtifact join from our keyword and hybrid recall queries entirely and reconstruct artifact identity from the chunk slug downstream. That's a real loss of expressiveness on the hot retrieval path — you can't rank-and-join in one query at corpus scale.
Suggested direction
Push the rank + limit down so the join only materializes the top-k rows (rank first on the base table, then join the survivors), rather than materializing the joined column across the full corpus. If 0.9's filter/scan-pushdown work already changes this, it'd be great to confirm whether ranked-read-with-join is now bounded — we haven't yet re-tested on 0.9.
Activity
FlorianSeemann commentedon Aug 28, 2026
We dug into this further and found the root cause — it sits one layer below the join, and the join overflow is the amplification of it.
Root cause: the
bm25()search mode never captures the query'slimitIn
extract_search_mode(crates/omnigraph/src/exec/query.rs), the two ranked orderings are treated asymmetrically:Nearestarm requires the query'slimitand threads it into the scanner as top-k (v0.9.0exec/query.rs:169, thenscanner.nearest(prop, &query_arr, k)).Bm25arm captures no limit at all —bm25: Some((var, prop, text))(v0.9.0exec/query.rs:193, same on currentmain:215and in the RRF sub-mode atmain:293).At the exec site the FTS query is then built unbounded —
FullTextSearchQuery::new(text.clone()).with_column(prop)(v0.9.0:2479,main:2540) — andlance_index::scalar::FullTextSearchQuery::newdefaultslimit: None. With no limit, Lance returns every matching row, BM25-ranked; the query'slimit Nis only applied at the very end of the pipeline, after materialization.So even without a join, a
bm25()-ordered read materializes the full matching set × all projected columns. Two things then amplify it:Vectorcolumn even when unprojected (filed separately as performance: projecting non-vector columns still reads the full Vector column — read cost/OOM scale with the embedding column even when unused #564) — at 3072-dim that's ~13 KB/row on rows that will be thrown away.Utf8offset overflow in this issue's title appears.Evidence it's the limit, not the index
On our corpus the
textcolumn has a healthyInvertedindex (verified via index metadata: covers the table up to the latest-but-one version), andnearest()over the same 1.05M-row table returns in ~2 s. Yet:bm25()limit 20: ~20 s p50Top-k pruning simply never happens on the FTS path.
Fix sketch (mirrors the
nearestcontract)limitrequired for abm25()-ordered query, exactly likenearest()already errors ("nearest() ordering requires a limit clause").SearchMode::bm25((var, prop, text)→(var, prop, text, k)).FullTextSearchQueryalready haspub limit: Option<i64>.Bm25arm ofextract_sub_search_mode: it already receives the limit but only uses it fornearest. For therrf()legs a small multiple of the final limit (e.g. 3×) may be worth it so fusion quality doesn't degrade at the boundary.That one change should also dissolve the join overflow this issue reports: with k rows out of the FTS source, the join touches k rows, not the corpus.
Happy to open a PR against
main— we're applying the same patch to our 0.8.1 downstream build this week and can report before/after p50 at 1.05M rows.fix(query): push the query limit into bm25 FTS search — unbounded ful…