Skip to content

feat: expose row address allowlist/blocklist prefilters in C and C++ - #96

Merged
jja725 merged 1 commit into
lance-format:mainfrom
wanglun:row-addr-prefilter
Oct 14, 2026
Merged

jja725 merged 1 commit into
lance-format:mainfrom
wanglun:row-addr-prefilter

Conversation

@wanglun

@wanglun wanglun commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

C and C++ callers cannot restrict a scan or search to a known set of rows before ranking: a _rowid IN (...) filter is evaluated by a full row-id read plus an IN-list parse, which costs tens of milliseconds even for a few ids and seconds for large lists. Lance's Scanner::with_row_addr_prefilter does this natively; pylance exposes it as row_addr_allowlist / row_addr_blocklist.

Add lance_scanner_set_row_addr_allowlist and
lance_scanner_set_row_addr_blocklist, which take arrays of _rowid values (stable row ids when enabled, otherwise row addresses), and the C++ Scanner::row_addr_allowlist / row_addr_blocklist wrappers. The lists are combined like RowAddrMask::from_serialized_parts (allowed AND NOT blocked) and ANDed with any filter expression. An empty allowlist allows no rows.

A prepared FTS query context rebuilds its Match/Phrase execs around the shared global scorer, and the rebuilt execs did not carry the external mask, so the mask was silently dropped. The rewrite now re-attaches it, so prepared Match/Phrase searches score only the allowed rows with unchanged BM25 scores. Combining a row list with a segment-scoped scalar scan, which installs its own row mask, is rejected.

Tests cover plain scans (allow, block, both, empty, replacement, AND with a SQL filter) on a two-fragment dataset with stable row ids, invalid inputs (NULL with len > 0, setting after the scan starts, scalar_index_segment), plain and prepared full-text search (OR and AND operators, scores equal to the unmasked search), and nearest-neighbor top-k over allowed rows. C and C++ runtime tests exercise the exported symbols and wrappers.

Validation:

  • cargo fmt -- --check
  • cargo clippy --all-targets -- -D warnings
  • cargo test
  • cargo test --test compile_and_run_test -- --ignored --test-threads=1

C and C++ callers cannot restrict a scan or search to a known set of rows
before ranking: a `_rowid IN (...)` filter is evaluated by a full row-id
read plus an IN-list parse, which costs tens of milliseconds even for a few
ids and seconds for large lists. Lance's `Scanner::with_row_addr_prefilter`
does this natively; pylance exposes it as `row_addr_allowlist` /
`row_addr_blocklist`.

Add `lance_scanner_set_row_addr_allowlist` and
`lance_scanner_set_row_addr_blocklist`, which take arrays of `_rowid`
values (stable row ids when enabled, otherwise row addresses), and the C++
`Scanner::row_addr_allowlist` / `row_addr_blocklist` wrappers. The lists are
combined like `RowAddrMask::from_serialized_parts` (allowed AND NOT blocked)
and ANDed with any filter expression. An empty allowlist allows no rows.

A prepared FTS query context rebuilds its Match/Phrase execs around the
shared global scorer, and the rebuilt execs did not carry the external
mask, so the mask was silently dropped. The rewrite now re-attaches it, so
prepared Match/Phrase searches score only the allowed rows with unchanged
BM25 scores. Combining a row list with a segment-scoped scalar scan, which
installs its own row mask, is rejected.

Tests cover plain scans (allow, block, both, empty, replacement, AND with a
SQL filter) on a two-fragment dataset with stable row ids, invalid inputs
(NULL with len > 0, setting after the scan starts, scalar_index_segment),
plain and prepared full-text search (OR and AND operators, scores equal to
the unmasked search), and nearest-neighbor top-k over allowed rows. C and C++
runtime tests exercise the exported symbols and wrappers.

Validation:
- `cargo fmt -- --check`
- `cargo clippy --all-targets -- -D warnings`
- `cargo test`
- `cargo test --test compile_and_run_test -- --ignored --test-threads=1`

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The native row-mask approach preserves selection before ranking and shared BM25 scoring while keeping the C and C++ bindings thin. The new Rust tests, prepared-FTS regressions, additional phrase/top-k and indexed-vector checks, and both C/C++ runtime suites passed.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 13, 2026
@wanglun

wanglun commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

@jja725 thanks for the review, it seems I cannot merge it, can you help? thanks!

@jja725
jja725 merged commit 2089f6f into lance-format:main Oct 14, 2026
10 checks passed
@wanglun
wanglun deleted the row-addr-prefilter branch October 15, 2026 00:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants