Skip to content

[docs] Add Lance hybrid search best practices and correct type mappings - #4182

Merged
Gabriel39 merged 1 commit into
apache:masterfrom
Gabriel39:dev/lance-hybrid-search-best-practices
Sep 30, 2026
Merged

Gabriel39 merged 1 commit into
apache:masterfrom
Gabriel39:dev/lance-hybrid-search-best-practices

Conversation

@Gabriel39

Copy link
Copy Markdown
Contributor

Filtered Lance retrieval can repeat scalar-index work when scalar segments cross search-split boundaries. Add an English/Chinese guide explaining filter placement, physical-segment split planning, fragment-coverage alignment, construction and maintenance workflows, shared-cache pressure, and profile diagnosis.

The guide distinguishes fragment-scoped V2 scalar pruning from legacy reader paths, treats segment sizing as a workload experiment, and explains why logical indexes do not automatically align different columns. It includes filtered vector/FTS SQL and a clearly labeled SDK orchestration pattern with separate per-index commits.

Also correct the Catalog type mapping to document Arrow/Lance JSON as Doris JSON, together with the already-supported Duration, nullable Null, and BFloat16 mappings. Preserve storage-type checks, unsupported-extension restrictions, and the distinction between JSON extensions and ordinary strings containing JSON text. Remove JSON from the unsupported-column exclusion example. Type behavior was checked against the current branch-4.1 converter and reader, including apache/doris#67325.

Validation:

  • Docusaurus 3.6.3 MDX compilation passed for all six touched pages.
  • Front matter, Markdown structure, new-page SEO, targeted links/anchors, and diff checks passed.
  • Checked bilingual SQL/Python examples, Python syntax, metric names, heading anchors, sidebar JSON/reference, and sensitive-data absence in additions.
  • The i18n checker reports only missing current counterparts. These pages intentionally follow the existing 4.x-only Lance Catalog placement and retain its documented experimental/4.2 availability boundary.
  • No full-site build, dataset execution, or cluster benchmark was performed. No performance improvement is claimed as measured.

Self-review:

  • Goal: covers filtered retrieval tuning and aligned construction; corrects the requested JSON support statement.
  • Scope: two new localized pages, focused Catalog corrections, entry links, and one sidebar item.
  • Information architecture: follows the existing 4.x Lakehouse Best Practices layout; locales are synchronized.
  • Links/navigation: existing URLs and headings preserved; new page linked from the sidebar, Catalog, and cache guide.
  • UI/configuration: no component, styling, routing configuration, or build-script changes; sidebar JSON remains valid.
  • Verification: targeted checks above cover document parsing and navigation; runtime validation limits are explicit.
  • Other issues: no additional correctness, accessibility, SEO, or maintainability issues found in self-review.

@Gabriel39
Gabriel39 merged commit 4024df5 into apache:master Sep 30, 2026
3 checks passed

This branch was previously deployed

1 inactive deployment
Production — 2db7cbdd Deployed Sep 30, 2026 by Gabriel39 via Build Check #8444
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant