[Tracking] Lance feature matrix for Doris 4.2 #66340

Goal

Plan and track the Doris 4.2 support scope for Lance Catalog, Lance dataset access, and vector search.

Doris 4.2 work items

Check an item only after implementation, tests, documentation, and verification are complete.

Tracking convention (2026-09-30): completion on branch-4.1 counts as completion for this Doris 4.2 feature matrix; a separate branch-4.2 merge is not a tracking prerequisite. This records implementation delivery, not availability in a published release.

Catalog integration

• Filesystem Lance Catalog with a local path
• Filesystem Lance Catalog with file://
• Filesystem Lance Catalog with s3://
• Lance REST Namespace Catalog

Authentication and credentials

• Bearer Token authentication
• API Key authentication
• Custom HTTP headers
• Temporary storage credentials returned by the Namespace service

Namespace and metadata

• Map multilevel Lance Namespaces to Doris databases
• SHOW DATABASES
• SHOW TABLES
• DESC

Namespace and table DDL

The baseline implementation in #65730 is read-only. SHOW DATABASES, SHOW TABLES, and DESC are supported through ListNamespaces, ListTables, and DescribeTable, while LanceExternalCatalog does not currently provide ExternalMetadataOps or invoke namespace/table mutation APIs. The remaining DDL work below is planned for Doris 4.2 and tracked by #66516.

• CREATE DATABASE / Lance Namespace
• DROP DATABASE / Lance Namespace
• CREATE TABLE / Lance dataset and catalog registration
• SHOW CREATE TABLE
• ALTER TABLE schema evolution — Add, alter/backfill, rename, and drop columns where supported by Lance.
• RENAME TABLE
• DROP TABLE / deregister table — Define whether each operation deletes the underlying dataset or only removes the catalog entry.
• Metadata cache refresh and failure semantics for DDL

Versioning and consistency

• Pin the dataset version during planning — Preserve snapshot consistency for the full query lifecycle.
• REST Namespace managed versioning in the BE reader — [feature](lance) Support time travel, tags, branches and REST Namespace managed versioning #68453 and [docs](lance) Document time travel, tags, branches and managed versioning doris-website#4172; FE resolves the selected snapshot for the existing BE reader.
• Time travel and explicit version selection for catalog scans — [Feature](lance) Support time travel and REST managed versioning #66491 / [feature](lance) Support time travel, tags, branches and REST Namespace managed versioning #68453; search TVF snapshot selection remains separate in [Feature](lance) Support snapshot selection in vector and full-text search #68625.

Scan execution

• Lance reader in File Scanner V2
• Parallel scan by Lance Fragment
• Column pruning
• Retain unsupported predicates for execution in Doris — Keep predicates in Doris whenever pushdown cannot preserve SQL semantics.

Predicate pushdown

• Push semantically compatible scalar predicates to Lance — Push only predicates whose semantics are compatible with Lance.
• SQL filter for vector_search()
• Substrait filter for regular Lance scans

Observability

• Show pushed predicates in EXPLAIN — Expose pushdown decisions for troubleshooting and verification.

Direct dataset access

• Read a Lance dataset through the s3() TVF
• Read a Lance dataset through the local() TVF

Paths and splits

• Split one Lance dataset into multiple Fragment splits — Generate parallel scan splits from Lance Fragments.
• Expand glob patterns to multiple Lance datasets — Resolve one path pattern into multiple datasets.
• Read multiple Lance datasets in one TVF invocation — Plan and scan more than one dataset in a single TVF call.
• Support additional file TVF entry points and broader path handling

Read operations

• Read Lance datasets

Search modes

• ANN index search
• Flat exact search
• Radius/Range Search — Expose lower and upper distance bounds.
• Full-text search with BM25 — Expose standalone BM25 full-text search.
• Multi-vector Search — Support multi-vector columns and multiple sub-vectors in one logical query.

Vector index algorithms

• IVF_FLAT query support
• IVF_SQ query support
• IVF_PQ query support
• IVF_HNSW_FLAT query support
• IVF_HNSW_SQ query support
• IVF_HNSW_PQ query support

Distance metrics

• L2
• Cosine
• Dot Product
• Hamming — Support compatible packed binary vectors and indexes.

Data types

• FixedSizeList
• FixedSizeList
• FixedSizeList — Define compatible ANN index and distance-metric combinations.
• FixedSizeList — Support UInt8 and compatible packed binary-vector use cases.
• FixedSizeList
• List<FixedSizeList<...>> for multi-vector columns — Map nested vector columns and define multi-vector scoring semantics.
• Blob v2
• Arrow JSON Extension
• BFloat16 Extension
• Arrow null
• Arrow duration

Query input

• One query vector per search
• Multiple sub-vectors in one logical query

Filtering

• Scalar pre-filter — Apply scalar predicates before vector candidate selection.
• Scalar post-filter — Allow scalar predicates to run after vector candidate selection.

Index lifecycle

• Create vector indexes
• Create scalar indexes
• Show the number of indexes — Exact logical-index counts remain deferred in the accepted [Feature](lance) Add vector and scalar index lifecycle management #66497 inspection contract.
• Show logical index names, columns, and types for Directory catalogs — SHOW INDEX, [feature](lance) Support SHOW INDEX for Lance Directory tables #66637 and [doc](lance) add doc about SHOW INDEX for lance catalog doris-website#4090.
• Show index names, UUIDs, columns, types, and dataset versions — Code and tests are merged in [feature](lance) Support SHOW INDEX for Lance Directory tables #66637 / [feature](lance) Add lance_index_entries inspection TVF #66671; user documentation for the physical lance_index_entries TVF remains outstanding. REST inspection is outside the delivered scope.
• Replace or rebuild an index with the same name
• Drop indexes by name

Index selection

• Automatically select an index for the target vector column — Use Lance index selection for the requested vector column.

ANN tuning

• nprobes
• refine_factor
• ef
• Disable the ANN index with use_index=false

Result output

• Top-K
• Offset
• Return distance through the _distance virtual column — Document exact, approximate, and refined distance semantics.

Tests and documentation

• FE unit tests
• BE unit tests
• Regression tests for Catalog queries, predicate pushdown, file TVFs, and vector search
• Lance Catalog user documentation — Publish the supported scope, configuration, examples, and limitations.

Explicitly not planned for Doris 4.2

These capabilities are excluded from the Doris 4.2 plan.

Completion criteria

• The capability is merged into branch-4.1; this satisfies branch completion for this Doris 4.2 tracker.
• FE/BE unit tests and regression tests cover the capability.
• User documentation describes the supported behavior, parameter semantics, and limitations.
• The capability is verified with representative Lance datasets and object-storage access where applicable.

Sub-issue status audit — 2026-09-30

Checked all ten direct sub-issues; none has nested sub-issues. Status below follows the branch-4.1 completion convention above. This was an audit of merged changes, documented validation, and existing CI results, not a new test run.

ANN coverage is bounded by the documented algorithm/type/metric compatibility matrix; checking an algorithm does not assert that every possible combination is supported. Multi-vector scoring and limits remain those documented in #68028 / apache/doris-website#4148.