Vector search performance optimization roadmap #6820

Summary

Track a focused sequence of vector query performance work in Lance.

The main themes are:

• Batch vector query support: one query request can contain multiple query vectors.
• Flat KNN MVP: share table scan/load work across the batch and return per-query top-k results.
• Fused batch distance kernels: avoid turning batch queries into a loop over single-query kernels.
• Metric-specific kernels: especially small-dimensional Hamming-like metrics where SIMD utilization can matter.
• Pruning: partition-level pruning now that vector search can process partitions serially, plus in-partition pruning such as PDX-style approaches.
• Benchmarks and guardrails: measure p50/p95/p99 latency, QPS, recall, and regression risk across dimensions, metrics, and partition sizes.

Linear project: Vector Search Performance Optimizations
Linear parent: #6820

Tasks

• Support batch vector query API and shared flat KNN scan #6821 / Support batch vector query API and shared flat KNN scan #6821
• Extend batch vector queries to ANN and indexed search #6822 / Extend batch vector queries to ANN and indexed search #6822
• Add fused batch distance kernels for vector search #6823 / Add fused batch distance kernels for vector search #6823
• Add small-dimension batch kernels for Hamming and binary metrics #6824 / Add small-dimension batch kernels for Hamming and binary metrics #6824
• Add partition-level pruning for serial vector search #6825 / Add partition-level pruning for serial vector search #6825
• Add in-partition pruning for vector search scans #6826 / Add in-partition pruning for vector search scans #6826
• Add benchmark coverage for vector search optimization work #6827 / Add benchmark coverage for vector search optimization work #6827