Add SpillStore trait with local-disk implementation #7300

Activity

1. wjones127

   added

   enhancementNew feature or requestNew feature or request

   rustRust related tasksRust related tasks

   A-indexVector index, linalg, tokenizerVector index, linalg, tokenizer

   on Jun 16, 2026
2. wjones127 commented on Jun 17, 2026

   

   wjones127

   on Jun 17, 2026

   Last edited by wjones127

   ContributorAuthor

   More actions

   Design: SpillStore — generic RAII scratch for index builds

   Supersedes the earlier byte-stream / MeteredObjectStore sketches in this thread. This is the design we're building.

   Scope

   Ship the generic mechanism: SpillStore/SpillFile traits + local-disk impl + per-file RAII + disk cap + Session integration + tests. Consumer migration (IVF first) is a follow-up — the mechanism stays uncoupled.

   FTS is excluded. InvertedIndexBuilder::flush() writes finalized partitions via builder.write(self.dest_store), where dest_store is the dataset's object store at indices_dir/uuid. Those partitions are the final index — never read back during the build, never deleted. FTS has no local-disk scratch tier; its overflow lever is worker_memory_limit_bytes (the MemoryPool tier, not SpillStore). BTree/Bitmap remain on DataFusion's DiskManager.

   Crate layering

   Lives in lance-io — the trait traffics in lance-io's Reader/Writer, and lance-core doesn't depend on lance-io. The DiskCapExceeded error variant stays in lance-core. Session (in lance) holds Arc<dyn SpillStore>.

   The interface

   pub trait SpillStore: Send + Sync + 'static {
       /// One empty scratch file; deleted when the handle drops.
       fn create_spill_file(&self) -> Result<Box<dyn SpillFile>>;
   }
   
   pub trait SpillFile: Send + Sync {
       async fn writer(&self) -> Result<Box<dyn Writer>>;   // cap-enforcing LocalWriter
       async fn reader(&self) -> Result<Box<dyn Reader>>;
   }

   No ObjectStore is exposed — tighter encapsulation, simpler RAII. writer() feeds FileWriter::try_new / PreviousFileWriter::with_object_writer directly (both already take Box<dyn Writer>, and ObjectStore::create() already returns one). reader() feeds a v2 FileReader via the new bridge below.

   LocalSpillStore + per-file RAII

   • Owns: Arc<TempDir> (shared backstop), one shared base ObjectStore::local() carrying the DiskQuota, an atomic file counter.
   • create_spill_file() → unique path under the temp dir → LocalSpillFile { store: <shared>, path, quota }.
   • LocalSpillFile::Drop is the reclaimer: stat file size → quota.release(size) → remove_file(path). The shared TempDir mops up anything leaked at Session end.
   • No per-file ObjectStore clone, no inner wrapper, no into_parts — the shared store mints readers/writers; the handle owns cleanup.

   Disk cap (lance-io LocalWriter hook)

   The local write fast-path bypasses the object_store::ObjectStore trait (create() for the file scheme returns a LocalWriter that writes via tokio::fs and renames on shutdown — it never calls the wrapped inner). So the cap is enforced in LocalWriter, not via a wrapping store:

   • DiskQuota { cap_bytes, Arc<AtomicU64> used } — shared via the base store, so the cap is Session-level across all live scratch.
   • ObjectStore gains disk_quota: Option<DiskQuota> (a general "write byte budget"); create() threads it into LocalWriter.
   • LocalWriter::poll_write reserves incrementally; over cap → typed Error::DiskCapExceeded.
   • lance_core's From<io::Error> gains downcast-recovery so the typed error survives the io::Error boundary (additive, safe).

   Accounting: reserve-on-write + release-on-SpillFile-drop (by stat). Exact for the documented contract (one SpillFile, written once, then read). Aborted-at-cap writes leak their reservation until the store drops (error path); rewrites double-count. Documented, not engineered around. Cloud/ObjectWriter (Enterprise remote) accounts for itself later.

   The read bridge (new lance-io API)

   A v2 FileReader consumes a FileScheduler (= Arc<dyn Reader> + Arc<ScanScheduler>), not a bare Reader, and the only scheduler-free EncodingsIo (BufferScheduler) is in-memory-only. So we add:

   impl ScanScheduler {
       /// Open a FileScheduler over an already-open Reader (no path/store lookup).
       pub fn open_reader(self: &Arc<Self>, reader: Arc<dyn Reader>) -> FileScheduler;
   }

   ~10 lines, mirroring open_file minus the path-open. A consumer turns spill_file.reader() into a v2 FileReader using its own ScanScheduler; the spill ObjectStore is never exposed.

   // consumer (v2 read)
   let fs = scheduler.open_reader(spill_file.reader().await?.into());
   let rdr = FileReader::try_open(fs, ...).await?;

   v1 legacy reads (PreviousFileReader::try_new_self_described(&ObjectStore, path)) stay store-based — v2 is the migration target.

   Usage contract

   Hold the SpillFile alive for the file's lifetime; its reader()/writer() reference the backing file, which the handle deletes on Drop. The TempDir is the backstop.

   What lands in this PR

   • lance-io: DiskQuota; ObjectStore.disk_quota + LocalWriter enforcement; ScanScheduler::open_reader; SpillStore/SpillFile + LocalSpillStore.
   • lance-core: From<io::Error> downcast recovery (DiskCapExceeded variant already added); remove the old utils/spill.rs.
   • lance: Session integration.
   • Tests for each.

   Follow-up: migrate the IVF v3 shuffler (lance-index/src/vector/v3/shuffler.rs) to SpillFile; it already has the (object_store, output_dir) shape and keeps a TempStdDir alive for cleanup, so it's the natural first consumer.
3. linear-code

   assigned

   wjones127

   on Jun 17, 2026
4. wjones127

   added 2 commits that reference this issue on Jun 17, 2026

   feat: add SpillStore for generic RAII scratch space

   ...

   This issue will close once commit ad19afe is merged into the 'main' branch.

   ad19afe

   feat: add SpillStore for generic RAII scratch space

   ...

   This issue will close once commit b2c0c08 is merged into the 'main' branch.

   b2c0c08
5. wjones127

   closed this as completedin #7311on Jun 23, 2026
6. pull

   added a commit that references this issue on Jun 23, 2026

   feat: add SpillStore trait with local-disk implementation (lance-form…

   ...

   This issue will close once commit bae3c6a is merged into the 'main' branch.

   bae3c6a