Skip to content

Lazy secondary index fd release on startup#340

Draft
albe with Copilot wants to merge 2 commits into
mainfrom
copilot/implement-lazy-index-opening
Draft

Lazy secondary index fd release on startup#340
albe with Copilot wants to merge 2 commits into
mainfrom
copilot/implement-lazy-index-opening

Conversation

Copilot AI commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Secondary indexes were holding open file descriptors throughout startup, and pool LRU evictions (triggered when N > 1024 indexes) unconditionally called fdatasync even for indexes with no pending writes — making startup cost superlinear in index count.

Changes

WritableIndex.onBeforeClose

  • Skip fdatasync when writeBufferCursor === 0; previous guard (!this.writeBuffer) was always false since writeBuffer is allocated in initialize

ReadableStorage.openIndex + afterRegisterSecondaryIndex hook

  • Replace the no-op index.open() call with a new afterRegisterSecondaryIndex(index) hook
  • Default implementation immediately evicts the fd via fileHandlePool.evict(index, false)false preserves opened=true and the data array so index.length/index.lastEntry remain valid; the pool transparently reopens the fd on first real access

WritableStorage.afterRegisterSecondaryIndex

  • Replaces the openIndex override: performs stale-entry truncation (needed after crash-recovery checkTornWrites) while the fd is still live, then delegates to super to release it

bench/bench-scalability.js

  • Fixed measureStartup/measureWrite/measureRead to await storage.open(callback) so the async partition scan completes before reads — was causing Partition does not exist errors at runtime

Benchmark (Node v24, ~108 bytes/doc)

Scenario A — 1 index/partition:

Partitions main PR #339 this PR
1 000 47.2 ms 40.5 ms 50.5 ms
5 000 225.8 ms 176.6 ms 210.7 ms
10 000 430.2 ms 352.7 ms 401.9 ms
20 000 820.7 ms 717.4 ms 794.5 ms

Scenario B — 100 partitions, growing index count:

Indexes main PR #339 this PR
100 5.8 ms 42.3 ms 4.0 ms
1 000 27.4 ms 21.1 ms 33.0 ms
5 000 125.9 ms 98.9 ms 117.5 ms

At low index counts the lazy fd release wins (Scenario B/100: −31% vs main); PR #339's snapshot approach dominates at high counts by skipping file opens entirely. The two are complementary. Write/read per-op latency is unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants