perf(retrieval): cache the search weighting maps per index generation (v6.0.0) - #200
Conversation
`_owners`, `_weight_generated`, and `_weight_superseded` each rebuilt their maps from a full scan of `chunks`/`notes` on every call, so query latency grew with vault size rather than with the fused candidate set — three full scans per query on top of the fusion work. They now share one `_Weights` snapshot built once per index `generation` and cached per process, the same invalidation contract `_vector_matrix` already used: a local refresh clears it, and a write from another process bumps the generation so the next query rebuilds. Behaviour is unchanged; the superseded alias resolution and the generated-note classification are byte-for-byte the same rules, just computed once. Closes #186. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
📝 WalkthroughSummary by CodeRabbit
WalkthroughSearch weighting data now uses a per-generation cache. Refresh invalidates the cache. Generated and superseded weighting reuse shared maps. Tests cover cache rebuilding, new worklog chunks, and version 6.0.0 metadata. ChangesGeneration-scoped weighting cache
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant SearchIndex
participant SQLite
participant WeightingMethods
SearchIndex->>SQLite: Build weighting maps for the current generation
SQLite-->>SearchIndex: Return ownership and note sets
SearchIndex->>WeightingMethods: Supply cached _Weights data
WeightingMethods-->>SearchIndex: Apply generated and superseded weighting
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@CHANGELOG.md`:
- Around line 10-11: Insert a blank line between the “### Performance” heading
and its list item in CHANGELOG.md to satisfy markdownlint MD022.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 8340fa86-731f-40cd-8d08-b7612bdd2077
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (6)
BACKLOG.mdCHANGELOG.mdpyproject.tomlsrc/omind/__init__.pysrc/omind/searchindex.pytests/test_searchindex.py
📜 Review details
⏰ Context from checks skipped due to timeout. (11)
- GitHub Check: test (ubuntu-latest, 3.14)
- GitHub Check: test (macos-latest, 3.10)
- GitHub Check: test (ubuntu-latest, 3.13)
- GitHub Check: test (windows-latest, 3.10)
- GitHub Check: test (ubuntu-latest, 3.10)
- GitHub Check: test (windows-latest, 3.14)
- GitHub Check: test (ubuntu-latest, 3.12)
- GitHub Check: test (ubuntu-latest, 3.11)
- GitHub Check: test (macos-latest, 3.14)
- GitHub Check: MCP conformance (live stdio)
- GitHub Check: Analyze (python)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
**/*.py: Retrieval must fail open: every search layer returnsNoneon errors and callers fall back to the older path. Missing models, corrupt indexes, locked databases, and missing FTS5 support must degrade search rather than break it; failure branches must be tested.
All note writes must go throughOmiStore; external writers usenotes.upsert_note. Writes must retain flocking, atomic rename, LamportRev:stamping, and soft deletion. Deletes archive notes withDisabled: true; onlyomind mesh purgepermanently removes them.
OmiStore.safe_namemust guard every note read and write so path traversal remains impossible.
Credential notes must be de-prioritized in search and gate suggestions unless the query concerns credentials, using_CREDENTIAL_PENALTY; the gate must never steer agents toward secrets notes.
MCP tools must never return unbounded output. Every list-shaped tool must paginate throughserver._pageand exposelimit,offset,total, andhas_more.
index.mdandMemory Template.mdare scaffolding, not memories; reading either must not clear the consult gate. Preservepaths.NON_CONSULT_FILENAMES.
Recency is a ranking leg only: it may re-rank notes already matched by content legs but must never add unmatched notes to search results.
link_targets()must preserve wikilink case for dangling-link reports; only link resolution lowercases it.
Do not mutate aNoteSummaryreturned from_cached_summary; it is shared cached state. Usedataclasses.replace, as instore._indexed_search.
Treatembed.encoderesults as potentially plain lists in tests; coerce vectors throughsearchindex._query_vectorrather than assuming a.shapeattribute.
Do not reintroduce document-frequency query-term filtering; use threshold-free graded matching, where all-words matches rank above any-words matches.
Files:
src/omind/__init__.pytests/test_searchindex.pysrc/omind/searchindex.py
**/*.md
📄 CodeRabbit inference engine (AGENTS.md)
Markdown note files are the source of truth; indexes, caches, and vectors are derived and disposable and must live only in
paths.state_dir(), never in the vault.
Files:
BACKLOG.mdCHANGELOG.md
🪛 markdownlint-cli2 (0.23.1)
CHANGELOG.md
[warning] 10-10: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below
(MD022, blanks-around-headings)
🔇 Additional comments (5)
src/omind/searchindex.py (1)
179-192: LGTM!Also applies to: 478-479, 636-636, 994-1047, 1063-1079
tests/test_searchindex.py (1)
266-281: LGTM!BACKLOG.md (1)
17-20: LGTM!pyproject.toml (1)
3-3: LGTM!src/omind/__init__.py (1)
5-5: LGTM!
| ### Performance | ||
| - **The search weighting passes no longer re-scan the whole index on every |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add a blank line after the heading.
markdownlint-cli2 reports MD022 at Line 10. Insert one blank line before the list item at Line 11.
🧰 Tools
🪛 markdownlint-cli2 (0.23.1)
[warning] 10-10: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below
(MD022, blanks-around-headings)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@CHANGELOG.md` around lines 10 - 11, Insert a blank line between the “###
Performance” heading and its list item in CHANGELOG.md to satisfy markdownlint
MD022.
Source: Linters/SAST tools
1 participant
Closes #186.
What
_owners,_weight_generated, and_weight_supersededeach rebuilt their maps from a full scan ofchunks/noteson every query. Three full-table Python scans per search, so latency grew with vault size instead of with the fused candidate set.They now share a single
_Weightssnapshot (chunk→filename owners, the generated chunk-id set, the superseded filename set) built once per indexgenerationand cached per process — the same invalidation contract_vector_matrixalready used.Why it stays correct
generation, so a write from any process invalidates it on the next query, not just a localrefresh()(which also clears it explicitly).okf_type/filename prefixes for generated notes, sameSupersedes:/Superseded by:alias resolution. Only the number of times they run changed.test_weighting_maps_are_built_once_per_generationpins both halves: a second query reuses the cache, and a refresh after a new note rebuilds it.Version
Bumped to 6.0.0.
mainwas already carrying two unreleased breaking changes with no release cut — the MCP 2.x SDK adoption (#192) and the removal of the deprecatedgraph-*MCP aliases (#185) — so the next number off5.0.1is a major regardless of this change. The## [Unreleased]changelog section is closed out as6.0.0.Gates
ruff check .·mypy src(strict, 46 files) ·pytest(842 passed, 1 skipped) ·pip-audit— all green locally.🤖 Generated with Claude Code