Repository navigation
Indexer ingests .stversions backups, sync-conflict copies and .obsidian junk as notes #1
Description
What happens
index_notes.py collects notes with sorted(notes_dir.rglob('*.md')) (line 51). rglob descends
into dot-directories, so on a real Obsidian vault — the exact audience of this repo — the index fills
with files that are not notes, and stale copies of a note compete with the live one at retrieval
time.
Repro
from pathlib import Path
import tempfile
d = Path(tempfile.mkdtemp())
(d / "real.md").write_text("real note")
(d / ".stversions").mkdir(); (d / ".stversions" / "real~20260701.md").write_text("OLD VERSION")
(d / "real.sync-conflict-20260801-ABC.md").write_text("CONFLICT COPY")
(d / ".obsidian").mkdir(); (d / ".obsidian" / "cache.md").write_text("plugin junk")
(d / ".git").mkdir(); (d / ".git" / "COMMIT_EDITMSG.md").write_text("git junk")
print(sorted(p.relative_to(d).as_posix() for p in d.rglob("*.md")))['.git/COMMIT_EDITMSG.md', '.obsidian/cache.md', '.stversions/real~20260701.md',
'real.md', 'real.sync-conflict-20260801-ABC.md']
5 files indexed, 1 of them an actual note. Anyone syncing a vault with Syncthing has both
.stversions/ and *.sync-conflict-* files; anyone using Obsidian has .obsidian/.
Why it bites: an old version of a note is semantically almost identical to the live one, so it
lands next to it in the top-K and quietly eats a retrieval slot with outdated content. The answer
looks right and is stale.
Second, smaller bug in the same line
Line 53 reads with errors='ignore', which also swallows the UTF-8 BOM:
Path("bom.md").write_bytes(b"\xef\xbb\xbf---\ntitle: x\n---\nbody")
Path("bom.md").read_text(encoding="utf-8", errors="ignore")[:12] # '---\ntitle: 'The leading means the first line is not ---, so any frontmatter parser downstream sees the
note as having no frontmatter. encoding="utf-8-sig" handles this correctly.
Shape of the fix
- Skip dot-directories while walking, and skip
*.sync-conflict-*by name. - Read with
encoding="utf-8-sig". - Make the exclusion list overridable (env var, consistent with the repo's env-only configuration) —
someone will legitimately want to index a hidden folder.
Definition of done
- The repro above indexes exactly
real.md. - A test with a synthetic tree covering: dot-dir, sync-conflict name, BOM file, normal note.
python index_notes.py <dir>still prints the same summary line for a clean directory.
Comment "claiming this" to take it — yours for 7 days.
Activity
fix(index): stop indexing .stversions, sync-conflict copies and dot-d…
tonydzi commentedon Sep 5, 2026
Fixed in 698a6c7.
Both bugs in those two lines are gone:
*sync-conflict*names are skipped, so.stversions/revisions,.obsidian/cache and.git/internals no longer enter the index. The repro from the description now yields['real.md']and nothing else.BRAIN_INDEX_HIDDEN=1opts hidden folders back in — someone will legitimately keep notes in one.utf-8-sig, so a BOM'd note starts with---again and its frontmatter (date, title) is parsed instead of silently lost.errors='replace'rather thanignore, so a damaged file degrades visibly instead of quietly dropping characters.The selection and reading now live in
iter_notes()/read_note()— a seam that needs no model and no network, which is also what #2 asked for to make tests possible at all.Tests: 7 cases, and they were shown red before the fix. With the seam extracted and the old behaviour deliberately kept:
5 failed, 2 passed. With the fix:7 passed in 5.80s. A test that has never been red proves nothing, so the red run came first on purpose.Worth saying plainly: this was reported by us and fixed by us, which is not the same as a contributor finding it. The queue is still open — #2 (a smoke test that runs without downloading a model) is the natural next one and is a genuine design conversation, not a chore.
docs: describe test coverage by shape, not by a number that rots
Add TangleClaw hybrid graph-temporal memory adapter to memory-bench (#21