Skip to content

fix(core): search complete SQLite note content - #1071

Merged
phernandez merged 1 commit into
mainfrom
codex/1065-sqlite-full-content-search
Jul 16, 2026
Merged

fix(core): search complete SQLite note content#1071
phernandez merged 1 commit into
mainfrom
codex/1065-sqlite-full-content-search

Conversation

@phernandez

Copy link
Copy Markdown
Member

Why

SQLite text search only matched title and the Postgres-sized content_stems prefix. Because content_stems is capped near 6,000 characters while the full note is already indexed in FTS5 as content_snippet, exact terms later in large notes were invisible.

Closes #1065.

What Changed

  • Include SQLite's full content_snippet FTS5 column in text-search matching.
  • Add a regression test with a marker after 7,000 characters that is absent from content_stems.
  • Verify both result retrieval and count use the complete content.

Implementation Details

The existing SQLite query builder is shared by search() and count(), including the metadata-filter subquery path. Extending that single match predicate fixes all SQLite keyword consumers without changing the stored index or requiring users to reindex.

Postgres behavior is unchanged; its indexed content_stems remains capped for the 8 KB index-row constraint.

Testing

  • uv run pytest tests/repository/test_search_repository.py -q — 48 passed.
  • just fast-check — passed (ruff, format, and type checking; existing Python 3.14 deprecation warnings only).
  • just fast-test — 123 passed and 1 skipped before the selected live OpenAI/Postgres benchmark failed with external 429 insufficient_quota.
  • env OPENAI_API_KEY= just fast-test — the remaining live OpenAI case skipped as intended.

Risks / Follow-ups

SQLite BM25 ranking can shift slightly because terms present in both content_stems and content_snippet contribute from both columns. That is preferable to silently dropping exact matches in long notes, and no migration or reindex is required.

Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez
phernandez merged commit 5e4b119 into main Jul 16, 2026
23 checks passed
@phernandez
phernandez deleted the codex/1065-sqlite-full-content-search branch July 16, 2026 05:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SQLite text search misses all content beyond ~6000 chars (content_stems truncated, content_snippet never matched)

1 participant