Skip to content

fix(review): skip re-embedding unchanged files on full RAG reindex - #4395

Merged
JSONbored merged 1 commit into
mainfrom
claude/review-cache-and-embed-cache
Jul 9, 2026
Merged

fix(review): skip re-embedding unchanged files on full RAG reindex#4395
JSONbored merged 1 commit into
mainfrom
claude/review-cache-and-embed-cache

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • The cron fan-out's full-reindex path (indexRepo) unconditionally re-fetched, re-chunked, and re-embedded every indexable file in a repo on every cycle, regardless of whether the file actually changed since the last index.
  • Now compares each file's git blob SHA (already returned free on the tree-fetch response) against what was stored at last index, and skips the fetch/chunk/embed entirely when it matches.
  • A changed file's old chunks are now deleted before its new ones are inserted (mirrors the existing incremental reindexChangedPaths path), which also fixes a latent bug: a file that shrinks to fewer chunks than it previously had could leave stale trailing chunk rows behind.
  • Adds a nullable blob_sha column to repo_chunks (migration 0128); existing rows get NULL, which simply never matches and self-heals on the first post-migration reindex.

Why

Confirmed live: all 3 gated repos' RAG indexes are fully populated at the MAX_CHUNKS_PER_REPO cap, and the cron re-embeds unconditionally every cycle today — this closes that waste for the (common) case of unchanged files.

Scope

  • Stayed within wantedPaths (src/, test/, migrations/)
  • No secrets/wallets/trust-score/reward values anywhere
  • No changes to site/, CNAME, **/lovable/**, or CHANGELOG.md

Validation

  • npm run typecheck
  • npx vitest run test/unit/rag.test.ts test/unit/rag-index.test.ts — 137/137 passing
  • Scoped coverage check on src/review/rag.ts + src/review/rag-index.ts — 98.67% stmts / 96.86% branch / 100% funcs / 99.73% lines (the one uncovered line is a pre-existing, untouched catch-all)
  • npm run db:migrations:check — contiguous, no duplicates
  • npm run db:schema-drift:check — clean
  • npm run selfhost:env-reference:check — clean (no new env.* reads)
  • npm audit --audit-level=moderate — 0 vulnerabilities

Safety

  • No secrets/tokens in code, comments, tests, or this description
  • Fail-safe preserved: every new code path degrades to "reprocess as before" on any ambiguity (missing SHA, storage-read error), never silently skips real work

…4365)

indexRepo (the cron fan-out's full-reindex path) unconditionally re-fetched,
re-chunked, and re-embedded every indexable file on every cycle, even when
nothing changed. Now it compares each file's git blob SHA against what was
stored at last index and skips the fetch/chunk/embed entirely on a match.
Also fixes a latent bug where a changed file with fewer chunks than before
left stale trailing rows behind.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored JSONbored self-assigned this Jul 9, 2026
@JSONbored JSONbored added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 9, 2026
@codecov

codecov Bot commented Jul 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.96%. Comparing base (ca5576f) to head (27ef7e0).
⚠️ Report is 4 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4395   +/-   ##
=======================================
  Coverage   93.96%   93.96%           
=======================================
  Files         401      401           
  Lines       36869    36887   +18     
  Branches    13480    13485    +5     
=======================================
+ Hits        34644    34662   +18     
  Misses       1569     1569           
  Partials      656      656           
Files with missing lines Coverage Δ
src/review/rag-index.ts 93.90% <100.00%> (+0.43%) ⬆️
src/review/rag.ts 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JSONbored
JSONbored merged commit 2aba3fe into main Jul 9, 2026
11 checks passed
@JSONbored
JSONbored deleted the claude/review-cache-and-embed-cache branch July 9, 2026 08:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Development

Successfully merging this pull request may close these issues.

1 participant