fix(review): skip re-embedding unchanged files on full RAG reindex - #4395
Merged
Conversation
…4365) indexRepo (the cron fan-out's full-reindex path) unconditionally re-fetched, re-chunked, and re-embedded every indexable file on every cycle, even when nothing changed. Now it compares each file's git blob SHA against what was stored at last index and skips the fetch/chunk/embed entirely on a match. Also fixes a latent bug where a changed file with fewer chunks than before left stale trailing rows behind.
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4395 +/- ##
=======================================
Coverage 93.96% 93.96%
=======================================
Files 401 401
Lines 36869 36887 +18
Branches 13480 13485 +5
=======================================
+ Hits 34644 34662 +18
Misses 1569 1569
Partials 656 656
🚀 New features to boost your workflow:
|
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
indexRepo) unconditionally re-fetched, re-chunked, and re-embedded every indexable file in a repo on every cycle, regardless of whether the file actually changed since the last index.reindexChangedPathspath), which also fixes a latent bug: a file that shrinks to fewer chunks than it previously had could leave stale trailing chunk rows behind.blob_shacolumn torepo_chunks(migration 0128); existing rows getNULL, which simply never matches and self-heals on the first post-migration reindex.Why
Confirmed live: all 3 gated repos' RAG indexes are fully populated at the
MAX_CHUNKS_PER_REPOcap, and the cron re-embeds unconditionally every cycle today — this closes that waste for the (common) case of unchanged files.Scope
wantedPaths(src/,test/,migrations/)site/,CNAME,**/lovable/**, orCHANGELOG.mdValidation
npm run typechecknpx vitest run test/unit/rag.test.ts test/unit/rag-index.test.ts— 137/137 passingsrc/review/rag.ts+src/review/rag-index.ts— 98.67% stmts / 96.86% branch / 100% funcs / 99.73% lines (the one uncovered line is a pre-existing, untouched catch-all)npm run db:migrations:check— contiguous, no duplicatesnpm run db:schema-drift:check— cleannpm run selfhost:env-reference:check— clean (no newenv.*reads)npm audit --audit-level=moderate— 0 vulnerabilitiesSafety