feat(miner): cache policy docs with conditional-GET revalidation - #5508
Merged
Conversation
Add a local ETag cache (policy-doc-cache.js) so a repeated discover run revalidates each repo's AI-USAGE.md/CONTRIBUTING.md with If-None-Match instead of re-downloading them on every pass. A 304 serves the cached body (GitHub-confirmed current); any cache read/write failure degrades to "no cache" rather than affecting discovery, and the store itself is opened in its own try/catch in discover-cli.js so a corrupt/unwritable cache DB can never abort a run. Closes #4842
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
gittensory-ui | 62b677a | Commit Preview URL Branch Preview URL |
Jul 12 2026, 11:51 PM |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5508 +/- ##
==========================================
+ Coverage 94.75% 94.77% +0.01%
==========================================
Files 565 566 +1
Lines 44920 44965 +45
Branches 14675 14675
==========================================
+ Hits 42565 42614 +49
+ Misses 1621 1617 -4
Partials 734 734
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Contributor
|
Important 🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪 🔍 Gittensory is reviewing…AI analysis is in progress. This comment will update when the review is complete. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟪 Reviewing |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
policy-doc-cache.js, a local SQLite ETag cache for repo policy docs (AI-USAGE.md/CONTRIBUTING.md), following the same local-store conventions as the miner's other stores (local-store.js+schema-version.js).opportunity-fanout.js'sfetchRepoDocnow revalidates a cached doc withIf-None-Matchand serves the cached body on a real304; a fresh200with a non-blank ETag is written back to the cache. Both cache read and write failures degrade to "no cache" — discovery is never affected by a corrupt/unwritable cache DB.discover-cli.jsopens the cache in its own try/catch, separate from the requiredportfolioQueueopen: the cache is a pure performance optimization (the feature is inert without one), so a real open failure there must degrade topolicyDocCache: nullrather than aborting the whole run. This is the one blocker two prior contributor attempts (feat(miner): cache policy docs with conditional-GET ETags in discovery (#4842) #5474, feat(miner): cache policy docs with conditional-GET ETags in discovery (#4842) #5478) hit — feat(miner): cache policy docs with conditional-GET ETags in discovery (#4842) #5478 got everything else right but opened the cache store outside any try/catch.docs/env-reference.md(newGITTENSORY_MINER_POLICY_DOC_CACHE_DB) and added the new store to the README's local-storage table.Test plan
npx vitest run test/unit test/contract— 777 files / 15147 tests passing, 0 regressionsnpm run build --workspace @jsonbored/gittensory-miner(node --checkover every lib file, including the new one)npx tsc --noEmitcleannpm run docs:drift-check && npm run manifest:drift-check && npm run engine-parity:drift-check && npm run miner:env-reference:checknpm audit --audit-level=moderate— 0 vulnerabilitiesinitPolicyDocCachestill yieldsexitCode === 0andpolicyDocCache: nulldownstreamCloses #4842