Skip to content

feat(miner): cache policy docs with conditional-GET revalidation - #5508

Merged
JSONbored merged 1 commit into
mainfrom
feat/policy-doc-cache-4842
Jul 12, 2026
Merged

feat(miner): cache policy docs with conditional-GET revalidation#5508
JSONbored merged 1 commit into
mainfrom
feat/policy-doc-cache-4842

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • Adds policy-doc-cache.js, a local SQLite ETag cache for repo policy docs (AI-USAGE.md/CONTRIBUTING.md), following the same local-store conventions as the miner's other stores (local-store.js + schema-version.js).
  • opportunity-fanout.js's fetchRepoDoc now revalidates a cached doc with If-None-Match and serves the cached body on a real 304; a fresh 200 with a non-blank ETag is written back to the cache. Both cache read and write failures degrade to "no cache" — discovery is never affected by a corrupt/unwritable cache DB.
  • discover-cli.js opens the cache in its own try/catch, separate from the required portfolioQueue open: the cache is a pure performance optimization (the feature is inert without one), so a real open failure there must degrade to policyDocCache: null rather than aborting the whole run. This is the one blocker two prior contributor attempts (feat(miner): cache policy docs with conditional-GET ETags in discovery (#4842) #5474, feat(miner): cache policy docs with conditional-GET ETags in discovery (#4842) #5478) hit — feat(miner): cache policy docs with conditional-GET ETags in discovery (#4842) #5478 got everything else right but opened the cache store outside any try/catch.
  • Regenerated docs/env-reference.md (new GITTENSORY_MINER_POLICY_DOC_CACHE_DB) and added the new store to the README's local-storage table.

Test plan

  • npx vitest run test/unit test/contract — 777 files / 15147 tests passing, 0 regressions
  • npm run build --workspace @jsonbored/gittensory-miner (node --check over every lib file, including the new one)
  • npx tsc --noEmit clean
  • npm run docs:drift-check && npm run manifest:drift-check && npm run engine-parity:drift-check && npm run miner:env-reference:check
  • npm audit --audit-level=moderate — 0 vulnerabilities
  • Dedicated regression test for the exact bug that closed feat(miner): cache policy docs with conditional-GET ETags in discovery (#4842) #5478: a throwing initPolicyDocCache still yields exitCode === 0 and policyDocCache: null downstream

Closes #4842

Add a local ETag cache (policy-doc-cache.js) so a repeated discover run
revalidates each repo's AI-USAGE.md/CONTRIBUTING.md with If-None-Match
instead of re-downloading them on every pass. A 304 serves the cached
body (GitHub-confirmed current); any cache read/write failure degrades
to "no cache" rather than affecting discovery, and the store itself is
opened in its own try/catch in discover-cli.js so a corrupt/unwritable
cache DB can never abort a run.

Closes #4842
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
gittensory-ui 62b677a Commit Preview URL

Branch Preview URL
Jul 12 2026, 11:51 PM

@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.77%. Comparing base (f6e5d07) to head (62b677a).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5508      +/-   ##
==========================================
+ Coverage   94.75%   94.77%   +0.01%     
==========================================
  Files         565      566       +1     
  Lines       44920    44965      +45     
  Branches    14675    14675              
==========================================
+ Hits        42565    42614      +49     
+ Misses       1621     1617       -4     
  Partials      734      734              
Flag Coverage Δ
shard-1 43.75% <0.00%> (-0.55%) ⬇️
shard-2 35.37% <18.75%> (+0.03%) ⬆️
shard-3 32.11% <54.16%> (+0.17%) ⬆️
shard-4 31.36% <18.75%> (-0.60%) ⬇️
shard-5 33.04% <81.25%> (+0.11%) ⬆️
shard-6 44.16% <12.50%> (+0.61%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
packages/gittensory-miner/lib/discover-cli.js 100.00% <100.00%> (ø)
...ackages/gittensory-miner/lib/opportunity-fanout.js 98.08% <100.00%> (+2.18%) ⬆️
packages/gittensory-miner/lib/policy-doc-cache.js 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@loopover-orb loopover-orb Bot added the gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. label Jul 12, 2026
@loopover-orb

loopover-orb Bot commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Important

🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪

🔍 Gittensory is reviewing…

AI analysis is in progress. This comment will update when the review is complete.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟪 Reviewing

@JSONbored JSONbored self-assigned this Jul 12, 2026
@JSONbored
JSONbored merged commit a798778 into main Jul 12, 2026
19 checks passed
@JSONbored
JSONbored deleted the feat/policy-doc-cache-4842 branch July 12, 2026 23:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add conditional-GET caching for policy-doc fetches

1 participant