feat(miner): cache policy docs with conditional-GET ETags in discovery (#4842) - #5474
feat(miner): cache policy docs with conditional-GET ETags in discovery (#4842)#5474rust-toml wants to merge 2 commits into
Conversation
JSONbored#4842) `discover` refetched each target repo's AI-USAGE.md/CONTRIBUTING.md from the GitHub Contents API on every run, spending rate-limit budget on near-static content. Add a local ETag cache (policy-doc-cache.sqlite3, the fifth local SQLite store) so the fan-out revalidates those docs with an If-None-Match conditional GET: a 304 reuses the cached body and costs no primary rate-limit budget. Stale-safe by construction -- a conditional GET always revalidates in the same run, so a 304 body is GitHub-confirmed current, and every miss, absent/blank ETag, read/write error, or non-304 response falls back to a full fetch. The feature is inert unless a cache is supplied: discover-cli wires the on-disk store in and owns its lifecycle (mirroring the portfolio queue), while the fan-out library defaults to no cache, so prior behavior is byte-identical. Closes JSONbored#4842.
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Signed-off-by: silver <168391793+rust-toml@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5474 +/- ##
==========================================
+ Coverage 94.72% 94.73% +0.01%
==========================================
Files 559 560 +1
Lines 44664 44705 +41
Branches 14664 14664
==========================================
+ Hits 42308 42353 +45
+ Misses 1621 1617 -4
Partials 735 735
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
loopover-orb
Bot
added
the
gittensor:feature
|
Caution 🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥 🛑 Gittensory review result - fixes requiredReview updated: 2026-07-12 22:17:10 UTC
🛑 Suggested Action - Fix Blockers
Review summary Nits — 1 non-blocking
CI checks failing
Review context
Contributor next steps
Signal definitions
[BETA] Chat with GittensoryAsk Gittensory a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://gittensory.aethereal.dev/docs/gittensory-commands 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.
|
|
Gittensory is closing this pull request on the maintainer's behalf (CI is failing (validate, validate-code); conflicts with the base branch — resolve and open a fresh PR). This is an automated maintenance action — to pursue this change, please open a new pull request with the issues resolved. Closed PRs may be analyzed later to improve review accuracy, but they are not automatically reopened or re-reviewed. |
1 participant
Summary
Closes #4842
Discovery refetched every target repo's policy docs (
AI-USAGE.md/CONTRIBUTING.md) from the GitHub Contents API on everydiscoverrun, spending rate-limit budget on content that rarely changes. This adds a small local ETag cache so those fetches become conditional GETs (If-None-Match): when GitHub answers304 Not Modified, discovery reuses the cached body and GitHub charges no primary rate-limit budget for therequest.
The cache is the fifth local SQLite store (
policy-doc-cache.sqlite3, tablepolicy_doc_cache), built on the samelocal-store.jspath/permission boilerplate as the existing four. It holds no miner state — deleting the file only forces the next run to refetch in full.Stale-safe by construction: a conditional GET always revalidates against GitHub in the same run, so a
304body is GitHub-confirmed-current, never served blindly. Any cache miss, absent/blank ETag, read/write error, or non-304 response falls back to a full fetch, so the feature can only ever make discovery cheaper — never let a stale policy wrongly permit autonomous work on an opted-out repo. It is inert unless a cache is supplied;discoverwires the on-disk store in, and the fan-out library defaults to no cache (byte-identical prior behavior).Scope
type(scope): short summaryConventional Commit format.CONTRIBUTING.mdand does not reintroduce GitHub Pages, VitePress,site/, orCNAME.Closes #4842.Validation
git diff --checknpm run typechecknpm run test:coverage— new/changed lines and branches inpolicy-doc-cache.js,opportunity-fanout.js, anddiscover-cli.jsare 100% covered on the diff (verified by intersecting the diff's added lines against the v8 uncovered-line set).npm run build:miner(newlib/policy-doc-cache.jsadded to the package'snode --checklist)npm run miner:env-reference:check(regenerateddocs/env-reference.mdfor the new env var)npm run db:migrations:check/npm run db:schema-drift:check(no schema change; the cache uses a raw local-store table)policy-doc-cachefiles accepted)304revalidation, absent/blank ETag, undecodable body, cache read-error and write-error fallbacks, thrown-fetch path, no-cache inert path, and cross-run on-disk persistence.No secrets, wallets, hotkeys, trust scores, or private scoring are touched. Not applicable to this change:
actionlint(no workflow edits),test:workers/build:mcp/test:mcp-pack(no worker/MCP change), and theui:*checks + UI Evidence (no UI/frontend change).npm audit --audit-level=moderatereports 0 vulnerabilities(no dependency change).
Safety
If-None-Matchrequest header to an existing unauthenticated-capable Contents GET; no auth surface changes.README.md"Local storage" table + generateddocs/env-reference.md.Notes
The cache never weakens the AI-policy hard-skip: a
304returns the exact previously-fetched body, and every non-304 path re-runs the full fetch + parse, soresolveAiPolicyVerdictsees the same input it would without the cache.