feat(miner): cache policy docs with conditional-GET ETags in discovery (#4842)

[rust-toml]

Summary

Discovery refetched every target repo's policy docs (AI-USAGE.md / CONTRIBUTING.md) from the GitHub
Contents API on every discover run, spending rate-limit budget on content that rarely changes. This adds
a small local ETag cache so those fetches become conditional GETs (If-None-Match): when GitHub answers
304 Not Modified, discovery reuses the cached body and GitHub charges no primary rate-limit budget for the
request.

The cache is the fifth local SQLite store (policy-doc-cache.sqlite3, table policy_doc_cache), built on the
same local-store.js path/permission boilerplate as the existing four. It holds no miner state — deleting the
file only forces the next run to refetch in full.

Stale-safe by construction: a conditional GET always revalidates against GitHub in the same run, so a 304
body is GitHub-confirmed-current, never served blindly. Any cache miss, absent/blank ETag, read/write error, or
non-304 response falls back to a full fetch, so the feature can only ever make discovery cheaper — never let a
stale policy wrongly permit autonomous work on an opted-out repo. It is inert unless a cache is supplied;
discover wires the on-disk store in, and the fan-out library defaults to no cache (byte-identical prior
behavior).

Closes #4842

Scope

• The PR title follows type(scope): short summary Conventional Commit format.
• This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
• This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
• I linked a currently open issue this PR resolves — Closes #4842.

Validation

Branch is rebased on current main (0 commits behind, no conflicts); all checks below were run against the
merged tree, including its interaction with the fan-out rate-limit throttling already on main.

• git diff --check
• npm run typecheck
• npm run test:coverage — new/changed lines and branches in policy-doc-cache.js, opportunity-fanout.js, and discover-cli.js are 100% covered on the diff (verified by intersecting the diff's added lines against the v8 uncovered-line set). The broader fan-out suite (105 tests) also passes with the change applied.
• npm run build:miner (new lib/policy-doc-cache.js added to the package's node --check list)
• npm run miner:env-reference:check (regenerated docs/env-reference.md for the new env var)
• npm run db:migrations:check / npm run db:schema-drift:check (no schema change; the cache uses a raw local-store table)
• Miner package pack allow-list validated (both policy-doc-cache files accepted)
• New behavior has unit tests for every branch: cold-miss caching, 304 revalidation, absent/blank ETag, undecodable body, cache read-error and write-error fallbacks, thrown-fetch path, no-cache inert path, and cross-run on-disk persistence.

No secrets, wallets, hotkeys, trust scores, or private scoring are touched. Not applicable to this change:
actionlint (no workflow edits), test:workers / build:mcp / test:mcp-pack (no worker/MCP change), and the
ui:* checks (no UI/frontend change). npm audit --audit-level=moderate reports 0 vulnerabilities (no
dependency change).

Safety

• No secrets, wallet details, hotkeys, coldkeys, PATs, private keys, raw trust scores, private rankings, or private evidence are exposed.
• Public GitHub text stays sanitized and low-noise (this change adds no public output).
• Auth/cookie/CORS/GitHub App/session: n/a — the only network change is adding an If-None-Match request header to an existing unauthenticated-capable Contents GET; no auth surface changes.
• API/OpenAPI/MCP behavior: n/a (no API/MCP surface change).
• UI changes: n/a.
• Public docs updated: miner README.md "Local storage" table + generated docs/env-reference.md.

Notes

The cache never weakens the AI-policy hard-skip: a 304 returns the exact previously-fetched body, and every
non-304 path re-runs the full fetch + parse, so resolveAiPolicyVerdict sees the same input it would without the
cache.

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.75%. Comparing base (849e11e) to head (acfe2a3).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5478      +/-   ##
==========================================
+ Coverage   94.73%   94.75%   +0.01%     
==========================================
  Files         561      562       +1     
  Lines       44794    44834      +40     
  Branches    14667    14667              
==========================================
+ Hits        42437    42481      +44     
+ Misses       1622     1618       -4     
  Partials      735      735              

Flag	Coverage Δ
shard-1	43.74% <0.00%> (-0.54%)	⬇️
shard-2	35.36% <20.93%> (+0.08%)	⬆️
shard-3	31.69% <48.83%> (-0.34%)	⬇️
shard-4	31.63% <20.93%> (+0.23%)	⬆️
shard-5	32.82% <90.69%> (-0.40%)	⬇️
shard-6	44.21% <13.95%> (+0.61%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
packages/gittensory-miner/lib/discover-cli.js	100.00% <100.00%> (ø)
...ackages/gittensory-miner/lib/opportunity-fanout.js	98.00% <100.00%> (+2.30%)	⬆️
packages/gittensory-miner/lib/policy-doc-cache.js	100.00% <100.00%> (ø)

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[loopover-orb]

Caution

🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥

🛑 Gittensory review result - reject/close recommended

_{Review updated: 2026-07-12 22:52:47 UTC}

13 files · 2 AI reviewers · 1 blocker · readiness 100/100 · CI green · clean

🛑 Suggested Action - Reject/Close

Review summary
This adds a local SQLite ETag cache (packages/gittensory-miner/lib/policy-doc-cache.js) and wires it into opportunity-fanout.js's fetchRepoDoc so a repeated `discover` sends `If-None-Match` and can reuse the cached body on a GitHub 304, cutting rate-limit spend on unchanged AI-USAGE.md/CONTRIBUTING.md fetches. The stale-safety argument holds: the 304 branch (opportunity-fanout.js:187) is only reachable when `cached` is non-null because the conditional header is only sent when a cache entry exists, so `cached.content` can't be read from a null. Cache miss, read/write errors, missing/blank ETag, and non-304/404 responses all correctly fall back to a full fetch and never fail discovery, and the feature is inert when no cache is injected (byte-identical to prior behavior) — well-scoped, matches its description, and closes the linked issue #4842 with real (non-fabricated) tests exercising the actual fanout call path.

Blockers

• packages/gittensory-miner/lib/discover-cli.js:158 opens the optional policy-doc cache outside the try/fallback path, so a corrupt or unwritable policy-doc cache DB rejects `runDiscover` before discovery can proceed; change it to fall back to `null` for the fan-out cache on open failure, e.g. `let policyDocCache = null; let ownsPolicyDocCache = false; try { ownsPolicyDocCache = options.initPolicyDocCache === undefined; policyDocCache = (options.initPolicyDocCache ?? initPolicyDocCacheStore)(); } catch { policyDocCache = null; ownsPolicyDocCache = false; }` and close only when a store was actually opened.

Nits — 5 non-blocking

• packages/gittensory-miner/lib/opportunity-fanout.js has grown past ~430 lines; consider splitting the policy-doc cache read/write helpers into their own module alongside policy-doc-cache.js if it grows further.
• packages/gittensory-miner/lib/discover-cli.js opens/owns the policy-doc cache store even when `--search` is passed and no target repos are given up front; harmless but worth a comment on why it's unconditional.
• A decoded empty-string policy-doc body (payload.content === "") would still be cached by writeCachedPolicyDoc since only `content !== null` is checked, not non-empty — low-risk edge case worth a one-line guard or comment.
• Consider adding a short comment at opportunity-fanout.js:157 (the catch-and-warn-null pattern) noting this is intentional fail-open-to-full-fetch behavior, since a future reader might mistake it for a swallowed error.
• docs/env-reference.md and README.md updates look complete and consistent with the other four local stores — no changes needed there.

Why this is blocked

• packages/gittensory-miner/lib/discover-cli.js:158 opens the optional policy-doc cache outside the try/fallback path, so a corrupt or unwritable policy-doc cache DB rejects `runDiscover` before discovery can proceed; change it to fall back to `null` for the fan-out cache on open failure, e.g. `let policyDocCache = null; let ownsPolicyDocCache = false; try { ownsPolicyDocCache = options.initPolicyDocCache === undefined; policyDocCache = (options.initPolicyDocCache ?? initPolicyDocCacheStore)(); } catch { policyDocCache = null; ownsPolicyDocCache = false; }` and close only when a store was actually opened.

📋 Copy for AI agents — paste into your coding agent

Fix the following blocker(s) from this PR review:

1. packages/gittensory-miner/lib/discover-cli.js:158 opens the optional policy-doc cache outside the try/fallback path, so a corrupt or unwritable policy-doc cache DB rejects \`runDiscover\` before discovery can proceed; change it to fall back to \`null\` for the fan-out cache on open failure, e.g. \`let policyDocCache = null; let ownsPolicyDocCache = false; try \{ ownsPolicyDocCache = options.initPolicyDocCache === undefined; policyDocCache = \(options.initPolicyDocCache ?? initPolicyDocCacheStore\)\(\); \} catch \{ policyDocCache = null; ownsPolicyDocCache = false; \}\` and close only when a store was actually opened.

Signal	Result	Evidence
Code review	❌ 1 blocker	2 reviewers, synthesized
Linked issue	✅ Linked	#4842
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 17 registered-repo PR(s), 8 merged, 5 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	rust-toml; Gittensor profile; 17 PR(s), 5 issue(s).
Gate result	❌ Blocking	Repo-configured hard blocker found.
Improvement	✅ Minor	risk: clean · value: minor — Code changes are accompanied by test evidence. LLM value judgment: moderate — The change is a well-targeted, correctly-implemented rate-limit optimization for an existing hot path (policy-doc fetch in discovery) with thorough real-path test coverage and no scope creep beyond the linked issue.

Linked issue satisfaction

Addressed
The PR implements ETag-based conditional GET for policy-doc fetches with a local SQLite cache, sends If-None-Match on repeat requests, serves cached content on 304 without extra rate-limit spend, and fails safe to a full fetch on any cache miss/error/ambiguous state, matching the issue's scope and acceptance criteria.

Review context

• Author: rust-toml
• Role context: outside_contributor
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: TypeScript, JavaScript, Rust, C++, Go, Java, Python, Solidity
• Official Gittensor activity: 17 PR(s), 5 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Keep the PR focused and include validation evidence before maintainer review.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

[BETA] Chat with Gittensory

Ask Gittensory a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

• @gittensory ask <question> answers contribution-quality Q&A with source citations and freshness.
• @gittensory chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
• A plain-language @gittensory mention with a real question is routed to the closest matching read-only command automatically -- no exact syntax required.

Full command reference: https://gittensory.aethereal.dev/docs/gittensory-commands

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

• Re-run Gittensory review

[loopover-orb]

Gittensory is closing this pull request on the maintainer's behalf (AI reviewers agree on a likely critical defect: packages/gittensory-miner/lib/discover-cli.js:158 opens the optional policy-doc cache outside the try/fallback path, so a corrupt or unwritable policy-doc cache DB rejects `runDiscover` before discovery can proceed; change it to fall back to `null` for the fan-out cache on open failure, e.g. `let policyDocCache = null; let ownsPolicyDocCache = false; try { ownsPolicyDocCache = options.initPolicyDocCache === undefined; policyDocCache = (options.initPolicyDocCache ?? initPolicyDocCacheStore)(); } catch { policyDocCache = null; ownsPolicyDocCache = false; }` and close only when a store was actually opened.). This is an automated maintenance action — to pursue this change, please open a new pull request with the issues resolved. Closed PRs may be analyzed later to improve review accuracy, but they are not automatically reopened or re-reviewed.