fix(db): dedupe signal_snapshots to bounded latest-per-key retention

[JSONbored]

Summary

• signal_snapshots has no dedup: generate-signal-snapshots inserts a new row per (signal_type, target_key) on every run instead of replacing the prior one. Within the existing 90-day retention window this let a single key accumulate hundreds of superseded rows — 342,243 rows for 2,183 distinct keys, contributing to hitting D1's size cap on 2026-07-06.
• Adds dedupeSignalSnapshots (src/db/retention.ts), which keeps only the latest row (highest rowid) per (signal_type, target_key), batched per signal_type so each DELETE stays within D1's per-statement CPU budget — the same batching split used during the incident's manual remediation.
• Wires it into the existing daily prune-retention job (runRetentionPrune), merging its audit-event detail/metadata with the existing prune results, and into the read-only /v1/internal/retention/preview route alongside pruneExpiredRecords.

This addresses the dedup/retention-job requirement of #3810, which the issue itself calls "the actual root-cause fix." The remaining #3810 asks (a scheduled Cloudflare D1 size probe via the Management API, new gittensory_d1_* gauge metrics, and new Prometheus alert/Grafana panel rules) are deliberately not included here: there are no Cloudflare Management API credentials/integration anywhere in this codebase today, and the self-host Prometheus/Grafana stack has no path to scrape a metric computed inside the Cloudflare Worker that owns D1 — wiring that up is a materially different, higher-risk architectural change (new secret handling, a new cross-deployment metrics bridge) that deserves its own scoped PR rather than being bundled with a clean, narrow, fully-tested fix. Left a comment on #3810 with this breakdown.

Scope

• The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
• This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
• This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
• I linked an issue, or this is small enough that the summary explains why an issue is not needed.

Validation

• git diff --check
• npm run actionlint
• npm run typecheck
• npm run test:coverage locally; codecov/patch requires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.
• npm run test:workers
• npm run build:mcp
• npm run test:mcp-pack
• npm run ui:openapi:check
• npm run ui:lint
• npm run ui:typecheck
• npm run ui:build
• npm audit --audit-level=moderate
• New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

All run via npm run test:ci (full local gate, green) plus a standalone npm audit --audit-level=moderate (0 vulnerabilities). Verified every new/changed line and branch in src/db/retention.ts, src/queue/processors.ts, and src/api/routes.ts is covered locally (v8 lcov branch report), including the defensive ?? 0 fallback arms via hand-mocked env.DB results (no meta / no row), mirroring the existing pattern in test/unit/alerts.test.ts.

Safety

• No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
• Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
• Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests.
• API/OpenAPI/MCP behavior is updated and tested where needed.
• UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks.
• Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails.
• Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

Backend-only data-retention change: no auth/CORS/session surface, no UI, no docs/changelog impact.

UI Evidence

N/A — backend-only change, no visible UI surface.

Notes

• Deferred from obs(selfhost): monitor Cloudflare D1 database size and add bounded dedup retention for signal_snapshots #3810 (tracked there, not in this PR): scheduled D1 size probe via Cloudflare Management API, gittensory_d1_database_size_bytes/gittensory_d1_table_row_count gauges, and the corresponding Prometheus alert rule + Grafana panel.

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.48%. Comparing base (a873a46) to head (5df3b64).
⚠️ Report is 7 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3857   +/-   ##
=======================================
  Coverage   93.48%   93.48%           
=======================================
  Files         333      333           
  Lines       33162    33188   +26     
  Branches    12128    12136    +8     
=======================================
+ Hits        31001    31027   +26     
  Misses       1530     1530           
  Partials      631      631           

Files with missing lines	Coverage Δ
src/api/routes.ts	94.94% <100.00%> (+<0.01%)	⬆️
src/db/retention.ts	96.36% <100.00%> (+2.07%)	⬆️
src/queue/processors.ts	94.52% <100.00%> (+<0.01%)	⬆️

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[loopover-orb]

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

_{Review updated: 2026-07-06 20:49:11 UTC}

4 files · 1 AI reviewer · 2 blockers · readiness 93/100 · CI green · clean

⏸️ Suggested Action - Manual Review

• No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
• Maintainer requires a linked issue — Link the relevant issue (for example Closes #123) before opening the PR.

Review summary
Adds dedupeSignalSnapshots to src/db/retention.ts, keeping only the highest-rowid row per (signal_type, target_key), batched per signal_type with a LIMIT-based delete loop capped by maxPerType, and wires it into both the daily prune-retention job and the read-only retention preview route. The SQL logic (NOT IN (SELECT MAX(rowid) ... GROUP BY target_key)) is correct, the batch/cap loop terminates deterministically, and the new tests exercise the dry-run count path, the multi-batch/cap-stopping path, and both defensive `?? 0` fallback arms with mocked envs. The PR explicitly scopes out the remaining #3810 asks (D1 size probe, metrics/alerts) with a stated rationale, and does not touch the generate-signal-snapshots writer itself — it mitigates via a bounded retention pass rather than preventing the duplicate inserts at the source, which the PR description says the issue itself endorses as the intended root-cause fix.

Nits — 5 non-blocking

• The batch delete's staleCondition re-runs the full `MAX(rowid) ... GROUP BY target_key` aggregate over the signal_type's rows on every iteration of the loop in src/db/retention.ts dedupeSignalSnapshots, which is more read work per batch than strictly necessary for large duplicate counts, though it's bounded by maxPerType so not unbounded.
• src/db/retention.ts interpolates `batchSize` directly into the SQL string (`LIMIT ${batchSize}`) instead of binding it as a parameter; harmless today since no caller passes a user-controlled batchSize, but worth a comment or a bound param for defense-in-depth.
• The external brief's 'magic numbers' flags (3810, 342, 243, 183) are just incident/issue numbers embedded in the doc comment at src/db/retention.ts, not real numeric literals needing a named constant — safe to ignore.
• The long-file flags on src/api/routes.ts and src/queue/processors.ts reflect pre-existing file size, not something this diff meaningfully grows — not an actionable concern here.
• Consider whether generate-signal-snapshots should upsert on (signal_type, target_key) instead of always inserting, which would eliminate the duplicate-accumulation problem at the source rather than relying on a periodic cleanup pass — worth a one-line note in the PR or issue if that's intentionally deferred.

Concerns raised — review before merging

• No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
• Maintainer requires a linked issue — Link the relevant issue (for example Closes #123) before opening the PR.

Signal	Result	Evidence
Code review	❌ 2 blockers	1 reviewer
Linked issue	⚠️ Missing	No linked issue or no-issue rationale found.
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (no linked issue context).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 51 registered-repo PR(s), 43 merged, 372 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 51 PR(s), 372 issue(s).
Gate result	❌ Blocking	Repo-configured hard blocker found.

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: Python, TypeScript, JavaScript, Ruby, Go, Kotlin, MDX, Shell
• Official Gittensor activity: 51 PR(s), 372 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Treat this as maintainer-lane context rather than normal contributor-lane activity.
• Explain no-issue PR.
• Link the issue being solved, or explicitly explain why this is a no-issue PR.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

• Re-run Gittensory review