obs(selfhost): add Cloudflare D1 size/row-count probe, metrics, alerts, panel

[JSONbored]

Summary

• obs(selfhost): monitor Cloudflare D1 database size and add bounded dedup retention for signal_snapshots #3810's "actual root-cause fix" (the write-side dedup job, dedupeSignalSnapshots) already shipped and merged in fix(db): dedupe signal_snapshots to bounded latest-per-key retention #3857. This PR covers the remaining, deliberately-deferred half the issue and fix(db): dedupe signal_snapshots to bounded latest-per-key retention #3857 both called out: observability for the Cloudflare D1 storage cap that the 2026-07-06 incident hit (~10GB, 342,243 signal_snapshots rows for 2,183 keys).
• Adds an opt-in Cloudflare Management API probe (src/selfhost/d1-size-probe.ts), gated on three new CLOUDFLARE_D1_MONITOR_{ACCOUNT_ID,DATABASE_ID,API_TOKEN} env vars (presence-gated, same convention as isOrbBrokerMode's ORB_ENROLLMENT_SECRET check) — absent on almost every self-host install, which runs its own SQLite/Postgres backend and has no real Cloudflare D1 to watch. Wired into the self-host process's own boot-time timer (mirrors the existing Orb relay registration retry timer in server.ts), polling every 15 minutes.
• Publishes three new gauges + a counter via the existing src/selfhost/metrics.ts registry:
  • gittensory_d1_database_size_bytes — the monitored database's file size.
  • gittensory_d1_table_row_count{table=...} — row count per RETENTION_POLICY table (single source of truth with the age-based retention policy, src/db/retention.ts).
  • gittensory_signal_snapshots_rows_per_key — rows-per-distinct-(signal_type, target_key), scoped to exactly the four latest-only-dedup signal types dedupeSignalSnapshots converges to ~1 row per key (not the whole table, which intentionally keeps bounded multi-row history for other signal types like queue-health). A climbing ratio means the daily dedup job stopped running or its allowlist regressed.
  • gittensory_d1_probe_errors_total{part=...} — the probe's own failure counter, so an operator can tell "the probe broke" apart from "the database stopped growing".
• Size and each table's row count are fetched independently; a failure in one never blanks the other or resets a previously-good reading (verified in tests) — a transient Cloudflare API hiccup reads as "stale", not a false "suddenly zero" or an absent series.
• Four Prometheus alerts (prometheus/rules/alerts.yml, new gittensory-d1-storage group): D1 size warn (~70% of the ~10GB cap) / critical (~90%), the signal_snapshots dedup-regression ratio (warn >10, vs. the incident's actual ~157 ratio), and probe-failure.
• A new "Cloudflare D1 (Central Cloud, obs(selfhost): monitor Cloudflare D1 database size and add bounded dedup retention for signal_snapshots #3810)" row on the self-host Grafana dashboard (grafana/dashboards/gittensory.json — the general self-host observability dashboard the issue itself points at; deliberately not maintainer-reviews.json, which sibling PRs in this batch are touching).

Scope

• The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
• This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
• This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
• I linked a currently open issue this PR resolves (e.g. Closes #123) — a linked open issue is required for every contributor PR.

Validation

• git diff --check
• npm run actionlint
• npm run typecheck
• npm run test:coverage locally; codecov/patch requires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.
• npm run test:workers
• npm run build:mcp
• npm run test:mcp-pack
• npm run ui:openapi:check
• npm run ui:lint
• npm run ui:typecheck
• npm run ui:build
• npm audit --audit-level=moderate
• New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

src/selfhost/d1-size-probe.ts is 100% covered (statements/branches/functions/lines) both in isolation and inside the full npm run test:coverage run (26 dedicated tests in test/unit/selfhost-d1-size-probe.test.ts, covering: config resolution with each of the 3 env vars individually missing; the Cloudflare API client's success/HTTP-error/success:false/malformed-JSON paths; the unsafe-table-identifier guard; signal_snapshots' numbered-placeholder dedup query and its distinct-key ratio, including the division-by-zero guard; and runD1SizeProbe's independent size-vs-table-count failure isolation, including "keeps the previous sample on a later failure" and "one bad table doesn't block the others"). Also ran the full local gate: npm run selfhost:validate-observability, npm run selfhost:env-reference:check, npm run cf-typegen:check, npm run db:migrations:check, npm run db:schema-drift:check, and the existing test/unit/selfhost-grafana-dashboard.test.ts / test/unit/worker-entry-boundary.test.ts / test/unit/observability-ci.test.ts / test/unit/docs-selfhost-troubleshooting-metric-names.test.ts suites — all green, none needed changes since this PR is purely additive (new panels/alerts/metric names, no existing ones touched).

If any required check was skipped, explain why:

• None skipped.

Safety

• No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
• Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
• Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests.
• API/OpenAPI/MCP behavior is updated and tested where needed. — N/A, no API/OpenAPI/MCP surface touched (ui:openapi:check confirms no drift).
• UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks. — N/A, no UI change; apps/gittensory-ui/src/lib/selfhost-env-reference.ts is mechanically regenerated (npm run selfhost:env-reference) to add 3 rows in the same format as the other 96, not hand-authored UI.
• Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots. — N/A, no visible UI change (see above).
• Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs. — Deliberately not touching docs.self-hosting-troubleshooting.tsx in this PR (keeping scope to the code + observability-config surface the issue asks for); test/unit/docs-selfhost-troubleshooting-metric-names.test.ts only checks that names mentioned in that doc are real, so it doesn't require adding a new section, and CHANGELOG.md is never edited in a normal PR per house rules.

UI Evidence

N/A — no visible UI surface (backend/observability-only change; see Safety above).

Notes

• dedupeSignalSnapshots (the actual root-cause fix, fix(db): dedupe signal_snapshots to bounded latest-per-key retention #3857) is unaffected by this PR — this is purely additive observability layered on top of it.
• The probe is wired into the self-host process's own timer/gauge registrations in server.ts rather than the Cloudflare Worker's scheduled() cron: server.ts is Codecov-exempt (real reason documented in codecov.yml — "exercised by the Docker build+boot smoke test, not unit-coverable without booting a server/subprocess") and is the one place that already runs a persistent process capable of carrying an in-memory sample from a periodic probe through to a later /metrics scrape; the ephemeral, multi-isolate Cloudflare Worker request lifecycle can't reliably do that, and (today) has no /metrics route at all.
• Left the Cloudflare API credentials as env vars (not .gittensory.yml), matching the existing convention that raw secrets/credentials (GitHub App keys, ORB_ENROLLMENT_SECRET, etc.) stay env-only while policy/tunables live in config-as-code.

Closes #3810

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	gittensory-ui	c5e062c	Commit Preview URL Branch Preview URL	Jul 08 2026, 08:03 AM

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.73%. Comparing base (97f087b) to head (c5e062c).
⚠️ Report is 5 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4153      +/-   ##
==========================================
+ Coverage   93.72%   93.73%   +0.01%     
==========================================
  Files         385      386       +1     
  Lines       36243    36303      +60     
  Branches    13281    13296      +15     
==========================================
+ Hits        33968    34028      +60     
  Misses       1618     1618              
  Partials      657      657              

Files with missing lines	Coverage Δ
src/db/retention.ts	96.49% <100.00%> (ø)
src/selfhost/d1-size-probe.ts	100.00% <100.00%> (ø)
src/selfhost/metrics.ts	100.00% <ø> (ø)

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[loopover-orb]

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

_{Review updated: 2026-07-08 08:16:14 UTC}

9 files · 1 AI reviewer · no blockers · readiness 100/100 · CI green · clean

⏸️ Suggested Action - Manual Review

• Touches a guarded path — held for manual review: This PR changes guardrail-protected path(s): src/selfhost/d1-size-probe.ts (matched src/selfhost/**), src/selfhost/metrics.ts (matched src/selfhost/**).

Review summary
This adds a fully opt-in Cloudflare D1 size/row-count observability probe (src/selfhost/d1-size-probe.ts) plus metrics, Grafana panels, and Prometheus alerts, wired into self-host's own boot-time timer in src/server.ts. It correctly closes out the deferred observability half of #3810 (the write-side dedup fix already shipped in #3857), is gated on presence of all three CLOUDFLARE_D1_MONITOR_* env vars so it's a byte-identical no-op for the vast majority of self-host installs, and the accompanying test suite directly exercises the partial-failure/stale-value-retention behavior the PR claims (independent per-fetch isolation, no reset to -1 on a later failure). No schema change occurs (LATEST_ONLY_SIGNAL_SNAPSHOT_TYPES is only exported, not altered), so no migration is needed, and the feature is config-driven rather than hardcoded to any specific repo/owner.

Nits — 7 non-blocking

• src/selfhost/d1-size-probe.ts: the numbered-placeholder reuse in the signal_snapshots query (`?1..?4` referenced twice, bound once) relies on Cloudflare D1's HTTP query endpoint supporting SQLite-style positional parameter reuse the same way dedupeSignalSnapshots does in src/db/retention.ts — worth a one-line comment confirming this was verified against a live D1 database, not just unit-mocked.
• src/server.ts:~1027: the 900_000ms probe interval isn't cleared on shutdown (no clearInterval paired with the setInterval); likely harmless given the process-lifetime nature of the timer but worth confirming it matches the Orb relay timer's own lifecycle handling it claims to mirror.
• grafana/dashboards/gittensory.json / prometheus/rules/alerts.yml: thresholds (7GB/9GB warning/critical, ratio>10) are reasonable given the incident numbers cited, but consider pulling the ~10GB D1 cap into a single named constant referenced from both the alert file and dashboard description so the two don't drift if Cloudflare changes the cap.
• Consider a brief follow-up note in the PR description or a code comment cross-referencing exactly where dedupeSignalSnapshots' own `?1` reuse is proven safe against the real D1 HTTP API (vs. just local better-sqlite3), since this new probe's SQL leans on that same assumption for a second query.
• d1-size-probe.ts's `lastSample` is a bare module-level singleton — fine for a single self-host process, but if this pattern gets reused for any per-request or multi-tenant path later, it would need to move off module state.
• PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.
• Touches a guarded path — held for manual review — A maintainer must review and merge this change.

Signal	Result	Evidence
Code review	✅ No blockers	1 reviewer
Linked issue	✅ Linked	#3810
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 52 registered-repo PR(s), 43 merged, 496 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 52 PR(s), 496 issue(s).
Gate result	⚠️ Not blocking	Advisory; not blocking this PR.

Linked issue satisfaction

Partially addressed
The PR adds the observability half of the issue (D1 size/row-count probe, three metrics, alerts, and a Grafana panel), directly satisfying the monitoring/alerting/dashboard requirements, but it explicitly does not include the dedup/retention job itself, instead pointing to a separate already-merged PR (#3857) for that root-cause fix.

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: Python, TypeScript, JavaScript, Ruby, Go, Kotlin, MDX, Shell
• Official Gittensor activity: 52 PR(s), 496 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Treat this as maintainer-lane context rather than normal contributor-lane activity.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

• Re-run Gittensory review

[JSONbored]

Re: the "Linked issue satisfaction: Partially addressed" note above (advisory, not a gate blocker) — the dedup/retention job it says is "not present in this diff" is correct about this diff, but that half of #3810 already shipped and is live on main, verified rigorously rather than just asserted:

• fix(db): dedupe signal_snapshots to bounded latest-per-key retention #3857 ("fix(db): dedupe signal_snapshots to bounded latest-per-key retention") merged 2026-07-06T20:51:34Z (merge commit 071b0b8b), adding dedupeSignalSnapshots to src/db/retention.ts, wired into the daily prune-retention job (runRetentionPrune, src/queue/processors.ts) and the /v1/internal/retention/preview route (src/api/routes.ts).
• It's present right now in this exact branch (rebased on today's origin/main) at src/db/retention.ts:122-158 — batched per signal_type, keeping only the highest-rowid row per (signal_type, target_key).
• It's been actively maintained since: fix(db): limit signal snapshot dedupe to cache signals #3971 ("limit signal snapshot dedupe to cache signals", merged 2026-07-07) and fix(db): dedupe focus manifest snapshots #4015 ("dedupe focus manifest snapshots", merged 2026-07-07) both refined it further.
• The issue's own second comment (posted by me, 2026-07-06T19:22:48Z, before this PR existed) already documents the split: "Advances obs(selfhost): monitor Cloudflare D1 database size and add bounded dedup retention for signal_snapshots #3810 via fix(db): dedupe signal_snapshots to bounded latest-per-key retention #3857, which ships the dedup/retention job — the issue's own 'actual root-cause fix' ... Not included in that PR, still open here: [the D1 size probe, gauges, and Prometheus/Grafana rules]" — which is exactly what this PR (obs(selfhost): add Cloudflare D1 size/row-count probe, metrics, alerts, panel #4153) delivers.

So #3857 + #4153 together fully resolve #3810's requirement list; this PR was never meant to reintroduce the dedup job itself. Leaving this open for the separate verification pass rather than merging.