fix(selfhost): alert explicitly when the monitored D1 approaches its 10 GB cap

[JSONbored]

Problem

#9435: the D1 that backs the hosted deployment filled to its 10 GB cap twice (2026-07-06, 2026-07-26). At the cap every write fails — including the webhook relay's INSERT — so inbound webhook delivery stops fleet-wide. Both times the first signal was the outage itself.

The size probe (#3810, d1-size-probe.ts) already samples file_size via the Cloudflare Management API — but only into a /metrics gauge. With no scrape/dashboard stack watching that endpoint, the reading existed and alerted no one: the same config-dependent-observability-ships-inert failure mode #9433 tracks.

Change

checkD1SizeThreshold, fed by each fresh probe sample:

• 70% (warn) — ~3 GB headroom, time to widen retention or plan a migration
• 85% (critical) — headroom is weeks at the measured fleet write rate

On crossing a level: one structured console.error (event: "d1_size_threshold"), a loopover_d1_size_threshold_alerts_total{level} counter tick, and a PostHog exception capture (d1_size_threshold) — surfaces the operator already watches, no metrics stack required.

Latched per level with hysteresis: an alert fires on crossing, never per tick; dropping back below logs an info-grade recovery line and re-arms the latch; a later re-crossing fires again. A failed size fetch (the probe's carried-forward sample) neither fires nor recovers the latch.

Tests

All six behaviors pinned: silent below warn, single fire + capture at warn, warn→critical escalation without intra-level re-fire, recovery + re-arm, critical→warn downgrade without alert, and the probe-integration case (fresh reading alerts; failed fetch doesn't). d1-size-probe.ts is at 100% statements and branches.

Closes #9435

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[loopover-orb]

Warning

⏸️ LoopOver review result - manual review recommended

_{Review updated: 2026-07-27 20:22:38 UTC}

3 files · 1 AI reviewer · no blockers · CI green · clean

⏸️ Suggested Action - Manual Review

Review summary
This PR adds a latched, hysteresis-based alert (console.error + counter + PostHog capture) that fires when a fresh D1 size sample crosses 70%/85% of the 10GB cap, addressing #9435's fleet-wide outage. The logic is correctly traced: `runD1SizeProbe` only calls `checkD1SizeThreshold` when `freshInfo` is truthy, so a failed/carried-forward sample never re-alerts or recovers the latch, and the latch/rank comparison correctly distinguishes crossing (fire), staying within a level (silent), and dropping (recovery log, re-arm). Tests cover all six stated behaviors and directly assert on the real `runD1SizeProbe` integration path rather than fabricating state, which is solid coverage for this size.

Nits — 6 non-blocking

• The two `console.error`/`console.log` calls in `checkD1SizeThreshold` (d1-size-probe.ts:214,227) are intentional structured alerting output per the PR's stated design, not leftover debug statements — worth confirming this is the team's accepted convention for op-facing alerts (matches `logD1ProbeError`'s existing pattern) rather than flagging as debug residue.
• `d1SizeAlertLevelFor` and the threshold constants use ratio-based comparisons; consider a brief inline example of the byte math (e.g., 0.7 * 10GB ≈ 7.37GB reported as 70%, not literally '3 GB headroom' as the comment states, since 30% of 10GB is 3GB but only from the cap itself) — the comment is directionally correct but slightly loose.
• The module-level `lastAlertLevel` mutable singleton mirrors the existing `lastSample` pattern in this file, but if `runD1SizeProbe` is ever called concurrently for multiple databases in the future, this single latch would conflate them — not an issue today since only one D1 database is monitored per process.
• Consider extracting the repeated `errorSpy.mock.calls.filter(...).length` idiom in the test file into a small helper if more probe/threshold interaction tests are added later.
• The PostHog capture message and the console.error JSON both encode `percentOfCap`/`fileSizeBytes` — verify downstream PostHog dashboards can parse the free-text `message` if `context` fields aren't surfaced directly.
• PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.

Decision drivers

• ✅ Code review — No blockers (1 reviewer)
• ✅ Gate result — Passing (No configured blocker found.)

Context & advisory signals — never blocks the verdict

Signal	Result	Evidence
Linked issue	✅ Linked	#9435
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 13 registered-repo PR(s), 13 merged, 313 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 13 PR(s), 313 issue(s).
Improvement	✅ Minor	risk: clean · value: minor

Linked issue satisfaction

Addressed
The PR wires the previously-inert D1 size probe into an explicit alerting path (structured console.error, a counter, and a PostHog capture) at the 70%/85% thresholds the issue proposed, exactly the 'no alerting on D1 size' gap called out; tests cover latching, hysteresis, and probe-integration behavior.

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: Python, TypeScript, Ruby, Go, MDX, Shell, Solidity, JavaScript
• Official Gittensor activity: 13 PR(s), 313 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Start here: Treat this as maintainer-lane context rather than normal contributor-lane activity.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

• @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
• @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
• A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

Decision record

• action: hold · clause: success
• config: 2fce10c7a1bdf5d851453f44abcdb87ee50e3a25446fa95118e4d5ef1db34ca3 · pack: oss-anti-slop · ci: failed
• record: 8e43435019f659dbc6a6f01b57fb263bffa5db6e0629c050633072e67203bdae (schema v5, head 5d2e466)

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

• Re-run LoopOver review

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.62%. Comparing base (fb8e2eb) to head (e473116).
⚠️ Report is 8 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9458      +/-   ##
==========================================
+ Coverage   89.54%   93.62%   +4.07%     
==========================================
  Files         843      741     -102     
  Lines      110011    60373   -49638     
  Branches    26184    21291    -4893     
==========================================
- Hits        98511    56522   -41989     
+ Misses      10238     2900    -7338     
+ Partials     1262      951     -311     

Flag	Coverage Δ
backend	93.62% <100.00%> (-1.64%)	⬇️
control-plane	?
engine	?
rees	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/selfhost/d1-size-probe.ts	100.00% <100.00%> (ø)
src/selfhost/metrics.ts	100.00% <ø> (ø)

... and 275 files with indirect coverage changes