fix(retention): dedup the contributor-intelligence snapshot types — 2026-07 recurrence of #3810 - #8240
Merged
Merged
Conversation
…026-07 recurrence of #3810 signal_snapshots reached D1's absolute 10GB cap again, 61 days into this database's life: the contributor-intelligence writers (the scoring pass in processors.ts) append one ~36KB row per contributor per pass for contributor-evidence-graph / contributor-outcome-history / contributor-strategy — ~6GB in three weeks at current review volume, refilling the cap before the 90-day age window could ever engage. Live writes were one page-split from failing; the phase-2 label apply already did. No reader consumes these types as a series (the canonical latest lives in the dedicated contributor_evidence / contributor_scoring_profiles upsert tables; nothing calls listSignalSnapshots for contributor-*), so the existing latest-only dedup job now covers them — lossless for every actual consumer. contributor-decision-pack stays excluded as the bounded trend series the retention doc records it to be. Production remediation (the incident playbook's chunked per-type manual dedup) executed alongside.
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8240 +/- ##
==========================================
- Coverage 92.11% 91.58% -0.53%
==========================================
Files 777 777
Lines 78364 78364
Branches 23677 23676 -1
==========================================
- Hits 72183 71768 -415
- Misses 5062 5524 +462
+ Partials 1119 1072 -47
Flags with carried forward coverage won't be shown. Click here to find out more.
|
loopover-orb
Bot
added
the
gittensor:bug
…Y_SIGNAL_SNAPSHOT_TYPES The pin hardcoded four placeholders; the list just grew to seven (the #3810 recurrence fix) and will grow again — derive it so the guard can never silently desync from the probe's real SQL.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Production incident, found mid-#8170: the cloud D1 hit its absolute 10 GB maximum —
signal_snapshotsalone held 6 GB, generated in the last ~3 weeks.Root cause
The contributor-intelligence writers (the scoring pass) append one ~36 KB row per contributor per pass for
contributor-evidence-graph/contributor-outcome-history/contributor-strategy. At current review volume that's ~8k rows/day — the table refills D1's cap faster than the 90-day retention window can ever engage (only 299 rows were old enough to prune). This is a recurrence of #3810 with new offender types that postdate that incident's latest-only list.Fix
LATEST_ONLY_SIGNAL_SNAPSHOT_TYPES, so the existing scheduled dedup keeps exactly one row per contributor. Lossless for every actual consumer: no reader consumes these as a series — the canonical latest lives in the dedicatedcontributor_evidence/contributor_scoring_profilesupsert tables, and nothing callslistSignalSnapshotsfor any contributor-* type.contributor-decision-packstays excluded as the documented bounded trend series.Validation
Retention suite 19/19 green; root typecheck green. The changed function's behavior for previously-covered types is pinned by the existing tests.
Advances #3810 (recurrence)