Skip to content

feat(api): weekly rule-calibration trend endpoint over fired/override history and backtest runs - #8156

Merged
JSONbored merged 1 commit into
mainfrom
feat/calibration-trend
Jul 23, 2026
Merged

feat(api): weekly rule-calibration trend endpoint over fired/override history and backtest runs#8156
JSONbored merged 1 commit into
mainfrom
feat/calibration-trend

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

Closes #8113

Its Boundaries said "don't scope until real data exists" — picked up tonight on the maintainer's explicit call to complete the epic end-to-end; the aggregation is defined entirely over already-shipped, fixed data shapes, and the empty-corpus state renders honestly (fresh-DB test included).

Scope

  • The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • I linked a currently open issue this PR resolves (e.g. Closes #123) — a linked open issue is required for every contributor PR.

Validation

  • git diff --check
  • npm run actionlint
  • npm run typecheck
  • npm run test:coverage locally; codecov/patch requires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.
  • npm run test:workers
  • npm run build:mcp
  • npm run test:mcp-pack
  • npm run ui:openapi:check
  • npm run ui:lint
  • npm run ui:typecheck
  • npm run ui:test
  • npm run ui:build
  • npm audit --audit-level=moderate
  • New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

If any required check was skipped, explain why:

  • npm run test:ci ran green through every step except ui:version-audit, which failed on npm's mid-session 3.5.0 publish (fixed on main in chore(ui): sync MCP_PACKAGE_KNOWN_LATEST_VERSION with npm dist-tags.latest 3.5.0 #8155; this branch is rebased onto it and the audit re-ran green). Every step after the audit in the chain (docs/branding/manifest/engine-parity/nvmrc/release-manifest/command-reference drift checks + ui:lint/ui:typecheck/ui:test/ui:build) ran green individually. The new service has 100% line AND branch coverage (11 tests: pure bucketing, D1-backed loader, route auth + leak guard).

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests.
  • API/OpenAPI/MCP behavior is updated and tested where needed.
  • UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks.
  • Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

Negative-path: the 401-without-token test covers the auth gate; the leak-guard test seeds a fired event carrying raw diff metadata and asserts none of it reaches the response.

UI Evidence

Not applicable — no UI change (operator API endpoint only).

Notes

  • MCP exposure for this read (per the MCP-first convention) is a natural follow-up once the dashboard consumer shape settles — the endpoint is deliberately dashboard-shape-agnostic JSON.

… history and backtest runs (#8113)

The #8101/#8104 fired+override history and the #8138/#8139 persisted backtest
runs had no aggregate view -- answering 'is precision for rule X trending up
or down' meant manually re-running CLIs. Adds rule-calibration-trend.ts (the
maintainer-facing sibling of public-accuracy-trend.ts: same live no-cron
re-bucketing posture, same null-below-min-sample discipline) aggregating
per-rule fired/confirmed/reversed/precisionPct per UTC week plus backtest-run
verdict counts across both sibling event types, served at
/v1/internal/calibration-trend next to the existing operator calibration
endpoint (same INTERNAL_JOB_TOKEN gate). Trend-grained precision semantics
(decision-week bucketing, confirmed/decided) are documented as deliberately
distinct from computeRulePrecision's per-target pairing. Aggregate counts and
rule ids only; the leak-guard test pins that no metadata raw context escapes.
100% line+branch coverage on the service.
@JSONbored JSONbored self-assigned this Jul 23, 2026
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.15%. Comparing base (ac2190f) to head (5cc403c).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8156      +/-   ##
==========================================
+ Coverage   89.54%   90.15%   +0.60%     
==========================================
  Files          97       99       +2     
  Lines       22622    25244    +2622     
  Branches     3866     4715     +849     
==========================================
+ Hits        20257    22758    +2501     
- Misses       2187     2214      +27     
- Partials      178      272      +94     
Flag Coverage Δ
shard-1 29.44% <11.66%> (?)
shard-2 44.62% <100.00%> (?)
shard-3 69.37% <11.66%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/api/routes.ts 95.27% <100.00%> (ø)
src/services/rule-calibration-trend.ts 100.00% <100.00%> (ø)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feed rule/AI-judgment calibration data into an accuracy trend view

1 participant