feat(api): weekly rule-calibration trend endpoint over fired/override history and backtest runs - #8156
Merged
Merged
Conversation
… history and backtest runs (#8113) The #8101/#8104 fired+override history and the #8138/#8139 persisted backtest runs had no aggregate view -- answering 'is precision for rule X trending up or down' meant manually re-running CLIs. Adds rule-calibration-trend.ts (the maintainer-facing sibling of public-accuracy-trend.ts: same live no-cron re-bucketing posture, same null-below-min-sample discipline) aggregating per-rule fired/confirmed/reversed/precisionPct per UTC week plus backtest-run verdict counts across both sibling event types, served at /v1/internal/calibration-trend next to the existing operator calibration endpoint (same INTERNAL_JOB_TOKEN gate). Trend-grained precision semantics (decision-week bucketing, confirmed/decided) are documented as deliberately distinct from computeRulePrecision's per-target pairing. Aggregate counts and rule ids only; the leak-guard test pins that no metadata raw context escapes. 100% line+branch coverage on the service.
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8156 +/- ##
==========================================
+ Coverage 89.54% 90.15% +0.60%
==========================================
Files 97 99 +2
Lines 22622 25244 +2622
Branches 3866 4715 +849
==========================================
+ Hits 20257 22758 +2501
- Misses 2187 2214 +27
- Partials 178 272 +94
Flags with carried forward coverage won't be shown. Click here to find out more.
|
This was referenced Jul 23, 2026
Closed
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
src/services/rule-calibration-trend.tsis the maintainer-facing sibling ofpublic-accuracy-trend.ts— the same deliberate live-no-cron re-bucketing posture (audit_events is durable; every request recomputes any historical week correctly, no rollup copy to drift), the same null-below-min-sample discipline (a week with <3 decided verdicts reports null, never a fake 0% or 100%).fired / confirmed / reversed / precisionPct(precision = confirmed/decided, bucketed by the decision week — deliberately trend-grained and documented as distinct fromcomputeRulePrecision's per-target corpus pairing), plus weekly backtest-run verdict counts across both sibling event types (calibration.threshold_backtest_run+calibration.logic_backtest_run; a run with a missing/unrecognized verdict counts asunchangedrather than vanishing).GET /v1/internal/calibration-trend, directly beside the existing/v1/internal/calibrationoperator endpoint, behind the sameINTERNAL_JOB_TOKENgate via the/v1/internal/*middleware. Aggregate counts and rule ids only — the leak-guard test pins that captured raw-context metadata (diffs, issue text, model responses) can never escape through this surface. Deliberately NOT in the public stats payload, and (matching the/v1/internal/decision/calibrationprecedent) not in the OpenAPI spec (ui:openapiregen confirmed a no-op).Closes #8113
Its Boundaries said "don't scope until real data exists" — picked up tonight on the maintainer's explicit call to complete the epic end-to-end; the aggregation is defined entirely over already-shipped, fixed data shapes, and the empty-corpus state renders honestly (fresh-DB test included).
Scope
type(scope): short summaryConventional Commit format, for examplefix(api): restore profile access checks.CONTRIBUTING.mdand does not reintroduce GitHub Pages, VitePress,site/, orCNAME.Closes #123) — a linked open issue is required for every contributor PR.Validation
git diff --checknpm run actionlintnpm run typechecknpm run test:coveragelocally;codecov/patchrequires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.npm run test:workersnpm run build:mcpnpm run test:mcp-packnpm run ui:openapi:checknpm run ui:lintnpm run ui:typechecknpm run ui:testnpm run ui:buildnpm audit --audit-level=moderateIf any required check was skipped, explain why:
npm run test:ciran green through every step exceptui:version-audit, which failed on npm's mid-session 3.5.0 publish (fixed on main in chore(ui): sync MCP_PACKAGE_KNOWN_LATEST_VERSION with npm dist-tags.latest 3.5.0 #8155; this branch is rebased onto it and the audit re-ran green). Every step after the audit in the chain (docs/branding/manifest/engine-parity/nvmrc/release-manifest/command-referencedrift checks +ui:lint/ui:typecheck/ui:test/ui:build) ran green individually. The new service has 100% line AND branch coverage (11 tests: pure bucketing, D1-backed loader, route auth + leak guard).Safety
UI Evidencesection below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.Negative-path: the 401-without-token test covers the auth gate; the leak-guard test seeds a fired event carrying raw diff metadata and asserts none of it reaches the response.
UI Evidence
Not applicable — no UI change (operator API endpoint only).
Notes