Context
Found while grounding the accuracy metric (#8820, #8822). The gate_decision row recorded for a PR can contradict the action the bot actually took, because the verdict is written from the quality-gate evaluation while the close may be driven by a later, separate signal (CI failure, policy).
Concrete timeline — #5861:
| time |
event |
value |
| 20:23:31 |
agent.action.close |
"CI is failing (validate, validate-tests (5))" |
| 20:23:42 |
pr_outcome |
closed |
| 20:23:44 |
gate_decision |
merge / success ← written 13s AFTER the close |
The bot closed the PR, then recorded a verdict of merge. Fleet calibration reads the latest gate_decision as the gate's prediction, so this PR is scored as "the gate said merge and it ended up closed" — a false positive that never happened. The gate decided close, and the close was correct.
Scope, measured on the live self-host across the merge-verdict-but-closed-outcome class (n=210):
- 84 were closed by the bot itself
- 59 have their
gate_decision timestamped after the close action — definitively stale/contradictory
- the remaining 126 were closed by a human despite a
merge verdict (genuine disagreements — a separate, real signal worth studying, not an instrumentation artifact)
This biases published accuracy downward, the opposite direction from the reversal under-counting in #8820/#8823. Between the two, no accuracy figure is currently trustworthy in either direction — which is the core problem for the provable-accuracy/TEE work.
Requirements
- Make the recorded
gate_decision reflect the decision the bot actually acted on. A terminal close driven by CI failure or policy must not leave a merge verdict as the latest recorded prediction.
- Fix the ordering so the verdict is never written after the terminal action it is supposed to explain.
- Decide and document how policy closes (contributor cap, duplicate, evasion, blacklist) should be scored: they are deliberate decisions, not quality mispredictions, and conflating them with gate verdicts distorts precision in both directions. Either record them with a distinct verdict/reason class or exclude them from precision with an explicit, documented rationale.
- Backfill or explicitly quarantine the 59 known-contradictory rows so historical calibration is not silently wrong.
- Invariant test: the latest
gate_decision for a PR never contradicts a completed terminal action on that PR.
- Regression test for the exact ordering above (close, then verdict write).
Test Coverage Requirements
99%+ patch coverage, branch-counted, on all changed src/** lines — both arms of every new conditional.
Expected Outcome
Every recorded gate verdict matches the action the bot took, policy closes are scored as their own class rather than as quality mispredictions, and the published decision accuracy is trustworthy in both directions.
Context
Found while grounding the accuracy metric (#8820, #8822). The
gate_decisionrow recorded for a PR can contradict the action the bot actually took, because the verdict is written from the quality-gate evaluation while the close may be driven by a later, separate signal (CI failure, policy).Concrete timeline — #5861:
agent.action.closepr_outcomeclosedgate_decisionmerge/ success ← written 13s AFTER the closeThe bot closed the PR, then recorded a verdict of
merge. Fleet calibration reads the latestgate_decisionas the gate's prediction, so this PR is scored as "the gate said merge and it ended up closed" — a false positive that never happened. The gate decided close, and the close was correct.Scope, measured on the live self-host across the
merge-verdict-but-closed-outcome class (n=210):gate_decisiontimestamped after the close action — definitively stale/contradictorymergeverdict (genuine disagreements — a separate, real signal worth studying, not an instrumentation artifact)This biases published accuracy downward, the opposite direction from the reversal under-counting in #8820/#8823. Between the two, no accuracy figure is currently trustworthy in either direction — which is the core problem for the provable-accuracy/TEE work.
Requirements
gate_decisionreflect the decision the bot actually acted on. A terminal close driven by CI failure or policy must not leave amergeverdict as the latest recorded prediction.gate_decisionfor a PR never contradicts a completed terminal action on that PR.Test Coverage Requirements
99%+ patch coverage, branch-counted, on all changed
src/**lines — both arms of every new conditional.Expected Outcome
Every recorded gate verdict matches the action the bot took, policy closes are scored as their own class rather than as quality mispredictions, and the published decision accuracy is trustworthy in both directions.