Verification is documented as non-negotiable and enforced by nothing: three PRs merged tonight with no verdict on the artifact, and one shipped a regression

[kyle-sexton]

Three PRs merged tonight — #2262, #2263, #2312 — each with zero verification evidence on the artifact. In one case that directly shipped a regression which needed a second PR to undo.

This is not a proposal to add process for its own sake. It is a request to decide whether verification should be a gate or remain an aspiration, because right now it is the latter and the difference just cost a release.

What happened, precisely

time	event
02:07:19Z	#2262 merged
02:10:54Z	Codex posts a P1 — 3 minutes after the merge
02:47Z	do-not-merge applied to an already-merged PR; inert
03:06Z	regression confirmed on main, filed as #2310
03:31Z	#2312 merges the fix

The P1 was neither missed nor overruled. Review ran and was correct; independent verification ran and was sound. Neither gated anything, because the merge did not wait for either. The verifier was explicit that its APPROVE covered only its assigned items and that remote resolution "was outside my item set and I did not assess it" — so nothing contradicted Codex; the finding simply arrived too late to matter.

The same pattern repeated on #2263 and #2312. In one case a verifier produced a full verdict that was never posted, and a separate lane reported "no independent review could be obtained" while three completed verdicts existed elsewhere. A verdict that lives only in an agent transcript is not evidence and dies with the session.

Why the existing mechanisms did not catch it

Each of these works as designed, and none of them is a verification gate:

• Draft status — the strongest signal available, and the merge wrapper honours it (PR is a draft -- mark ready first). But the author un-drafts themselves, so it gates nothing an author does not want gated.
• do-not-merge label — honoured by the wrapper, but requires a human or agent to be watching at the right moment. Applied 33 minutes late here, to two already-merged PRs.
• babysit_merge.py — enforces owner allowlist, draft, mergeable, mergeStateStatus, review decision, unresolved threads, failing/pending checks, unmet required contexts, merge queue, moved head, dependency author, unprotected base. It has no notion of verification evidence. (It also computes requireSignatures and never enforces it — source-control:babysit-prs: babysit_merge computes requireSignatures and never enforces it — a merge held by an unsigned or mis-authored head reports four causes, none of them the real one #2265, same shape one layer down.)
• Required thread resolution — the ruleset requires it, but the author resolves their own threads.

The proposal, and the trap it must avoid

Make the guarded merge wrapper refuse a merge when the PR carries no verification evidence — a marker comment in an agreed machine-readable form.

The identity constraint is the whole design, not a detail. This repo already learned this the hard way in #2163's D7 fix: a SHA appearing in a comment body proves the SHA was mentioned, not that you posted it — a reviewer quoting your fix satisfies the check while your own write silently failed. The fix was to pin .user.login. A verification marker has exactly the same failure mode: if the author can post their own marker, the gate is decorative. So the check must require a marker from a login other than the PR author, and that constraint should be code, not convention.

Sketch, deliberately loose on format because the shape is the decision:

merge refused unless: a comment exists on the PR
  whose body carries the agreed verdict marker
  AND whose .user.login != the PR author
  AND (optionally) which references the current head SHA

The head-SHA clause is worth debating separately: it makes a verdict stale on every push, which is correct in principle and may be intolerable in practice on a busy lane.

What this does NOT solve, stated plainly

• A presence check cannot judge quality. It gates on "a verdict exists", not "the verdict is any good", and that invites rubber-stamping. This is a real cost, not a quibble — the honest claim is that it raises the floor from nothing to someone independent looked, which is exactly the gap that shipped source-control: D6's reachability gate hardcodes origin, regressing the skill's non-origin and triangular-fork support #2310.
• It adds a failure mode: when verifiers stall — which happened repeatedly tonight — the gate blocks legitimate work. It needs a documented, visible override rather than an easy one, and overrides need to be rare enough to notice.
• It does not help trivial PRs, and forcing verification on a typo fix will train people to route around it. Some scope rule (touched paths, diff size, plugin vs docs) probably belongs in the design.

The decision being asked for

Not "should we add this check" — that is downstream. The question is: should verification evidence be a merge precondition at all, or is the current posture (verification encouraged, merge ungated) the deliberate choice?

Either answer is defensible. What is not defensible is the current in-between, where the process documents verification as non-negotiable and nothing enforces it, so whether it happens depends on whether an agent remembers to publish before it merges. Tonight it did not, three times, and the one time it mattered a regression reached main.

Related: #2265 (a rule computed and never enforced), #2310 (the regression this let through), #2244 (the original defect class — a check that answers confidently about the wrong thing).

[kyle-sexton]

Evidence against my own proposal: verifier dispatch failed ~50% of the time tonight

I filed this asking whether verification should gate merges. Having now spent a full batch trying to obtain verdicts, I owe the counter-evidence, because it is the strongest argument against what I proposed.

The numbers

Verifiers I asked directly, by name, for their verdict verbatim:

responded	never responded
verify-2262	verify-2312
verifier-2263	verify-2316
branch-verifier	verify-2316b
verifier2	verify-2341
codex-audit	verify-2352

5 of 10. The five that answered did so on their first idle after the request, and returned excellent work — enumerated items, controls in both directions, explicit scope limits, and in two cases findings that changed decisions. The five that did not have each since idled two to four times without processing a queued message. Repeated nudges changed nothing.

Other lanes report the same rate independently:

• The parity lane dispatched two fresh-context verifiers with enumerated contracts; neither returned. It shipped a labelled non-independent evidence summary instead.
• The residue lane reported five dispatch attempts returning nothing, including one that carried the exact command inline. It concluded "no independent review could be obtained" — which was itself wrong, because three verdicts had gone to the spawning session instead. That is a second failure mode: verdicts that exist but are routed somewhere nobody looks.

Why this cuts against gating

I wrote in the original body that a gate "adds a failure mode: when verifiers stall, the gate blocks legitimate work" and that it needs an override rare enough to notice. That was speculative. It is now measured, and the number is bad enough to change the shape of the answer.

A merge gate whose precondition fails half the time is not a gate, it is a coin flip with extra steps. Every one of tonight's seven merges would have been blocked pending a verdict that, on these odds, arrives about half the time. The override would stop being exceptional within a day, and an override used routinely is indistinguishable from no gate at all — except that it also trains everyone to reach for it.

What I think this means — revised

The problem decomposes into two, and only one of them is about gating:

1. Verdicts that exist but never reach the artifact. This is a plumbing failure, it is cheap to fix, and it is most of the loss. Two verdicts tonight existed and died in transcripts; a lane declared a verification gap while three completed verdicts sat elsewhere. A convention that a verdict is posted to the PR — not returned to a parent — would have recovered those without gating anything.
2. Dispatches that produce nothing at all. No policy fixes this. Gating on it converts an unreliable subsystem into an unreliable blocker.

So the honest recommendation, revised from what I filed: fix the routing before considering the gate. If posting verdicts to artifacts becomes reliable and the observed dispatch-success rate rises materially above tonight's 50%, the gating question is worth reopening with real numbers behind it. Gating first would import a coin-flip dependency into the merge path.

I am leaving the original proposal above intact rather than editing it, so the reasoning that produced it stays visible alongside the evidence that undercut it.

The one thing worth keeping regardless

The identity constraint. If any verification marker is ever introduced — gate or not — it must require a poster other than the PR author, in code. This repo already learned that in #2163's D7 fix: a SHA in a comment body proves the SHA was mentioned, not that you posted it. A marker the author can satisfy themselves is decorative whether or not anything gates on it.

[kyle-sexton]

Drain note: fail-closed security-review evidence gate landed in #2430; duration false-pass regression addressed in open PR #2481 (cursor/security-evidence-duration-fix-63e1). Leaving open until #2481 merges and the lane re-validates.

[kyle-sexton]

This was generated by AI (decide lane, operator-authorized frontier decision pass — human gates decided, not skipped).

Decide brief — verification is now a gate; the hold condition this issue was parked on is met

Decision: close as completed. The question this issue posed — gate or aspiration — was answered gate, and the mechanism has landed:

• #2430 (merged 2026-08-12T13:15Z) added the fail-closed security-review evidence gate — a merge can no longer proceed with zero verification evidence on the artifact, which is exactly the failure shape the fix(source-control): D6 verifies commit presence, not the branch tip; per_page=100 on every paginated read #2262/fix(review,work-items,claude-ops): correct the gh --paginate array-merge claim and conform per_page=100 #2263/fix(source-control): D6 resolves the push remote instead of hardcoding origin #2312 night demonstrated.
• #2481 (merged 2026-08-12T16:20Z) fixed the sub-45s duration false-pass regression in that gate. The drain note above held this issue open "until fix(ci): stop treating sub-45s security reviews as false passes #2481 merges" — that condition is now satisfied.

What is deliberately not carried by this close: the verifier-dispatch reliability finding (5-of-10 named verifiers never responded). That is an operational weakness of the verification supply, not of the gate — the fail-closed gate turns a missing verdict into a blocked merge rather than a silent pass, which is the correct failure direction. If non-response rates keep blocking merges in practice, that evidence should drive its own tracked item at that point; it does not justify holding this one open.