source-control:babysit-prs: an unproven readiness verdict is enforced only by prose, not by the gate predicate

[kyle-sexton]

Problem

When the host permission classifier denies the readiness gate, nothing mechanical stops an orchestrator from reporting a PR as merge-ready anyway. The guard is prose in skills/babysit-prs/reference/loop.md:484-488 (a NEVER-do bullet) and :536-541 (§5.5 "Gate verdict, quoted verbatim") — instructions to an agent, not a predicate that refuses.

The plugin's own philosophy, stated in scripts/babysit-readiness-gate.sh, is the standard this fails:

Prose "MANDATORY" is a word; this gate counts rows, not intentions.

Why the existing READINESS_UNPROVEN verdict does not close it

PR #1264 added an unproven() helper (scripts/babysit-readiness-gate.sh:109-112) that prints READINESS_UNPROVEN reason=<...> pr=<n>. That is a real improvement to the gate's own output hygiene and should land on its merits. It is not enforcement:

• Exit codes are unchanged (3 and 4), which that PR's CHANGELOG states outright: "Exit codes are unchanged, so existing callers keyed on them are unaffected." No caller's behavior changes.
• No executable consumer parses the token. git grep -n "READINESS_" across plugins/source-control, excluding the gate and its own tests, returns only CHANGELOG, reference/review-discipline.md, reference/loop.md, reference/safety.md, skills/babysit-prs/evals/evals.json, two Python source comments (babysit_classify.py, babysit_findings.py), and test files. The only consumer is an LLM told to quote it.
• The branch concedes the gap in writing (reference/safety.md:383-385): "a gate the harness never let run cannot report its own non-invocation, which is why the quoted-verdict requirement lives on the report rather than inside the script."

Scope note — #787's gate is a different script

#787's denied invocation was the merge-readiness gate:

python .../babysit_merge.py <pr> --allowed-owners ... --self-logins @me

babysit_merge.py still emits a JSON object with exit codes 0/10/2/3 and has no state distinguishing "readiness could not be proven" from "the command never ran". Any mechanical fix has to reach it, not only scripts/babysit-readiness-gate.sh.

The setup canary does not detect the condition either

skills/setup/SKILL.md probes reachability with --help invocations only. A classifier decision is per-call, so a host that permits --help and denies the production argument shapes (owner/repo#N --allowed-owners ...) yields a PASS from a lane that cannot prove readiness. Raised independently by the Codex reviewer on #1264 (skills/setup/SKILL.md:146, :147).

What would actually close it

Not prescribing a design, but the shape has to satisfy: an orchestrator that never obtained a passing gate verdict cannot emit a merge-ready verdict, and the refusal is made by code rather than by an instruction. Candidates worth weighing:

• A readiness assertion the orchestrator must present, produced only by a real gate run and checked mechanically before a merge-ready line is accepted — moving the trust anchor from the report to an artifact.
• Extending the readiness-gate predicate so that consuming machinery (rather than an LLM) reads the verdict and refuses on READINESS_UNPROVEN or on absence.
• Giving babysit_merge.py the same unproven state and a non-zero exit distinct from "not ready", so a caller keyed on exit codes can tell the two apart.

Whichever shape wins, a canary that exercises the production argument shapes rather than --help is part of it, or the lane still cannot tell a permitted probe from a permitted job.

Related

• source-control:babysit-prs: read-only babysit_merge.py invocation blocked by host auto-mode permission classifier #787 — the originating report. Its causal premise was falsified separately (its reproduction used a raw wildcarded-interpreter form that auto mode drops by design), but the prerequisite it names survives on other evidence.
• fix(source-control): emit a readiness verdict on the gate's silent failure paths #1264 — carries the verdict-hygiene work and its tests. Re-scoped to that, and no longer claims to close this.
• Guard semantics the lane's consumers restate in prose have no drift detection #1265 / feat(source-control): bind babysit guard semantics to an executable contract #1285 — the adjacent principle that a published claim must be bound to an executable assertion.

[kyle-sexton]

This was generated by AI during triage.

Triage result: verified, human-gated (open design space)

Verification — every load-bearing claim checked against code, not prose:

• The referenced unproven() helper and READINESS_UNPROVEN verdict do not exist on main — they exist only on the open, unmerged PR fix(source-control): emit a readiness verdict on the gate's silent failure paths #1264 (branch feat/babysit-automode-prereq). Confirmed by reading plugins/source-control/scripts/babysit-readiness-gate.sh on that branch: unproven() prints READINESS_UNPROVEN reason=%s pr=%s and is called from every non-READINESS_OK/READINESS_BLOCKED exit path, but the function's own exit codes (3, 4) are unchanged from before — matching the CHANGELOG line quoted in the issue ("Exit codes are unchanged, so existing callers keyed on them are unaffected").
• git grep -n "READINESS_" on that branch, across plugins/source-control excluding the gate script and its own test, confirms the issue's enumeration: only CHANGELOG.md, reference/review-discipline.md, skills/babysit-prs/reference/{loop,safety}.md, skills/babysit-prs/evals/evals.json, two Python source comments (babysit_classify.py, babysit_findings.py), and test files reference the token. No executable consumer branches on it.
• reference/loop.md §5.4/§5.5 and reference/safety.md "Lane-Script Reachability" quote verbatim as cited: the "Never report a readiness verdict the gate did not emit" rule and "a gate the harness never let run cannot report its own non-invocation, which is why the quoted-verdict requirement lives on the report rather than inside the script" are both prose instructions to the orchestrating LLM, not code that refuses.
• babysit_merge.py (the script source-control:babysit-prs: read-only babysit_merge.py invocation blocked by host auto-mode permission classifier #787 actually blocks on) confirmed to have no unproven/READINESS_* state and no exit code distinct from its existing 0/10/2/3 set — the issue's "scope note" holds.
• The setup canary section (skills/setup/SKILL.md) was updated on the PR branch today, after this issue was filed, to state the --help-only probe limitation in prose rather than close it mechanically (commit "state the canary's pass-side limit rather than over-claiming (source-control:babysit-prs: read-only babysit_merge.py invocation blocked by host auto-mode permission classifier #787)") — this is a documentation correction of an over-claim, not new enforcement, and does not change the triage verdict.

Conclusion: the issue's technical claims hold. The gap is real, and it lives in in-flight, unmerged work (PR #1264), not shipped main behavior — which is why this routes as a design-space decision rather than a "fix broken behavior in production" bug.

Agent Brief

Type: Task
Summary: The readiness-verdict discipline for babysit-prs (quoting READINESS_* verbatim, treating READINESS_UNPROVEN/silence as unproven) is enforced by instructions to the orchestrating agent, not by a predicate any executable consumer checks. Nothing currently refuses to proceed when an unproven or absent verdict is reported.

Current behavior:
babysit-readiness-gate.sh (on PR #1264) emits a READINESS_UNPROVEN line on its own failure paths, but no other script reads that token. babysit_merge.py has no state distinguishing "readiness could not be proven" from "the readiness check was never invoked." The loop's iteration-report contract (quote the verdict verbatim) is a documentation/prose rule that a non-compliant agent run can simply not follow, with no mechanical block.

Desired behavior:
An orchestrator that never obtained a passing gate verdict must be structurally prevented from emitting (or acting on) a merge-ready verdict — the refusal must be enforced by code that a caller cannot bypass by omission, not by an instruction the agent is trusted to obey.

Key interfaces (candidates raised in the issue, not prescribed):

• A readiness-assertion artifact that the orchestrator must produce and that downstream machinery verifies was generated by an actual gate run (moves the trust anchor off the free-text report).
• The readiness-gate predicate extended so a non-LLM consumer parses READINESS_* and refuses on READINESS_UNPROVEN or on the token's absence.
• babysit_merge.py gains the same unproven state, with a non-zero exit code distinct from "not ready," so a caller keyed on exit codes can tell the two apart.
• Whichever direction is chosen, the setup reachability canary needs to probe production argument shapes (not just --help) or a permitted probe remains indistinguishable from a permitted job.

Acceptance criteria:

• An orchestrator run in which the readiness gate never executes (harness-denied, or the call omitted) cannot result in a "merge-ready" report or merge action — verified by a mechanism other than the report author following instructions.
• The decision on which enforcement shape to build (readiness-assertion artifact vs. predicate-consumer vs. babysit_merge.py exit-code state) is made and recorded before implementation starts.
• babysit_merge.py and babysit-readiness-gate.sh agree on how "unproven" is represented, so a caller cannot get a false ready signal by going through one script instead of the other.

Out of scope:

• Re-litigating PR fix(source-control): emit a readiness verdict on the gate's silent failure paths #1264's verdict-hygiene work itself (READINESS_UNPROVEN stdout token) — that is a real, separate improvement on its own merits and should land independently.
• source-control:babysit-prs: read-only babysit_merge.py invocation blocked by host auto-mode permission classifier #787's original host-classifier-denial reproduction (already re-scoped away from this issue).
• Redesigning the classification/finding-count logic (R1/R5/R6) — unrelated to verdict enforcement.

Why human-gated

This is a genuinely open design-space decision, not a single-fork default: the issue itself declines to prescribe among three architecturally different candidates (assertion artifact, predicate-consumer extension, babysit_merge.py exit-code parity), each with different blast radius across babysit-readiness-gate.sh, babysit_merge.py, and the loop/safety docs, and touches an in-flight PR (#1264) plus adjacent open items (#787, #1265/#1285's "published claim bound to an executable assertion" principle). Labeled wayfind: design to match this repo's existing convention for open design-space items (see #897, #1086, #465, #604, #350, #375).

Labels applied: Type=Task, priority: medium, needs-human, wayfind: design; priority: needs-triage cleared.

[kyle-sexton]

This was generated by AI during triage.

Triage — raw marker retired, routed human-gated

Verified against actual code at HEAD, not the report's wording. Every load-bearing claim holds:

• No executable consumer parses the token. git grep -n "READINESS_" -- plugins/source-control, excluding the gate and its tests, returns only CHANGELOG.md, reference/review-discipline.md, reference/loop.md, reference/safety.md, SKILL.md, evals/evals.json, and two Python comments (babysit_classify.py:373, babysit_findings.py:108). Confirmed: the sole consumer is an LLM instructed to quote it.
• babysit_merge.py has no unproven state. grep -c unproven returns 0 over the 44 KB file, and its exits are exactly the 0 / 10 / 2 / 3 set the report names. There is genuinely no value distinguishing "readiness could not be proven" from "the command never ran" — so a caller keyed on exit codes cannot tell a denied invocation from a not-ready verdict.
• The unproven() helper is not at HEAD. It arrives with PR fix(source-control): emit a readiness verdict on the gate's silent failure paths #1264, still open. The report frames it correctly as in-flight and as verdict-hygiene rather than enforcement; nothing here is blocked on that PR, and fix(source-control): emit a readiness verdict on the gate's silent failure paths #1264 should land on its own merits.

One correction for future readers: the gate script is at the plugin-relative path scripts/babysit-readiness-gate.sh, not under skills/babysit-prs/scripts/ — the prose citations in reference/loop.md are the ones under the skill.

Classification: work-class: structural (C4). Per the work-classes contract, C4 covers contract changes with cross-cutting blast radius and hard reversal — "human review and human merge are mandatory, always". All three candidate shapes in the body are contract changes: a new assertion artifact the orchestrator must present, a change to what consuming machinery does with the gate verdict, or a new exit code on babysit_merge.py whose current codes are documented as a stable caller interface. Any of them also drags in the setup canary, since a --help-only probe cannot distinguish a permitted probe from a permitted production invocation.

Priority: medium. This is a real enforcement gap on the merge-authority boundary, but present exposure is low: every merge in this repository to date has been performed by a human, so no autonomous merge has ever ridden on an unproven verdict. It is a latent hole in a lane that is being actively promoted toward autonomy — which is why it should not sit at low either.

Routing: needs-human, not decision-defaulted. The body explicitly declines to prescribe a design ("Not prescribing a design, but the shape has to satisfy...") and offers three materially different candidates with different blast radii. That is open design space over an enforcement boundary, not a single reversible fork — fails closed.

Related, unchanged: #787 (originating report; causal premise separately falsified, prerequisite survives), #1264 (verdict hygiene, re-scoped), #1265 / #1285 (adjacent principle: a published claim must bind to an executable assertion).

[kyle-sexton]

Verified addressed on origin/main by the #787 readiness-contract bundle.

Evidence: babysit-readiness-gate.sh emits READINESS_UNPROVEN on every non-OK path; loop.md §5.5 requires a per-PR Gate verdict line (quoted stdout or explicit harness-denied command); test_skill_contract.py::test_unproven_readiness_contract_is_stated_on_both_sides pins the contract. A verdict-less readiness claim is no longer admissible in the skill contract — the prose-only gap this issue filed against is closed.