convention: false-green checks — surfaces report success while the capability is dead or findings invisible (liveness-assertion umbrella)

[kyle-sexton]

Category: convention + CI meta-check (umbrella) · Source: control-tower gap-hunt 2026-07-19 (signal mining)

Pattern — checks report green while the capability is dead or its findings are invisible

Cross-plugin recurrence with no umbrella:

• disk-hygiene: setup:check reports healthy while the engine is dead (misses #376 env-injection and bare-python failures) #385 — disk-hygiene setup:check reports healthy while the engine is dead (env-injection failure, cluster disk-hygiene: skill-frontmatter guard reads CLAUDE_PLUGIN_DATA from env but never receives it -> engine lane fails closed on all platforms #376/disk-hygiene: guard hook launches via bare python — fails to run (fail-open risk) on python3-only hosts #380/disk-hygiene: disk_hygiene_enabled kill switch does not gate the PowerShell lane and is inert under the env-injection failure #382/disk-hygiene: doc corrections — Windows mislabeled 'full', false CLAUDE_PLUGIN_DATA premise repeated, PowerShell-lane gaps #386).
• ci/agents: zizmor advisory findings are check-run annotations — invisible to every agent surface (green check, unread annotations) #510 — zizmor advisory lane exits green; real security findings live only in check-run annotations no agent reads.
• source-control:babysit-prs: babysit-readiness-gate.sh over-counts lifetime P-badges as 'findings', false READINESS_BLOCKED reason=under-decomposed on fully-classified PRs #465 — babysit readiness gate returns READINESS_BLOCKED/green on miscounted findings.
• source-control:babysit-prs: snapshot classifies an Approve-with-nits claude[bot] review as blocking bot feedback (gate correctly returns findings=0) #499 — gate passes on a misread approval state.

Root cause: checks assert "the check ran," not "the capability is live and its output is surfaced." By construction this class is review-agent-blind: green check = nobody looks.

Proposed fix

1. Liveness-assertion convention (documented, then enforced): a health/status/advisory surface must either fail loud or surface its findings into an agent-readable channel — green-with-hidden-findings and healthy-while-dead are contract violations. Self-test probes for engines (does the capability demonstrably execute, not just configure).
2. CI meta-check where feasible: advisory lanes must publish findings to a read channel (e.g. SARIF/code-scanning per ci/agents: zizmor advisory findings are check-run annotations — invisible to every agent surface (green check, unread annotations) #510's options) — annotations-only is non-conforming.

Relations

#510 (zizmor instance + its fix options), #385/#376 cluster (engine liveness instance), #465/#499 (gate-classifier instances — mechanical fix tracked separately in the babysit-classifier umbrella), #509 (a security-review lane is only as good as the signals surfaced to it).

[kyle-sexton]

This was generated by AI during triage. · Category: convention + CI meta-check (umbrella) · Priority: priority: high (systemic false-green class — by construction review-agent-blind: green check = nobody looks; includes security-signal loss #510). Routed needs-human + wayfind: design.

Why needs-human

Defining a liveness-assertion convention (a health/status/advisory surface must fail loud OR surface findings into an agent-readable channel — green-with-hidden-findings and healthy-while-dead are contract violations) is a new cross-plugin convention the maintainer must author, then decide how/where to enforce (documented convention + CI meta-check). That is product-convention design, not a defaultable fix.

The two design pieces

1. Liveness-assertion convention (document, then enforce): self-test probes that prove a capability demonstrably executes, not just configures; advisory surfaces must route findings to a read channel. Define what "conforming" means per surface type (engine health-check vs advisory lane vs gate).
2. CI meta-check where feasible: advisory lanes must publish findings to an agent-readable channel (e.g. SARIF/code-scanning per ci/agents: zizmor advisory findings are check-run annotations — invisible to every agent surface (green check, unread annotations) #510's options) — annotations-only is non-conforming.

Instances (the pattern this umbrella covers — cross-link, not block)

• ci/agents: zizmor advisory findings are check-run annotations — invisible to every agent surface (green check, unread annotations) #510 — zizmor advisory exits green; findings live in check-run annotations no agent reads (routed ready this session — the immediate visibility fix; this umbrella is the general convention behind it).
• disk-hygiene: setup:check reports healthy while the engine is dead (misses #376 env-injection and bare-python failures) #385 / disk-hygiene: skill-frontmatter guard reads CLAUDE_PLUGIN_DATA from env but never receives it -> engine lane fails closed on all platforms #376 cluster — disk-hygiene setup:check healthy while the engine is dead.
• source-control:babysit-prs: babysit-readiness-gate.sh over-counts lifetime P-badges as 'findings', false READINESS_BLOCKED reason=under-decomposed on fully-classified PRs #465 / source-control:babysit-prs: snapshot classifies an Approve-with-nits claude[bot] review as blocking bot feedback (gate correctly returns findings=0) #499 — gate/classifier misreads (mechanical fixes tracked in the babysit-classifier umbrella source-control:babysit-prs: one shared authorship/finding/approval classifier + golden fixtures; gate script owns the merge-ready boolean (6-issue class) #534).
• autonomy pipeline: no dedicated security-review pass — automated code review only #509 — a security-review lane is only as good as the signals surfaced to it.

Decompose candidate. Recommend: author the convention first (it's the contract), then the CI meta-check + per-surface self-tests decompose from it.

[kyle-sexton]

Decision brief (2-minute read)

Problem. A recurring cross-plugin class has checks that report green while the capability behind them is dead or its findings are invisible — a produced verdict that is wrong, which is review-agent-blind by construction because a green check means nobody looks. There is no convention that says a health, status, or advisory surface must either fail loud or route its findings to a channel an agent reads.

Today. The doctrine already exists in prose but only for one slice: docs/PLUGIN-PHILOSOPHY.md "Prerequisites and failure behavior" (lines 199–218) states "No black boxes: a silently skipped feature is a defect" and "Do not swallow errors or claim success when the promised result was not produced." One mechanical enforcement of that slice already ships: ci.yml's silent-skip-gate lane (lines 234–245, scripts/check-silent-skips.sh) fails hook entry scripts that quietly skip on a missing CLI, with a # silent-skip-ok: annotated-exemption escape hatch. But this covers only prerequisite-visibility in plugins/*/hooks/*.sh, not health-check, gate, or advisory surfaces. The gap is visible in this repo's own CI: the zizmor lane (lines 172–179) runs advisory — "findings annotate without blocking" — so real security findings live in check-run annotations no agent reads (the #510 instance). Instances elsewhere: disk-hygiene setup:check reports healthy while the engine is dead (#385/#376), and gate classifiers return green on misread state (#465/#499). A conforming counter-example already exists to model on: the hygiene lane runs a --self-test on its own result aggregator (ci.yml line 148) before trusting it.

Options.

• A — RECOMMENDED: convention-first, then enforce (staged). Author the liveness-assertion convention as the contract — a health/status/advisory surface must fail loud OR surface findings into an agent-readable channel; "green-with-hidden-findings" and "healthy-while-dead" are contract violations — defining "conforming" per surface type (engine health-check needs a self-test probe that proves it executes not just configures; advisory lane must publish to a read channel such as SARIF/code-scanning per ci/agents: zizmor advisory findings are check-run annotations — invisible to every agent surface (green check, unread annotations) #510; gate must fail-closed). Then decompose enforcement: a CI meta-check where mechanical, per-surface self-tests where not. Tradeoff: the convention is the load-bearing artifact and settling it first lets enforcement decompose cleanly, but full mechanical coverage arrives later. Future-state risk: low — this generalizes a pattern already proven in-repo (silent-skip-gate + the aggregator self-test), so the enforcement path is de-risked. Reversibility: high — convention is a doc; each self-test/meta-check is independently removable. Rationale: matches the maintainer's stated lean in triage ("author the convention first — it's the contract"), and there is existing doctrine and mechanism to extend rather than invent.
• B — meta-check now, convention implicit. Ship a CI meta-check that forces advisory lanes to publish to a read channel (fixes ci/agents: zizmor advisory findings are check-run annotations — invisible to every agent surface (green check, unread annotations) #510-shape immediately) before writing the general convention. Tradeoff: fast visible win, but codifies enforcement for one surface type while the cross-surface contract stays undefined, risking a narrow rule that later fights the general one. Reversibility: medium.
• C — convention-only, no enforcement. Document the liveness-assertion rule and rely on review. Tradeoff: zero build cost, but this class is review-agent-blind by construction — the thing that would catch it is exactly what is green and unread. Weakest fit; the doctrine already half-exists and instances still shipped.

Open design question (do not pre-decide): placement + per-surface conformance. What "conforming" means for each surface type (engine vs advisory lane vs gate), and whether the CI meta-check lives as a ci-workflows composite action or a repo-local script like silent-skip-gate, are design calls per the org cross-repo ownership rule (reusable Actions execution → ci-workflows, wiring → this repo).

Related.

• ci/agents: zizmor advisory findings are check-run annotations — invisible to every agent surface (green check, unread annotations) #510 — zizmor advisory findings invisible in annotations; the immediate visibility instance and its SARIF/code-scanning fix options.
• disk-hygiene: setup:check reports healthy while the engine is dead (misses #376 env-injection and bare-python failures) #385/disk-hygiene: skill-frontmatter guard reads CLAUDE_PLUGIN_DATA from env but never receives it -> engine lane fails closed on all platforms #376 — disk-hygiene setup:check healthy while engine dead; the runs-and-lies (dead-engine) instance. Belongs here, not quality-net: skill changes ship with zero regression detection — 120 evals.json unexecuted, skill-quality:check uninvoked (staged eval/gate/pass^k umbrella) #530: the check executes and returns a wrong verdict.
• source-control:babysit-prs: babysit-readiness-gate.sh over-counts lifetime P-badges as 'findings', false READINESS_BLOCKED reason=under-decomposed on fully-classified PRs #465/source-control:babysit-prs: snapshot classifies an Approve-with-nits claude[bot] review as blocking bot feedback (gate correctly returns findings=0) #499 — gate/classifier green on misread state; mechanical fixes tracked in the babysit-classifier umbrella source-control:babysit-prs: one shared authorship/finding/approval classifier + golden fixtures; gate script owns the merge-ready boolean (6-issue class) #534, the convention here.
• autonomy pipeline: no dedicated security-review pass — automated code review only #509 — a security-review lane is only as good as the signals surfaced to it.
• quality-net: skill changes ship with zero regression detection — 120 evals.json unexecuted, skill-quality:check uninvoked (staged eval/gate/pass^k umbrella) #530 — sibling umbrella; owns the adjacent-but-distinct class. Discriminator: quality-net: skill changes ship with zero regression detection — 120 evals.json unexecuted, skill-quality:check uninvoked (staged eval/gate/pass^k umbrella) #530 = signal absent (evals/gates never wired or invoked, no verdict produced); convention: false-green checks — surfaces report success while the capability is dead or findings invisible (liveness-assertion umbrella) #532 = signal false (check runs, reports green, and lies). Cross-link, do not merge.
• ci: portability lint lane — enforce declared agnosticism mechanically (19-issue coupling class + top review-churn source) #531 — sibling umbrella on a different axis (coupling/agnosticism, not liveness). Cross-link only.

[kyle-sexton]

Operator decision (2026-07-19, decision-session batch 3; relayed).

ACCEPTED — the decision brief's RECOMMENDED option verbatim.

Re-intake: removing needs-human, adding priority: needs-triage for routing.

[kyle-sexton]

This was generated by AI during triage.

Routing status: ready per ratified decision. Operator ACCEPTED Option A verbatim: author the liveness-assertion convention first — a health/status/advisory surface must fail loud OR surface findings into an agent-readable channel; "green-with-hidden-findings" and "healthy-while-dead" are contract violations — then decompose enforcement (CI meta-check where mechanical, per-surface self-tests where not). Per-surface conformance definitions and placement (ci-workflows composite vs repo-local) stay open implementation calls. Discriminator vs sibling umbrellas: #530 = signal absent, #531 = coupling axis, #532 = signal false (this one — check runs green and lies).

Priority: not in the explicit scaling-chain list (rule 8b) — downgraded from priority: high to priority: medium. wayfind: design removed (convention-first sequencing settled).