scripts/run-plugin-tests.sh exits 0 while silently skipping the only Python suite — a green run is not evidence

[kyle-sexton]

Problem

scripts/run-plugin-tests.sh exits 0 while silently not running the session-flow retro Python suite. A green run is therefore not evidence that a change to parse_transcript.py or test_parse_transcript.py is safe.

Reproduced this session:

$ bash plugins/session-flow/skills/retro/scripts/parse-transcript.test.sh
SKIP: pytest not installed for python3

The runner's own header documents the behavior as intentional:

Each test is self-contained and cwd-independent; an individual test SKIPs (exit 0) when an optional tool it needs (shellcheck, shfmt, ...) is absent, so this runner gates on real failures without requiring every tool to be present.

That trade is defensible for a linter like shfmt. It is a different thing for the only test coverage a Python module has: the skip converts "I have no coverage right now" into "everything passed", and the difference is invisible in the runner's output unless you read every line.

Why it nearly caused a wrong claim

While preparing PR #1296 (which removes a field from the handoff frontmatter and from that suite's chain fixtures), the runner returned exit 0 and I was one step from reporting the change as test-verified. It was not — nothing had run. The 30 passes eventually reported came from invoking pytest directly:

$ uvx --with pytest pytest test_parse_transcript.py -q
30 passed

An agent or a human reading only the exit code would have shipped an unverified change with a green checkmark next to it.

Note on the existing gate

The repo already ships scripts/check-silent-skips.sh, and it passes here — but its scope is hook entry scripts ("No silent prerequisite skips found in hook entry scripts"). Plugin test harnesses are not covered. The concept is already endorsed in this repo; it just has not been extended to this surface.

Candidate directions (not a decision)

• Make the skip loud and counted — a summary line naming which suites skipped and why, so exit 0 with skips is visibly different from exit 0 with all-green
• Treat a skipped suite as a failure in CI while remaining a skip locally, since CI has the toolchain and a skip there means the environment regressed
• Declare each harness's prerequisites so a missing one is an explicit environment error rather than a per-run judgment
• Extend check-silent-skips.sh beyond hook entry scripts to test harnesses

Relations

• Surfaced during PR refactor(session-flow): restructure the handoff save-point taxonomy #1296 (session-flow handoff taxonomy restructure)
• scripts/check-silent-skips.sh — same concept, currently scoped to hook entry scripts

[kyle-sexton]

This was generated by AI (triage lane — gate-audit claim). Claiming this item for one gate-audit cycle: an independent panel re-classifies whether the needs-human / status: needs-decision gate is still warranted, or whether the item is C2/C3 and liftable to agent-ready. Outcome comment follows in this cycle.

[kyle-sexton]

This was generated by AI (triage lane — gate audit). Independent panel: two fresh-context agents (one fable, one opus), each given only the issue content and the repo's own plugins/autonomy/reference/guardrails/work-classes.md, plus the triage lane's own read. This lane never executes work items, authors fixes, or merges.

Gate audit: GATE UPHELD — four candidate directions, explicitly "not a decision"

Panel: 2-of-2 STAYS-GATED, both C3. Lane read concurs.

The defect is not in dispute and is well evidenced: parse-transcript.test.sh prints SKIP: pytest not installed for python3 while run-plugin-tests.sh exits 0, and 30 tests only ran under a direct uvx --with pytest invocation. A green run is genuinely not evidence.

What holds the gate is that the issue carries a section literally titled "Candidate directions (not a decision)" listing four materially different policies — loud counted skips, skip-as-CI-failure, declared harness prerequisites, extending check-silent-skips.sh. Picking among them is setting the repo's test-skip policy, and the fourth option would itself be C4. An agent choosing here would be inventing CI policy, not implementing it.

Decision required

Should a skipped test suite be a hard CI failure, a loud counted skip, or both?

RECOMMENDED — both, sequenced. Make the skip loud and counted in run-plugin-tests.sh now, so an exit-0-with-skips run is visibly different from all-green; and simultaneously fail in CI on any skipped suite, because CI has the toolchain and a skip there means the environment regressed rather than that the developer lacks a tool. This preserves local developer ergonomics while making the CI signal honest, and it matches the repo's existing anti-silent-skip posture (check-silent-skips.sh, #391). Both panel legs reached this pairing independently.

Alternative — loud-and-counted only, leaving CI tolerant of skips. Cheaper, no CI-policy change, but it preserves exactly the "green run is not evidence" hazard this issue was filed about.

Also considered — extend check-silent-skips.sh to cover test harnesses as a static gate without changing runner semantics; that is a C4 contract change and should be a separate decision if wanted.

[kyle-sexton]

This was generated by AI (triage lane — claim release). Gate-audit cycle complete for this item; the triage claim posted earlier this cycle is released. No lease is held — another instance or lane may pick this up immediately. The outcome comment above this one carries the classification and either the lift rationale or the decision summary.

[kyle-sexton]

This was generated by AI in the decide-and-implement lane (operator authorization: human gates become decision work for frontier consensus). Decision: the triage panel's recommended pairing, adopted with one honest narrowing. The skip becomes loud and counted per suite — the runner summary now names every suite that emitted a SKIP and its first reason, and the final line distinguishes a clean pass from a pass with skipped coverage — and a new --strict-skips flag turns any optional skip into a failure for callers that declare a full toolchain. The narrowing: CI is not wired to --strict-skips in this change, because a live inventory of skip reasons shows CI's toolchain is missing far more than pytest (goimports, typos, PSScriptAnalyzer, shfmt — the last deliberately optional per ci.yml's own comment), so blanket strict enforcement is a toolchain-completion project, not this fix. What does land for CI is the concrete hazard this issue was filed about: pytest and its dependency closure are now hash-pinned into .github/requirements-ci.txt, so the session-flow parse_transcript suite (the only Python coverage) actually runs in CI — verified locally, 37 passed — instead of skipping green. Extending check-silent-skips.sh to test harnesses remains the separate C4 contract change the panel set aside.