fix(verification,testing): stop routing live-app verification through the user-invoked-only /verify

plugins/testing/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "testing",
-  "version": "0.3.2",
+  "version": "0.3.3",
   "description": "Test-stage discipline across all ecosystems: coverage-gap analysis and test planning (`/testing:plan`), TDD test authoring and placement (`/testing:write`), live E2E plus non-UI smoke verification (`/testing:run-e2e`), and failing-test root-cause diagnosis with the reproduce → isolate → fix → retest loop (`/testing:diagnose`).",
   "author": {
     "name": "Melodic Software",

plugins/testing/CHANGELOG.md

@@ -3,6 +3,19 @@
 All notable changes to the `testing` plugin are documented here. Format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this plugin uses semantic versioning.
 
+## [0.3.3]
+
+### Changed
+
+- **`/testing:run-e2e`'s handoff no longer delegates surface verification to the bundled `/verify`.**
+  Claude Code v2.1.215 made `/verify` user-invoked only, so **from v2.1.215** "delegate surface
+  verification to it first" named a surface the skill cannot invoke. The handoff now suggests the
+  user run it and consume its findings, and carries the v2.1.215 scope rather than stating the
+  restriction flatly — on 2.1.145–2.1.214 `/verify` is still model-invocable. The instruction itself
+  is uniform across the window, because suggesting is correct on every version `/verify` exists on.
+  The orchestrator path was already the fallback and is unchanged. The `≥ 2.1.145` availability floor
+  is a separate axis, unchanged and re-verified 2026-08-02.
+
 ## [0.3.2]
 
 ### Changed

plugins/testing/skills/run-e2e/SKILL.md

@@ -58,7 +58,7 @@ The workflow steps themselves live in [context/e2e.md](context/e2e.md).
 
 ## Handoff
 
-- Surface verification available → when the bundled `/verify` command is present (Claude Code ≥2.1.145), delegate surface verification to it first and consume its findings; when absent, the orchestrator path in this skill runs unchanged as the fallback
+- Surface verification available → the bundled `/verify` skill (Claude Code ≥2.1.145) covers the same surface, but is [user-invoked only from v2.1.215](https://code.claude.com/docs/en/skills#bundled-skills) — before v2.1.215 Claude could also run it on its own. Suggest the user run it and consume its findings rather than delegating to it, on every version: the suggestion holds across the whole `≥2.1.145` availability window, delegation does not. The orchestrator path in this skill runs unchanged either way
 - All scenarios pass → `/verification:confirm outcome` when the `verification` plugin is installed (composes intent + evidence; chains back here when needed); otherwise report the captured evidence for outcome sign-off directly
 - Visual bugs or API errors found → `/testing:diagnose`
 - Scenario planning needed first → `/testing:plan`

plugins/verification/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "verification",
-  "version": "0.3.5",
+  "version": "0.3.6",
   "description": "Outcome-verification stage: prove a change achieved its intended outcome (`/verification:confirm` — a mechanical build/test/lint prerequisite gate, then intent-match + evidence + verdict with the criterion auto-detected by change type), and verify measurable-improvement claims against a planning-time baseline (`/verification:measure`), never fabricating numbers.",
   "author": {
     "name": "Melodic Software",

plugins/verification/CHANGELOG.md

@@ -3,6 +3,28 @@
 All notable changes to the `verification` plugin are documented here. Format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this plugin uses semantic versioning.
 
+## [0.3.6]
+
+### Changed
+
+- **The live-app delegation path no longer tells the skill to invoke the bundled `/verify`.**
+  Claude Code v2.1.215 made `/verify` and `/code-review` user-invoked only — Claude does not run
+  them on its own — so **from v2.1.215** every instruction routing this skill's live-app run through
+  `/verify` named a surface it cannot reach, silently costing the fallback its primary leg. The
+  shipped wording carries that version rather than stating the restriction flatly: on 2.1.145–2.1.214
+  `/verify` is still model-invocable, and this repository declares no Claude Code support floor that
+  would make an unscoped statement true. The *instruction* stays uniform across the window even so —
+  suggesting `/verify` is correct on every version it exists on, so the skill never probes the
+  client's version. `/run` is unaffected
+  (the change names neither it nor the `run-skill-generator` sibling) and stays the supplementary
+  agent-invocable path; `/verify` is now surfaced as a suggestion for the user to run. The
+  `≥ 2.1.145` availability floor is **unchanged and re-verified 2026-08-02** against the bundled
+  skills reference, which still states it for all three of `/run`, `/verify`, and
+  `/run-skill-generator` — that note was never stale; what changed is who may invoke one of them.
+  The `confirm` skill's graded rubric moved with the behavior: eval 9
+  (`live-app-delegates-to-bundled-with-fallback`) had encoded the removed `/verify` delegation as a
+  pass condition, and would otherwise have graded the corrected skill as failing.
+
 ## [0.3.5]
 
 ### Changed

plugins/verification/README.md

@@ -18,8 +18,8 @@ green build into confirmed outcomes.
   `/toolchain:check` and `/toolchain:lint` when installed, and runs the project's own
   ecosystem-native commands otherwise — the STOP-on-fail gate is unchanged, only the
   executor differs. Live-app verification prefers the `testing` plugin's
-  `/testing:run-e2e` when installed and falls back to Claude Code's bundled `/verify` +
-  `/run` or a manual orchestrator launch, never silently downgrading to a static check.
+  `/testing:run-e2e` when installed and falls back to Claude Code's bundled `/run` or a
+  manual orchestrator launch, never silently downgrading to a static check.
 - **Never fabricates a measurement.** `/verification:measure` requires a baseline
   captured before the change; with none, it reports an honest "cannot quantify" plus a
   current-state measurement, never an invented delta.

plugins/verification/skills/confirm/SKILL.md

@@ -93,7 +93,7 @@ Stage 1 delegates to the `toolchain` plugin's `/toolchain:check` and `/toolchain
 
 Read the criterion context file for the dispatched mode, then run the flow below. The shared spine (intent → inventory → match → evidence → report) is in [context/outcome.md](context/outcome.md); `fix` / `refactor` adapt it.
 
-1. **Auto-trigger `/testing:run-e2e` (when runtime-affecting)** — inspect changed files. If any match an `e2e-*` category from the Runtime-affecting paths above, or touch observability code paths verifiable end-to-end, or the user said "test the app", invoke `/testing:run-e2e` via the Skill tool when the `testing` plugin is installed — otherwise drive the live app directly (Claude Code's bundled `/verify` + `/run`, or a manual orchestrator launch) and capture the same evidence. When present, it validates prerequisites, starts the app, exercises the changed flow, and captures evidence (screenshots, console, network, traces). Carry that into the evidence table. If not runtime-affecting (pure refactor, internal lib, doc-only): note "E2E not applicable" and skip.
+1. **Auto-trigger `/testing:run-e2e` (when runtime-affecting)** — inspect changed files. If any match an `e2e-*` category from the Runtime-affecting paths above, or touch observability code paths verifiable end-to-end, or the user said "test the app", invoke `/testing:run-e2e` via the Skill tool when the `testing` plugin is installed — otherwise drive the live app directly (Claude Code's bundled `/run`, or a manual orchestrator launch) and capture the same evidence. When present, it validates prerequisites, starts the app, exercises the changed flow, and captures evidence (screenshots, console, network, traces). Carry that into the evidence table. If not runtime-affecting (pure refactor, internal lib, doc-only): note "E2E not applicable" and skip.
 2. **Intent retrieval** — scan the conversation for the original request, the approved plan, refinements, and acceptance criteria. If none is clear, ask the user what the goal was.
 3. **Implementation inventory** — changed files, new capabilities, behavior changes, config/infra changes.
 4. **Intent match** — every requirement has implementation; every implementation traces to a requirement; flag scope additions and gaps (including implicit requirements — error handling, edge cases, tests).
@@ -109,8 +109,8 @@ When `/testing:run-e2e` ran, persist an assertion-only evidence manifest (what w
 For "run the live app and watch it behave" — beyond automated `/testing:run-e2e` — `/verification:confirm` delegates rather than reimplementing app-launch:
 
 - **Primary: `/testing:run-e2e`** (when the `testing` plugin is installed) — the reliable path for orchestrated apps (Aspire, docker-compose, tilt) via the project's orchestrator tooling + Playwright CLI. It can isolate the drive loop in a subagent so the orchestrator consumes only evidence paths, emit an optional recording / session-artifact evidence tier (config-driven, defaults off — screenshots stay the evidence floor), and on a failed prerequisite return a structured verification-environment gap report rather than a bare stop. Carry any recording and session-artifact pointers it produces into the evidence table.
-- **Supplementary: Claude Code's bundled `/verify` + `/run`** — when a quick interactive run is enough and the orchestrated harness is overkill (requires Claude Code ≥2.1.145 — verified 2026-07-18 against [bundled skills](https://code.claude.com/docs/en/skills#bundled-skills)).
-- **Graceful fallback** — if the bundled skills cannot infer the project's launch (or the CC version lacks them), fall back to `/testing:run-e2e` when the `testing` plugin is installed, or a manual orchestrator launch otherwise. Never silently downgrade live-app verification to a static check — surface the gap.
+- **Supplementary: Claude Code's bundled `/run`** — when a quick interactive run is enough and the orchestrated harness is overkill (requires Claude Code ≥2.1.145). Its sibling `/verify` covers the same ground and shares that `≥2.1.145` availability floor, but **from v2.1.215 it is user-invoked only** — before v2.1.215 Claude could also run it on its own. Suggest the user run it on every version rather than probing the client's: the suggestion holds across the whole availability window, delegation does not. Availability floor and invocability are separate axes; both were verified 2026-08-02 against [bundled skills](https://code.claude.com/docs/en/skills#bundled-skills).
+- **Graceful fallback** — if `/run` cannot infer the project's launch (or the CC version lacks it), fall back to `/testing:run-e2e` when the `testing` plugin is installed, or a manual orchestrator launch otherwise. Never silently downgrade live-app verification to a static check — surface the gap.
 
 ## Edge cases
 
@@ -125,7 +125,7 @@ For "run the live app and watch it behave" — beyond automated `/testing:run-e2
 |-----------|--------|
 | Review gate passes (no blocking findings — e.g. `/review:quality-gate` when installed) | Suggest `/verification:confirm` |
 | Stage 1 fails | Fix build/test/lint/cross-cutting, then re-run `/verification:confirm` |
-| Runtime-affecting change | Stage 2 auto-triggers `/testing:run-e2e` (bundled `/verify` / `/run` supplementary) |
+| Runtime-affecting change | Stage 2 auto-triggers `/testing:run-e2e` (bundled `/run` supplementary) |
 | `/verification:confirm` finds CONFIRMED | Suggest a retro (`/session-flow:retro` when installed), then the project's PR flow (`/source-control:pull-request` when installed) |
 | `/verification:confirm` finds gaps | Fix, then re-run `/verification:confirm` |
 | Improvement claimed without data | Redirect to `/verification:measure` (`performance` or `metrics`; baseline captured at planning time) |
@@ -145,4 +145,4 @@ For "run the live app and watch it behave" — beyond automated `/testing:run-e2
 - **Don't skip outcome confirmation for "obvious" changes.** Small changes drift from intent unnoticed; the intent check catches scope creep and missed requirements tests don't cover.
 - **Don't quantify improvement claims here.** "Faster" / "simpler" claims route to `/verification:measure` and its baseline discipline — never assert an improvement from this skill's evidence alone.
 - **Don't re-run Stage 1 if it already passed this conversation** and nothing changed since — reuse the results.
-- **Live-app fallback is fidelity-preserving.** If the bundled `/verify` / `/run` can't launch the app, fall back to `/testing:run-e2e` and SAY SO — never silently swap live observation for a static check.
+- **Live-app fallback is fidelity-preserving.** If the bundled `/run` can't launch the app, fall back to `/testing:run-e2e` and SAY SO — never silently swap live observation for a static check.

plugins/verification/skills/confirm/evals/evals.json

@@ -77,12 +77,12 @@
       "id": 9,
       "name": "live-app-delegates-to-bundled-with-fallback",
       "prompt": "/verification:confirm outcome — run the live app and show me the change working",
-      "expected_output": "For live-app run/observe, delegates to /testing:run-e2e (primary for orchestrated apps) or bundled /verify + bundled /run, with a fidelity-preserving graceful fallback: if bundled /verify or /run cannot infer the project's launch, falls back to /testing:run-e2e and surfaces the gap rather than silently downgrading to a static check.",
+      "expected_output": "For live-app run/observe, delegates to /testing:run-e2e (primary for orchestrated apps) or the bundled /run (supplementary, agent-invocable, on the ≥2.1.145 availability floor). The bundled /verify shares that floor but is user-invoked only from v2.1.215 — before v2.1.215 Claude could also run it — so it is offered as a suggestion for the user to run rather than delegated to, uniformly on every version rather than branched on the client's. Fidelity-preserving graceful fallback: if bundled /run cannot infer the project's launch, falls back to /testing:run-e2e and surfaces the gap rather than silently downgrading to a static check.",
       "files": [],
       "expectations": [
         "Delegates live-app observation rather than reimplementing app launch",
-        "Treats /testing:run-e2e as primary for orchestrated apps; bundled /verify + /run as supplementary",
-        "On bundled-tool failure, falls back to /testing:run-e2e and SAYS SO — never silently swaps live observation for a static check"
+        "Treats /testing:run-e2e as primary for orchestrated apps and the bundled /run as supplementary; surfaces the bundled /verify as a suggestion for the user to run rather than invoking it, scoping the user-invoked-only restriction to v2.1.215 and later rather than stating it unscoped",
+        "On bundled /run failure, falls back to /testing:run-e2e and SAYS SO — never silently swaps live observation for a static check"
       ]
     },
     {