feat(planning): add plan_stress_test per mode with its 12-plan eval suite

.changes/planning/feat-12478-plan-stress-test-7c41e2a9.md

@@ -0,0 +1,8 @@
+---
+bump: minor
+---
+
+### Added
+
+- `plan`: `plan_stress_test` sets, per mode of the draft (`plan`, `proposal`, `incumbent`), which stress-tests `/planning:plan` runs: `auto` (the default, current behavior), `reviewer`, `devils-advocate` or `both`. It resolves through the convention home; `reference/config.md` defines the modes and values (#12478).
+- `evals`: the `plan-stress-test` suite, 12 seeded drafts across the three modes with deterministic graders, and `fixtures/stress-arm/prepare-arm.sh`, which sets up a plugin copy for one arm or for the suite's negative control (#12478).

plugins/planning/README.md

@@ -20,7 +20,7 @@ where artifacts land in the consuming repo.
 | `/planning:design` | Design space | Explores types, contracts, module boundaries, and package topology through collaborative discussion rounds, producing capability-matrix / type-inventory / design-threads / topology artifacts; its `handoff` action delegates to `/planning:design-handoff`. Its `data`, `integration`, and `system` artifacts are typed: each carries a `scope` + `dialect` label and a fenced block in the dialect the `diagram_dialect` convention key resolves to (mermaid `erDiagram` or DBML for data; mermaid `sequenceDiagram` plus an OpenAPI 3.1 sketch for integration; a LikeC4 or C4-PlantUML container view for system, and no C4 view at all when that key is unset). `library` and `module` emit no typed artifact. |
 | `/planning:design-handoff` | Design→plan gate | Gates a finished design for `/planning:plan`. The gate is a binary check that every `design-threads.md` thread is RESOLVED, directional, or TAGGED-DEFERRED. Then it packages the plan-ready summary and resume prompt, or FAILs and routes back to `/planning:design`. On PASS and FAIL alike it also emits an advisory six-dimension coverage table (what, how, where, who, when, why) that never contributes to the verdict; uncovered dimensions ride along in the handoff summary and the resume prompt. |
 | `/planning:devils-advocate` | Adversarial review | Stress-tests plans via assumption extraction, evidence checks, failure scenarios, and operational-gotcha sweeps. Every finding evidence-backed, never generic warnings. An `incumbent` mode turns the same lens on the status quo: an Alternatives Sweep that stress-tests keeping an incumbent tool/approach against alternatives (native > official > vetted ladder, coupling priced, KEEP / MIGRATE / RESEARCH verdict), exploring the incumbent first-hand in a fresh sub-agent. |
-| `/planning:plan` | Implementation plan | Produces a structured plan (goal, approach, test strategy, blast radius, parallelism analysis, tagged unilateral decisions) with a mandatory fresh-context `plan-reviewer` stress-test and a user approval gate, persisted to PLAN.md. |
+| `/planning:plan` | Implementation plan | Produces a structured plan (goal, approach, test strategy, blast radius, parallelism analysis, tagged unilateral decisions) with a fresh-context `plan-reviewer` stress-test, on by default and set per mode by `plan_stress_test`, and a user approval gate, persisted to PLAN.md. |
 | `/planning:setup` | Configuration | `check` inspects the standards index read-only; `apply` interviews the consumer and bootstraps the standards index (idempotent; re-run to reconfigure). |
 
 The pipeline composes end-to-end. `wayfind` charts the fog upstream when an effort

plugins/planning/evals/fixtures/stress-arm/apply-arm.sh

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+# Run by each st-* case's scaffold, in the eval workspace: writes the arm's plan_stress_test
+# setting where /planning:plan resolves it and commits the seeded project.
+set -euo pipefail
+
+mkdir -p docs/conventions
+cp "$(dirname "${BASH_SOURCE[0]}")/planning.yaml" docs/conventions/planning.yaml
+git add .
+git commit -q -m "chore: import project"

plugins/planning/evals/fixtures/stress-arm/devils-advocate-degraded-body.md.txt

@@ -0,0 +1,9 @@
+**Arguments.** `[deep|shallow] [incumbent [target]|<plan text or file path>]`.
+
+## Variables
+
+Arguments: `$ARGUMENTS`
+
+## Purpose
+
+Read the plan or incumbent named in the arguments and report what you think of it.

plugins/planning/evals/fixtures/stress-arm/planning.yaml

@@ -0,0 +1,4 @@
+plan_stress_test:
+  plan: auto
+  proposal: auto
+  incumbent: auto

plugins/planning/evals/fixtures/stress-arm/prepare-arm.sh

@@ -0,0 +1,50 @@
+#!/usr/bin/env bash
+# Copies the planning plugin to a new directory outside any repository, set up for one arm of the
+# st-* suite, which `claude plugin eval` then runs with --ablation none.
+#
+#   prepare-arm.sh <reviewer|devils-advocate|both|auto> <dest> [--degrade-devils-advocate]
+#
+# The first argument is the plan_stress_test value every mode gets in the copy. With
+# --degrade-devils-advocate, the copy's devils-advocate skill keeps its frontmatter and takes its
+# body from devils-advocate-degraded-body.md.txt: the suite's negative control.
+#
+# Exit: 0 written; 2 a usage error, or a dest that already exists.
+set -euo pipefail
+
+usage() {
+  echo "usage: prepare-arm.sh <reviewer|devils-advocate|both|auto> <dest> [--degrade-devils-advocate]" >&2
+  exit 2
+}
+
+[[ $# -ge 2 && $# -le 3 ]] || usage
+value="$1"
+dest="$2"
+degrade="${3:-}"
+case "$value" in
+reviewer | devils-advocate | both | auto) ;;
+*) usage ;;
+esac
+[[ -z "$degrade" || "$degrade" == "--degrade-devils-advocate" ]] || usage
+if [[ -e "$dest" ]]; then
+  echo "prepare-arm.sh: $dest already exists" >&2
+  exit 2
+fi
+
+here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+root="$(cd "$here/../../.." && pwd)"
+
+mkdir -p "$dest"
+cp -R "$root/." "$dest/"
+rm -rf "${dest:?}/evals/results"
+
+fixture="$dest/evals/fixtures/stress-arm"
+printf 'plan_stress_test:\n  plan: %s\n  proposal: %s\n  incumbent: %s\n' "$value" "$value" "$value" >"$fixture/planning.yaml"
+
+if [[ -n "$degrade" ]]; then
+  skill="$dest/skills/devils-advocate/SKILL.md"
+  frontmatter="$(awk 'NR == 1 || !done { print } NR > 1 && /^---$/ { done = 1 }' "$skill")"
+  printf '%s\n\n' "$frontmatter" >"$skill"
+  cat "$fixture/devils-advocate-degraded-body.md.txt" >>"$skill"
+fi
+
+echo "$dest"

plugins/planning/evals/st-incumbent-1/case.yaml

@@ -0,0 +1,6 @@
+schema_version: "1.1"
+name: st-incumbent-1
+description: "A draft plan for a veterinary clinic reminder service decides whether to keep its in-process job scheduler."
+expected_outcome: "A good review reports that the kept scheduler's literal per-clinic UTC offset cannot hold 09:00 local time across daylight saving changes, that the job store the draft relies on is absent from the pinned 2.8.1 release, and that one store file shared by the two sharded workers makes both fire every job. It does not attack the batch size of 200, which matches the gateway limit."
+context:
+  scaffold_script: scaffold.sh

plugins/planning/evals/st-incumbent-1/graders/defect-a.md

@@ -0,0 +1,6 @@
+---
+type: regex
+pattern: '^(?![^\n]*\bNo plan gaps found)(?![^\n]*\b(?:is|are|was|were|looks?|seems?|remains?)\s+(?!not\b|never\b)(?:(?:perfectly|entirely|totally|already|also|still|actually)\s+)?(?:fine|ok|okay|good|correct|accurate|sound|acceptable|safe|justified|reasonable)\b(?![^\n]*\b(?:but|however|yet)\b))(?![^\n]*\b(?:no|not\s+an?|n''t\s+an?)\s+(?:\w+\s+){0,2}(?:issue|problem|concern|risk|defect|gap|finding|blocker)s?\b)(?![^\n]*\b(?:nothing\s+wrong|no\s+need\s+to|need\s+not|can\s+be\s+ignored|no\s+action\s+needed|no\s+changes?\s+needed)\b)(?![^\n]*\b(?<!n''t )(?<!not )(?<!never )(?<!cannot )(?<!can''t )(?:handles?|follows?|tracks?|covers?|accounts?\s+for|adjusts?\s+for|survives?)\s+(?:\w+\s+){0,2}(?:daylight|DST|clock\s+changes?))(?=[^\n]*(?:\btickwheel\b|\butc_offset\b|\boffsets?\b|\bclinics\.ya?ml\b|\bAC1\b))[^\n]*(?:\bdaylight\b|\bDST\b|\bclock\s+changes?\b|\bsummer\s+time\b|\bstandard\s+time\b|\b(?:an?|one|1)[- ]hour\s+(?:early|late|off)\b|\bseasonal\w*)'
+flags: mi
+target: last_message
+---

plugins/planning/evals/st-incumbent-1/graders/defect-b.md

@@ -0,0 +1,6 @@
+---
+type: regex
+pattern: '^(?![^\n]*\bNo plan gaps found)(?![^\n]*\b(?:is|are|was|were|looks?|seems?|remains?)\s+(?!not\b|never\b)(?:(?:perfectly|entirely|totally|already|also|still|actually)\s+)?(?:fine|ok|okay|good|correct|accurate|sound|acceptable|safe|justified|reasonable)\b(?![^\n]*\b(?:but|however|yet)\b))(?![^\n]*\b(?:no|not\s+an?|n''t\s+an?)\s+(?:\w+\s+){0,2}(?:issue|problem|concern|risk|defect|gap|finding|blocker)s?\b)(?![^\n]*\b(?:nothing\s+wrong|no\s+need\s+to|need\s+not|can\s+be\s+ignored|no\s+action\s+needed|no\s+changes?\s+needed)\b)(?![^\n]*\b(?:no|without(?:\s+an?)?)\s+(?:version\s+)?(?:upgrade|bump)\b)(?=[^\n]*(?:\bFileJobStore\b|\bjob\s*store\b|\bpersistent\s+store\b))(?=[^\n]*(?:\b2\.8\b|\b2\.8\.1\b|\brequirements\.txt\b|\bpin\w*|\b3\.0\b|\b3\.0\.0\b))[^\n]*(?:\blacks?\b|\babsent\b|\bmissing\b|\bnot\s+(?:in|available|present|shipped|included)\b|\b(?:isn''t|aren''t|doesn''t|does\s+not)\s+\w+|\bpredates?\b|\bonly\s+(?:in|from|since)\b|\badd(?:ed|s)\s+(?:\w+\s+)?in\b|\bintroduced\s+in\b|\barrives?\s+in\b|\bships?\s+in\b|\bnew\s+in\b|\brequires?\b|\bneeds?\b|\bupgrade\b|\bbump\b)'
+flags: mi
+target: last_message
+---

plugins/planning/evals/st-incumbent-1/graders/defect-c.md

@@ -0,0 +1,6 @@
+---
+type: regex
+pattern: '^(?![^\n]*\bNo plan gaps found)(?![^\n]*\b(?:is|are|was|were|looks?|seems?|remains?)\s+(?!not\b|never\b)(?:(?:perfectly|entirely|totally|already|also|still|actually)\s+)?(?:fine|ok|okay|good|correct|accurate|sound|acceptable|safe|justified|reasonable)\b(?![^\n]*\b(?:but|however|yet)\b))(?![^\n]*\b(?:no|not\s+an?|n''t\s+an?)\s+(?:\w+\s+){0,2}(?:issue|problem|concern|risk|defect|gap|finding|blocker)s?\b)(?![^\n]*\b(?:nothing\s+wrong|no\s+need\s+to|need\s+not|can\s+be\s+ignored|no\s+action\s+needed|no\s+changes?\s+needed)\b)(?![^\n]*\b(?:won''t|will\s+not|cannot|can''t|not|never|no)\s+(?:\w+\s+)?(?:\bduplicat\w*|double[- ]?\w+|fire\s+twice|send\s+twice))(?=[^\n]*(?:\bworker-[01]\b|\bSHARD_INDEX\b|\bshard\w*|\b(?:two|both|2)\s+(?:\w+\s+)?(?:workers?|instances?|containers?|processes|wheels?)\b|\bsecond\s+(?:worker|instance|container|process|wheel)\b|\bmultiple\s+workers?\b))[^\n]*(?:\bduplicat\w*|\bdouble[- ]?(?:send|fire|sent|fired|fires|sends|sending|firing)\b|\btwice\b|\b(?:both|each)\s+(?:\w+\s+)?(?:fire|send|load|read)s?\b|\bone\s+(?:Wheel|process|worker)\b|\bsingle\s+(?:Wheel|process|worker)\b|\bcorrupt\w*|\bno\s+(?:file\s+)?lock\w*|\bcontention\b|\bshar(?:e|es|ed|ing)\s+(?:\w+\s+){0,2}(?:store|file|jobs\.db|volume)\b)'
+flags: mi
+target: last_message
+---

plugins/planning/evals/st-incumbent-1/graders/sound-part.md

@@ -0,0 +1,7 @@
+---
+type: regex
+pattern: '^(?![^\n]*\bNo plan gaps found)(?![^\n]*\b(?:is|are|was|were|looks?|seems?|remains?)\s+(?!not\b|never\b)(?:(?:perfectly|entirely|totally|already|also|still|actually)\s+)?(?:fine|ok|okay|good|correct|accurate|sound|acceptable|safe|justified|reasonable)\b(?![^\n]*\b(?:but|however|yet)\b))(?![^\n]*\b(?:no|not\s+an?|n''t\s+an?)\s+(?:\w+\s+){0,2}(?:issue|problem|concern|risk|defect|gap|finding|blocker)s?\b)(?![^\n]*\b(?:nothing\s+wrong|no\s+need\s+to|need\s+not|can\s+be\s+ignored|no\s+action\s+needed|no\s+changes?\s+needed)\b)(?![^\n]*(?:\bnot|n''t)\s+(?:\w+\s+)?(?:too|arbitrar\w*|wrong|incorrect|unjustified|exceed\w*|a\s+bottleneck)\b)(?=[^\n]*(?:\bBATCH_SIZE\b|\bbatch\s+size\b|\bbatch(?:es)?\s+of\s+200\b|\b200[- ](?:recipient|per|message)\w*))[^\n]*(?:\btoo\s+(?:small|low|large|high|big)\b|\barbitrar\w*|\bunjustified\b|\bwrong\b|\bincorrect\b|\bexceed\w*|\bshould\s+(?:be\s+)?(?:raised?|increased?|lowered?|reduced?|larger|smaller)\b|\b(?:raise|increase|lower|reduce)\s+(?:it|the|BATCH)\b|\bbottleneck\b|\bunsupported\b|\bno\s+basis\b|\bunverified\b)'
+flags: mi
+match: not_contains
+target: last_message
+---

plugins/planning/evals/st-incumbent-1/prompt.md

@@ -0,0 +1,9 @@
+---
+tags: [plan-stress-test]
+runs: 3
+max_turns: 40
+timeout_seconds: 1500
+allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent]
+---
+
+/planning:plan Continue the draft at `.work/review-target/PLAN.md` from Step 3. The Brief and the design gate are settled; `.work/review-target/design/design-resolution.md` records the design early exit. Run Steps 3, 3b and 4 as this skill and this repository's settings say. Then list every finding you verified as one bullet, naming the file, phase or claim it concerns and saying what is wrong. Stop there: no Step 4.5 onward, and edit no file.

plugins/planning/evals/st-incumbent-1/samples/defect-a.json

@@ -0,0 +1,54 @@
+{
+  "pass": [
+    {
+      "answer": "- AC1 is not met: tickwheel's `utc_offset` is a fixed value, so sends drift when daylight saving time starts.\n",
+      "why": "bullet naming the option"
+    },
+    {
+      "answer": "| 1 | IMPORTANT | Plan | `config/clinics.yaml` stores a literal offset per clinic; after a DST switch the 09:00 send fires an hour late | Use a zone-aware trigger |\n",
+      "why": "table row naming the file"
+    },
+    {
+      "answer": "- Tickwheel has no zone database, so clinics leave 09:00 local at each clock change and AC1 fails half the year.\n",
+      "why": "clock change phrasing"
+    },
+    {
+      "answer": "| 2 | CRITICAL | Plan | `utc_offset` -06:00 only holds in winter: in summer time Maple Street sends one hour off | Replace the trigger |\n",
+      "why": "summer time phrasing"
+    },
+    {
+      "answer": "- The draft treats the offset as fine, but it is wrong for a clinic that observes DST.\n",
+      "why": "quotes the draft, then contradicts it"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "",
+      "why": "empty"
+    },
+    {
+      "answer": "No plan gaps found.\n",
+      "why": "clean pass"
+    },
+    {
+      "answer": "- Tickwheel's utc_offset handles DST across the year.\n",
+      "why": "dismissed: claims it handles DST"
+    },
+    {
+      "answer": "- The per-clinic offset is fine across DST.\n",
+      "why": "dismissed"
+    },
+    {
+      "answer": "- AC1 has no daylight saving problem: the offset is per clinic.\n",
+      "why": "negated"
+    },
+    {
+      "answer": "- Phase 3 has no test for a refused gateway request.\n",
+      "why": "another finding"
+    },
+    {
+      "answer": "- AC1 is untraceable to a test.\n",
+      "why": "AC1 named, nature absent"
+    }
+  ]
+}

plugins/planning/evals/st-incumbent-1/samples/defect-b.json

@@ -0,0 +1,54 @@
+{
+  "pass": [
+    {
+      "answer": "- `FileJobStore` is not in the release we run: `requirements.txt` pins tickwheel 2.8.1 and the changelog adds it in 3.0.0.\n",
+      "why": "bullet naming both files"
+    },
+    {
+      "answer": "| 1 | CRITICAL | Plan | Phase 1's job store needs Tickwheel 3.0; we pin 2.8.1 | Add the upgrade as a phase |\n",
+      "why": "table row"
+    },
+    {
+      "answer": "- Why claims the image includes FileJobStore, but the pin is 2.8.1, which predates it.\n",
+      "why": "predates"
+    },
+    {
+      "answer": "| 2 | IMPORTANT | Plan | `FileJobStore` only from 3.0.0, so requirements.txt must change before Phase 1 | Add an upgrade step |\n",
+      "why": "only from"
+    },
+    {
+      "answer": "- The draft says the job store is correct for our release, but 2.8.1 lacks it.\n",
+      "why": "quotes, then contradicts"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "",
+      "why": "empty"
+    },
+    {
+      "answer": "No plan gaps found.\n",
+      "why": "clean pass"
+    },
+    {
+      "answer": "- FileJobStore is available in the pinned 2.8.1, so no upgrade is required.\n",
+      "why": "dismissed: no upgrade"
+    },
+    {
+      "answer": "- The job store claim is correct: tickwheel 2.8.1 includes it.\n",
+      "why": "named and cleared"
+    },
+    {
+      "answer": "- FileJobStore has a good fit with AC2.\n",
+      "why": "no version or nature"
+    },
+    {
+      "answer": "- Phase 3 has no test for a refused gateway request.\n",
+      "why": "another finding"
+    },
+    {
+      "answer": "- Pin the httpx version in requirements.txt; it needs a floor.\n",
+      "why": "requirements.txt without the store"
+    }
+  ]
+}

plugins/planning/evals/st-incumbent-1/samples/defect-c.json

@@ -0,0 +1,54 @@
+{
+  "pass": [
+    {
+      "answer": "- Phase 1 mounts one `jobs.db` on `worker-0` and `worker-1`; each wheel loads every job in the file, so both fire every clinic and owners get duplicates.\n",
+      "why": "bullet naming the services"
+    },
+    {
+      "answer": "| 1 | CRITICAL | Plan | A single shared store breaks the SHARD_INDEX split: the two workers both load all jobs and send twice | One store per shard |\n",
+      "why": "table row"
+    },
+    {
+      "answer": "- The changelog says a store file serves one Wheel at a time and takes no lock, but compose runs two workers sharing the store file.\n",
+      "why": "one Wheel phrasing"
+    },
+    {
+      "answer": "| 2 | IMPORTANT | Plan | Sharding is lost once both workers read `/state/jobs.db`: double-sends break AC2 | Separate volumes |\n",
+      "why": "double-sends"
+    },
+    {
+      "answer": "- Two containers on the `reminders-state` volume risk a corrupted jobs.db, since the store has no file lock.\n",
+      "why": "corruption phrasing"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "",
+      "why": "empty"
+    },
+    {
+      "answer": "No plan gaps found.\n",
+      "why": "clean pass"
+    },
+    {
+      "answer": "- The two workers will not duplicate reminders because each has its own shard.\n",
+      "why": "negated"
+    },
+    {
+      "answer": "- Sharing the store across worker-0 and worker-1 is fine.\n",
+      "why": "dismissed"
+    },
+    {
+      "answer": "- A shard rebalance is not a problem here; double-sends are not a concern.\n",
+      "why": "negated concern"
+    },
+    {
+      "answer": "- Phase 3 has no test for a refused gateway request.\n",
+      "why": "another finding"
+    },
+    {
+      "answer": "- `worker-0` has no health check in compose.\n",
+      "why": "service named, nature absent"
+    }
+  ]
+}

plugins/planning/evals/st-incumbent-1/samples/sound-part.json

@@ -0,0 +1,46 @@
+{
+  "pass": [
+    {
+      "answer": "",
+      "why": "empty"
+    },
+    {
+      "answer": "No plan gaps found.\n",
+      "why": "clean pass"
+    },
+    {
+      "answer": "- `BATCH_SIZE` 200 is correct: it matches the Postrider limit.\n",
+      "why": "praises the element"
+    },
+    {
+      "answer": "- BATCH_SIZE of 200 is not arbitrary; the gateway refuses longer lists.\n",
+      "why": "negated attack"
+    },
+    {
+      "answer": "- AC1 fails: the utc_offset is fixed, so DST moves the send time.\n",
+      "why": "other finding"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "- `BATCH_SIZE` of 200 is arbitrary and nothing justifies it.\n",
+      "why": "bullet attack"
+    },
+    {
+      "answer": "| 1 | IMPORTANT | Plan | Batch size 200 is too small for 40,000 reminders | Raise it |\n",
+      "why": "table row"
+    },
+    {
+      "answer": "- A batch size of 200 will be a bottleneck on peak day.\n",
+      "why": "bottleneck"
+    },
+    {
+      "answer": "- BATCH_SIZE should be increased to shorten the send window.\n",
+      "why": "should be increased"
+    },
+    {
+      "answer": "| 2 | SUGGESTION | Plan | The 200-recipient batch claim is unverified | Check it |\n",
+      "why": "unverified"
+    }
+  ]
+}

plugins/planning/evals/st-incumbent-1/scaffold.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+# Seeds the workspace, then applies this arm's plan_stress_test setting.
+set -euo pipefail
+here="$(dirname "${BASH_SOURCE[0]}")"
+bash "$here/seed.sh"
+bash "$here/../fixtures/stress-arm/apply-arm.sh"