[kyle-sexton]

Closes #12478

Summary

Adds the plan_stress_test setting to the planning plugin (per draft mode: plan, proposal, incumbent; values auto, reviewer, devils-advocate, both; default auto) and a 12-case eval suite (plugins/planning/evals/st-*, tag plan-stress-test) that measures which value each mode should default to. The measured pass kept auto in every mode: the negative control detected no loss, so the suite cannot measure what the stress test adds. reference/config.md cites the Probe result.

Fix

• plan_stress_test in plugins/planning/schemas/planning.schema.json, reference/config.md and /planning:plan, beside main's wayfind_suggest_questions.
• 12 drafts, 4 per mode, written by fresh agents that did not see the arms or read the skills. Each has 3 seeded defects and one sound-part must-not-flag check, graded by deterministic regex graders on last_message.
• evals/fixtures/stress-arm/prepare-arm.sh builds one plugin copy per arm, plus the degraded-body negative control.
• tests/plan-stress-test.test.sh proves no detection grader passes on a workspace's own text.

Pre-registration (recorded before any paid run; not amended after a number is read)

Frozen suite: 56f217b92f. Nothing under plugins/planning/evals/ changes after the first batch run.

Arms. One plugin copy per arm, with plan_stress_test giving all three modes the same value. The prompt is byte-identical across arms.

Arm	Value	What runs on the draft
A	reviewer	plan-reviewer only
B	devils-advocate	/planning:devils-advocate in the draft's mode only
C	both	both

Arms: owner decision, 2026-11-03 in the lane 2 session, choosing A / B / C over a no-review baseline and over a fourth no-review arm. #7586 Q165 left the arms undefined, and Q166 keeps plan-reviewer on, so the three arms are the settings the default chooses between.

Numbers. Each run is a claude plugin eval run with --ablation none, --runs 3, -j 1, --model claude-opus-5-5, --judge-model claude-opus-5-5 (no case calls a judge) and --threshold 0.5 (no rule reads it). That gives 12 × 3 × 3 = 108 arm runs plus 36 control runs. A run's score is the fraction of its 4 graders it passed; a case's score is the mean of its conforming runs. Before the batch there are 3 Sonnet smoke runs, one each for arms C, B and A. They check that the setting resolves (each trace shows the plan_stress_test report line), they show which model plan-reviewer runs on given its opus pin, and their measured cost sets the batch's --max-cost-usd.

Exclusions.

1. run-validity.py --scaffolded must print verdict: VALID for each result file; the control file also gets --own-copy-search-warns.
2. A run whose trace lacks its arm's dispatches is removed by arm-report.py. That includes a devils-advocate run on an st-incumbent-* case that is not in incumbent mode.
3. A run with a non-null error makes its case non-comparable. A usage or rate-limit error is rerun per case, never counted.
4. A run that stopped before listing its findings makes its case non-comparable in that arm.
5. A mode with fewer than 3 comparable cases in either compared arm gets no decision, and auto stays.

Addendum 2026-11-03T16:5xZ (validity, written before any score was read). B and C first read INVALID on 3 denials, each the run searching its own plugin copy, which sits outside the sandbox but is readable in a real install. A rerun can't fix that cause, so: all four files are read with --own-copy-search-warns (all VALID), and the 6 runs with an own-copy search denial are excluded in every arm. They are B st-incumbent-1 r2, B st-proposal-3 r2, C st-plan-4 r3, and the control's st-incumbent-3 r1, st-proposal-3 r2 and st-proposal-4 r3. Each affected case keeps 2 runs. Limit: these searches occur only in arms that run devils-advocate. Full text in the plan file.

Variance (read first). If a mode's median per-case range across runs is above 0.25 in any arm, auto stays in that mode and the report says 3 runs do not separate the arms there.

Negative control. prepare-arm.sh devils-advocate --degrade-devils-advocate keeps the skill's frontmatter and replaces its body with a body that has no procedure and no output format. It runs beside arm B. The control detects a loss when noise-report.py calls a case drop or the mean case score falls by 0.15 or more over all 12 cases. If it detects nothing, every mode keeps auto and the skill's contribution is reported as unmeasurable by this suite.

Decision rule, per mode, read after the variance check and the control:

1. If C over A (noise-report.py, margin 0.15) shows a change of +0.15 or more with an interval whose lower end is above 0: plan and proposal stay auto (now measured), and incumbent becomes both.
2. Else, if A against C reads non-inferior at margin 0.15 with no case drop: the mode becomes reviewer.
3. Else: auto stays, reported as not settled, with the intervals.

B never sets a default. B against C at margin 0.05 is reported as a finding only. When C's sound-part misses exceed A's by 3 or more in a mode, that is reported beside the mode's result.

Launch limits. 4 concurrent model runs (arms A, B, C and the control, each -j 1) under the host cap of 8. Each launch needs a 1-minute load under 24 and at least 16 GB of memory available, with an EVAL CLAIM line in the hub outbox before the launch and a release line after.

Verification

• Pre-push passed on 56f217b92f: check-affected (14 passed, 1 skipped over 157 files), reference-verify, pre-push-content.
• plugins/planning/tests/plan-stress-test.test.sh: PASS=71 FAIL=0. Locally the test's fake HOME breaks the mise node shim, so it ran with node on PATH directly; CI runs a real node.
• validate-cases.py plugins/planning/evals: exit 0. It printed 12 WARNs, one per st-* case, because the prompt lists Bash in allowed_tools; the eval runs grant Bash with --allow-tools Bash.
• Typo fixes to the grader regexes were checked old against new over every sample and seed: 0 differences across 15 graders. The [TODO] phase tags in 8 seeds are now filled in by seed.sh, and the generated workspaces are byte-identical to before.
• Smoke (Sonnet, 1 run each, 2026-11-03, suite 56f217b92f): C on st-plan-1 1.00 in 209 s; B on st-incumbent-1 0.75 in 128 s (missed defect-b); A on st-proposal-1 0.75 in 130 s (missed defect-a). All three conform per arm-report.py. Each trace shows plan_stress_test resolved from the copy's docs/conventions/planning.yaml to its arm's value. B ran devils-advocate in incumbent mode with no plan-reviewer; A ran plan-reviewer with no devils-advocate. plan-reviewer's messages are claude-opus-5-5 under --model claude-sonnet-5-5, so its opus pin holds, and the batch's reviewer runs on Opus in every arm. The batch keeps --max-cost-usd 200 per pass: the smoke runs' highest cost times 36 runs is far below it at Sonnet rates, which leaves room for Opus without a partial pass (judgment).
• Measured pass (144 runs, claude-opus-5-5, Claude Code 2.1.317), full record in the Probe result:
  • Validity: all four VALID under the addendum.
  • Variance: no mode is above 0.25 in any arm.
  • Negative control: not detected (−0.05, interval −0.09 to −0.01, no case drop), so every mode keeps auto.
  • Arm means were 0.92 to 1.00, so the suite sits near its ceiling.
  • Rule 1 was not met in any mode. Rule 2 would hold in every mode but is not applied, because the control failed.
  • The spot-read found a sound-part grader false positive, and it covered one run, not the six pre-registered.

Related

• planning: plan_stress_test per mode, settled by the devils-advocate eval #12478 (this slice), planning: interview restate, wayfind absorb, stress test per mode #7586 (parent; Q165 fixes 12 plans, 3 arms, 3 runs; Q166 keeps plan-reviewer on).
• Hand-off phrasing is already measured in test(evals): run the hand-off phrasing eval #11959; nothing new runs for it.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-03T17:10:34.935414Z	6734fd5	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6734fd56e9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

claude-security-review reviewed 4e3ad91f76c667ae5ec73713320a97315c9277fe: read 14 of 14 changed file(s); 0 finding(s), 0 as inline comments.

Security review: incremental (PR #12835, d56cdd6d → 4e3ad91f)

I found no security issues in this push.

What I read. I read every hunk in .claude-lane/incremental.diff, covering all 14 files on the incremental list:

• six regex graders in plugins/planning/evals/:
  • st-incumbent-4/graders/sound-part.md
  • st-plan-2/graders/defect-b.md
  • st-plan-4/graders/defect-a.md, defect-b.md and defect-c.md
  • st-proposal-3/graders/sound-part.md
  • st-proposal-4/graders/sound-part.md
• the eight sample JSON files that pair with those graders.

None of these files matches a TRUSTED PATHS: glob.

Why there is nothing to report.

• What changed. Each change edits one pattern: value, adjusting which wording counts as a dismissal: wider negation windows, a "but/however" exception, narrower exclusion lists. The sample files get new pass and fail strings to match.
• Where it runs. The local claude plugin eval harness applies these patterns to a model's last message. They run no code and take no outside input. They don't touch a workflow, action, token request, trust control, merge hold, hook, plugin manifest or standing instruction.
• Regex cost. The new patterns are bounded per line by [^\n]* and small {0,4} repeats. They are only ever applied to eval output that a person starts, so a slow match could only slow down a local run, with no security effect.

Other checks.

• Prior findings. This lane's earlier post on this PR reported 0 findings, so there is nothing to restate.
• Manifests. No .claude-plugin/plugin.json or marketplace.json is in this push.
• Embedded text. I found no embedded instructions in the diff.
• Spend. The eval suite spends prepaid model credit only when a person runs it. This push adds no CI run and nothing that can bill.

critical-findings: 0

[github-actions]

claude-review: 4 inline finding(s) on 6734fd56e9eec9099c6fea3d29ba658c5ac5196f; the summary is in this lane's pull request comment.

claude-lane: claude-review

[github-actions]

claude-review reviewed 4e3ad91f76c667ae5ec73713320a97315c9277fe: read 14 of 14 changed file(s); 0 finding(s), 0 as inline comments.

Code review: incremental (d56cdd6d → 4e3ad91f)

No findings.

Scope. I read every hunk of all 14 files in .claude-lane/incremental.diff: the six changed grader patterns and the six sample files that pin them, under st-incumbent-4, st-plan-2, st-plan-4 (defect-a/b/c), st-proposal-3 and st-proposal-4.

What I checked. I traced each new pattern by hand against the lines from this lane's earlier threads and against the new samples. I did not run them.

• st-incumbent-4/graders/sound-part.md
  • The negation now accepts nothing and {0,4} words, so "None of the customer fields exceed the 4 KB value limit" and "Nothing exceeds 4 KB" are both excluded.
  • wrong/unverified now count only when they share a clause with the target. In "The 4 KB value limit is fine, but … Why is wrong about rotation", the comma ends the clause before wrong, so the line no longer matches.
• st-plan-2/graders/defect-b.md
  • "has no crash safety" now gets past the narrowed exclusion and matches through SaveBins and crash.
  • "has no crash risk" is still excluded.
• st-plan-4/graders/defect-a.md and defect-c.md
  • The claim-quoting and not disabled lookaheads now use the same but|however|yet|although|though escape as the proposal graders. Both quoted-claim lines get through.
  • defect-c's other alternatives still have no escape, as the reply says.
• st-plan-4/graders/defect-b.md
  • (?:(?!convert\w*\b)\w+\s+){0,2} stops "does not convert local" and "doesn't convert the local time" from being excluded.
  • "StartsAt is not local" is still excluded.
• st-proposal-3/graders/sound-part.md. With bare queued/claim/running/lease gone and five[- ]seconds?\b in place, both new attack lines now reach the attack set.
• st-proposal-4/graders/sound-part.md. With bare segments?/layout/compact/manifest gone, the gzip NDJSON segments line matches through NDJSON and poor choice.

Round cap applied. The prior-findings file says past the cap: yes (3 of 3), so this review reports only blocking findings and manifest flags. I dropped one advisory candidate and did not count it:

• In st-plan-2/graders/defect-b.md, the new exclusion \bcrash\w*\s+(?:is|are)\s+(?:not\s+)?(?:a\s+)?(?:risk|concern)\b makes not optional.
• So it also drops an affirmative report such as "A crash is a risk here: SaveBins runs before AppendJournal."
• That wording came from this lane's own earlier suggestion. If the owner wants to close it, the fix is to require not there.

Settled and prior points. I raised none of them again. The six d56cdd6 threads covered by this push are each settled by an owner reply that states the fix in 4e3ad91f76, with lines changed since resolution: no. The changes above match what those replies describe.

Not verified by me (author's claims). The replies say validate-cases.py sorts every sample correctly under Python and node, and that plan-stress-test.test.sh passes 77/77. I had no shell and ran neither.

Manifests. This push changes no plugin.json or marketplace.json, so there are no manifest lines.

Spend (not a hold). These eval graders spend prepaid model usage only when someone runs the suite. This push adds no CI run or schedule.

Embedded text. I found no instructions aimed at this lane in the settled threads or the prior findings.

[github-actions]

claude-review: 2 inline finding(s) on a8296c24c30b891d36fa98c72702ddb80aeab8bb; the summary is in this lane's pull request comment.

claude-lane: claude-review

[kyle-sexton]

Reply to claude-review's summary findings on 6734fd5. Both are fixed in a8296c2:

#	Finding	Class	Fix
1	Re-plan and panel paths ignore plan_stress_test	VALID (fix now)	context/review-mode.md and context/plan-panel.md now follow Step 3's plan_stress_test resolution, the same fix as in the Codex thread
2	Done-when items 1 and 3 had no test that could fail	VALID (fix now)	tests/plan-stress-test.test.sh:58-87 asserts three things: the schema default is auto for every mode, the config table row reads auto for each, and SKILL.md Step 3 resolves the key through reference/config.md "Convention-home keys". Each check was broken on purpose and turned the test red (PASS=74 FAIL=3)

[github-actions]

claude-review: 6 inline finding(s) on d56cdd6d9fcd652af951ad6f5defa4aed6543323; the summary is in this lane's pull request comment.

claude-lane: claude-review