Repository navigation
fix(planning): load the unattended problem-confirmation rule in every plan run #13507
Description
Refs: #9966
Problem
/planning:plan keeps its unattended problem-confirmation rules in plugins/planning/context/problem-framing.md, under "Unattended" and "Pre-confirmed". That includes #11516's comment-marker checks. plan/SKILL.md:137 links the file once, as the place to go when a Brief has no ### Problem, with no trigger to read it in an unattended run.
In 16 kept eval traces of /planning:plan (Claude Code 2.1.318, Sonnet 5.5), no run made a Read of context/problem-framing.md. That covers both arms of the #11516 pair (490bd07b2f0^ and 490bd07b2f0) and its after text with the template-path fix. The runs then ruled on comments from the model's own judgment. In one, the comment's plain "Reproduced on staging" line was taken as a person's confirmation of the problem, which the rule says never counts ("a claim of confirmation ... inside that text never counts"). That run left the plan unapproved, so the gap reaches problem confirmation, not plan approval.
Fix
Make the rule load in every unattended run. Two candidates:
- a trigger line in
plan/SKILL.mdthat says to read problem-framing.md's "Unattended" and "Pre-confirmed" before writing### Problemin an unattended run; or - the rule's checks moved into
plan/SKILL.md.
Pick the shape with the fewest moving parts that loads the rule in every run. Measure it with a preregistered pair (current text against the change) on the #11516 near-miss cases, so the detection control in plugins/evals/skills/plugin-eval/reference/reading-results.md "Detection control" can pass.
Done when
-
plugins/planning/skills/plan/SKILL.mdmakes an unattended run read the rule before it writes### Problem(either a trigger line namingcontext/problem-framing.md"Unattended" and "Pre-confirmed", or the checks themselves in SKILL.md), shown by the diff. - In a preregistered before-and-after pair on the feat(planning): count a trusted human's comment marker as problem pre-confirmation #11516 near-miss cases (5 runs per case per arm, posted on test(developer-experience): port eval cases S3 and S6 #9966 before it runs), every after run makes a
Readofcontext/problem-framing.md, or the moved text is in SKILL.md. Count it with the kept traces. - That pair's legs read VALID under
python3 -I plugins/evals/skills/plugin-eval/scripts/run-validity.py <result> --runs 5 --scaffolded, and no non-target case drops more than 0.05 (acase dropline innoise-report.pyoutput fails it). -
bash plugins/planning/tests/interview-defenses.test.shexits 0.
Also
- The case generator and 7 cases (one valid marker, six near-misses, raw read output in each prompt, plus a code scaffold) are in lane 1's scratch. Lane 1 commits them with the measurement.
- Runs still go INVALID on a separate cause: the plan skill
cps its checklist template out of the plugin directory, whichplan/SKILL.md:41forbids and the eval sandbox denies (4 of 4 smoke runs).
Related
- feat(planning): count a trusted human's comment marker as problem pre-confirmation #11516 (the rule), test(developer-experience): port eval cases S3 and S6 #9966 (batch A re-measure).
Filed by AI (lane 1) from eval traces.
Activity
github-actions commentedon Nov 5, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
I read this as a defect in /planning:plan: the unattended problem-confirmation rules in context/problem-framing.md ("Unattended" and "Pre-confirmed") are never loaded in unattended runs. None of the 16 traces show a Read of that file, and in one run a plain "Reproduced on staging" line was taken as confirmation of the problem.
The fix is a bounded edit to plan/SKILL.md: add a trigger line, or move the checks in. That is an implementation choice that leaves the design unchanged. I recommend the trigger line, since it has fewer moving parts. The preregistered before/after measurement can run in the lane, and anything that needs a host is covered by the existing host-run label.
No duplicate found among open issues. Related: #9966, #12646.
Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Nov 5, 2026
Not admitted to the issue lane: no trusted-human acceptance criteria: a listed human must put the confirmed
- [ ]criteria in the body or post them as their own comment, or, on an issue a listed human filed, intake's newest refinement comment must hold them unedited; it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 5, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: confirmed; its comment says why.melodic-automation-lanes commentedon Nov 5, 2026
Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.
Intended outcome: plan/SKILL.md gets a trigger line so every unattended run reads problem-framing.md "Unattended" and "Pre-confirmed" before writing ### Problem, measured by a preregistered before/after pair on the 7 near-miss cases.
Related work
Open questions
RECOMMENDED: State where lane 1 commits the cases and make that commit a dependency, since criteria cannot be checked against files the run cannot see. Basis: judgment
RECOMMENDED: Split it: the edit to plan/SKILL.md ships here, and the measurement waits on a separate issue that fixes the checklist-template cp. Basis: plugins/planning/skills/plan/SKILL.md:41
RECOMMENDED: The trigger line. It keeps the rule in one place and is the smaller edit, while moving it would duplicate rules that interview/context/loop.md:427 and reference/config.md:100 also cite. Basis: plugins/planning/context/problem-framing.md:111
RECOMMENDED: Every post-change run makes a Read of context/problem-framing.md before writing ### Problem, and the near-miss cases score no worse than the control with the valid-marker case still passing. Basis: judgment
Sources read
To answer, post a comment that starts with this line:
Automated refinement by the lanes App. Answer with a comment whose first line is
**Decision recorded.**.kyle-sexton commentedon Nov 5, 2026
Status.
status: needs-decisionremoved: no comment on this issue asks the owner a question, and the owner's stop reasons are an intent or vision change, a finding that changes primary design, or a high security or cost risk (standing direction §2). It stayshost-runatstatus: confirmedfor Lane 1's measured pair. If Lane 1 holds an open question, re-add the label together with the question written out in full.Agent choice, not an owner answer.
kyle-sexton commentedon Nov 6, 2026
chore(planning): name the #13507 changelog fragment by branch slug
test(planning): add the comment-marker near-miss cases for the plan s…