feat(planning): make questionnaire model-invoked - #3023
Conversation
Closes the one re-grade flip from course lane 8 (#2969). The fleet grade found `planning:questionnaire` was the single `disable-model-invocation: true` skill matching none of the rubric's three exception classes. Per the issue, the flip was gated on first looking for a latent rationale the grade could not see — a trigger collision with `planning:interview`. There is none: the two split on who holds the knowledge, and each description already routes to the other on that axis. Two costs of the hidden mode turned up instead, both now paid. Its trigger phrases were deliberately left unoptimized because a hidden skill's description is never matched against user text (planning 0.30.1), so it gains phrases a user would actually type. And its own description advertises a hand-off from an interview branch that the invocation-reach invariant made unreachable while it stayed hidden. Adds an eval case for what the flip enables: a natural-language request with no slash command routing here rather than to interview. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbfCrj3X9FfGL7VRZYmrn4
|
Claude finished @kyle-sexton's task in 2m 29s —— View job CI Code Review
Scope note: SummaryThis is a low-risk, well-scoped flip of
One nit posted inline: docs/conventions/invocation-mode/README.md:141 — a dropped verb (" No correctness bugs, no maintainability concerns, no test-coverage gaps found. |
|
Claude finished @kyle-sexton's task in 46s —— View job Security review —
|
|
Last security-reviewed head: |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d3ebb0bb83
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
Claude has reviewed this PR 3 times. The lane skips further automatic reviews after 5; deleting this comment resets the count. |
…te-away chain explicit Two review findings, both verified against the tree. The post-flip paragraph claimed 9 non-setup `true` skills, all class-attributed. A recount says 10: `session-flow:show-options` carries `true` and is absent from the grade table. It landed 2026-08-18, a day after the 2026-08-17 grade, so the table's population predates it and the ADR 0005 bound leaves it unswept rather than silently covered. The doc now records that gap instead of overclaiming, and grading it is filed as #3024. The route-away named `/planning:interview` as bare prose. That reads as advice to a human, which is the failure the rubric's cross-skill phrasing rule exists to prevent — and it only became live with this flip, since the model can now land here from a natural-language request and would strand the decision. It is now an explicit Skill-tool invocation, with the eval expectation updated to pin the mechanism rather than the recommendation. Also fixes a dropped verb in the same paragraph. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbfCrj3X9FfGL7VRZYmrn4
|
Claude finished @kyle-sexton's task in 47s —— View job Security review —
|
|
Claude finished @kyle-sexton's task in 1m 43s —— View job CI Code Review
SummaryThis review covers the latest commit,
One inline finding: the commit message for No correctness bugs, no maintainability concerns, no test-coverage gaps beyond the nit above. |
The clause relied on eliding "phrases" after "questionnaire's", which is grammatical but misparsed by two readers in a row as the noun "questionnaire's name" — so the sentence was ambiguous whatever its defence. It now names the subject outright rather than doubling the word into "name names". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbfCrj3X9FfGL7VRZYmrn4
|
Claude finished @kyle-sexton's task in 1m 56s —— View job CI Code Review
SummaryThis review covers the latest commit,
No correctness bugs, no maintainability concerns, no test-coverage gaps. Nothing further to flag — both previously open findings (bare Note: I did not independently execute the repo's check scripts ( |
2 participants
Closes #2969
Summary
The one re-grade flip from course lane 8 (#2910). The ADR 0005-bounded fleet grade found
planning:questionnairewas the singledisable-model-invocation: trueskill matching none of the rubric's three exception classes — no side effects beyond writing a Markdown document, not a setup skill, not maintainer-only — and no rationale for thetrueexisted in the skill body or the planning CHANGELOG.Fix
planning:interview(both plausibly firing on "I need to ask…"-shaped requests). There is none: the two split on who holds the knowledge, and each description already routes to the other on that axis —questionnairesays to run/planning:interviewwhen the user can answer themselves, whileinterview's phrases ("ask me questions first", "what do you need to know") are about interrogating the user andquestionnaire's name a third-party holder. So the flip proceeds rather than being recorded as a new exception class.disable-model-invocation: falseonplugins/planning/skills/questionnaire/SKILL.md.Use when:phrases unoptimized because a hidden skill's description is never matched against user text. Now that it is model-invoked, the description carries three phrases a user would actually type ("I don't know, that's the client's call", "send this to someone else to answer", "write up questions for our security team") beside the ones it already had. Every existing phrase is preserved verbatim (check-skill.shconfirms: "all 3 base-ref trigger phrase(s) preserved").model-routes-third-party-decision-here) pinning what the flip actually enables: a natural-language request with no slash command routing here rather than to/planning:interview, on the who-holds-the-knowledge axis, then interviewing the send only.false/ 58true, leaving 9 non-setuptrueskills, all class-attributed).planning0.30.7 → 0.31.0 with the matching CHANGELOG entry.A second cost the re-check surfaced
questionnaire's own description advertises a hand-off "when an interview branch defers to a person-arbiter", and the original changelog entry noted "the interview-side pull-out reference lands separately". It never landed — and while the skill wastrue, the invocation-reach invariant made it unreachable frominterviewanyway, so the advertised hand-off could not have worked. The flip unblocks it; wiring the interview-side pointer is deliberately out of scope here (it is its own deliverable, not part of #2969's acceptance criteria).Verification
bash scripts/check-changed-skills.sh origin/main— 1 skill checked, 0 failed; reportsinvocation mode: model-invoked (fleet default)and all 3 base-ref trigger phrases preserved. (One pre-existing WARN: the skill has no Gotchas surface — present before this change, untouched by it.)bash scripts/check-changed-skills.test.sh— PASS=13 FAIL=0.false/ 58trueacross 220 top-level skills.check-jsonschema+check-evals-quality.shon the eval suite — ok / PASS, 0 warnings.bash scripts/validate-plugins.sh— all manifests + catalog validated;generate-catalog.mjs --checkandgenerate-cheatsheet.mjs --checkin sync.bash scripts/check-changelog-parity.sh --check/--check-bump origin/main/--check-order— all green.npx markdownlint-cli2on all changed markdown — 0 issues;typosclean.Related
docs/upstream/aihero-course.mdlane 8 rows🤖 Generated with Claude Code
https://claude.ai/code/session_01QbfCrj3X9FfGL7VRZYmrn4
Generated by Claude Code