test(planning): eval cases for the interview's ship-it exit and bigger-than-you-thought trigger #7365

Activity

1. github-actions

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 13, 2026
2. github-actions commented on Oct 13, 2026

   

   github-actionsBot

   on Oct 13, 2026 – with GitHub Actions

   Contributor

   More actions

   Understood: this asks for outcome-graded eval cases covering two contract edges PR #7102 added to /planning:interview's routing — the mid-interview /planning:wayfind chart recommendation and the "ship it" exit — neither of which the interview or wayfind eval suites currently cover. The four listed cases (ship-it still running the Step 3/4 gates and deferring open rows, the one-round-vs-two-rounds wayfind trigger threshold, no-repeat-after-decline, and no self-switching) give a clear, bounded scope for the additions.

   Nothing appears missing to start the work; the brief is self-contained. No clear duplicates found in open issues.

   _{Automated triage. A maintainer reviews it.}
3. github-actions

   added

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   and removed

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 13, 2026
4. github-actions

   added the

   Task

   issue type on Oct 13, 2026
5. kyle-sexton commented on Oct 18, 2026

   

   kyle-sexton

   on Oct 18, 2026

   MemberAuthor

   More actions

   Agent Brief

   Type: Task
   Summary: Add outcome-graded eval cases for the interview's ship-it exit and its bigger-than-you-thought wayfind trigger.

   Current behavior:
   /planning:interview routes a small task to a ship-it exit and recommends /planning:wayfind chart mid-interview when rounds open more branches than they close. No eval case in the interview or wayfind suites covers either edge.

   Desired behavior:
   The interview eval suite holds cases that assert:

   • ship-it still runs the Step 3 confirmation gate and the Step 4 --brief cross-check, defers open rows, skips the assumption sweep, and lists the deferred rows in its handoff;
   • one round with more branches opened than closed does not recommend wayfind, and two such rounds running do;
   • after the user declines the wayfind recommendation, it is not repeated until the trigger escalates;
   • the model never switches to wayfind on its own.

   Key interfaces:

   • Interview eval suite cases, in the existing evals.json schema, each with expectations naming the behavior above.

   Acceptance criteria:

   • Four new cases exist, one per behavior above, each with expectations that name that behavior.
   • Each case's prompt and fixtures hold no raw session transcript.
   • scripts/check-changelog-fragments.sh passes.

   Out of scope:

   • Any edit to the interview SKILL.md text.
   • Running the suite or showing a case fails on main (paid eval run, after merge).
   • Wayfind skill behavior changes.

   ---

   This was generated by AI during backlog shaping. Held, not admitted: this brief is drafted and checked, but the issue lane would stop on it today. It is admitted, by adding agent-ready, once the lane can run the skill-quality check scripts (Automation Lanes' PR #7912). Held under the owner's 2026-10-18 lane-narrowing course correction.
6. kyle-sexton

   added

   needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

   agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

   work-class: mechanicalDeterministic, trivially reversible maintenance: dependency bumps, lint/format, sync.Deterministic, trivially reversible maintenance: dependency bumps, lint/format, sync.

   and removed

   needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

   on Oct 18, 2026
7. melodic-automation-lanes

   added a commit that references this issue on Oct 18, 2026

   test(planning): eval cases for the interview's ship-it exit and wayfi…

   ...

   e347ea5
8. kyle-sexton

   added a commit that references this issue on Oct 19, 2026

   test(planning): add case 35 for the Brief after a confirmed ship-it

   ...

   342905d
9. kyle-sexton

   closed this as completedin #8098on Oct 19, 2026