test(session-flow): eval coverage for the workflow continuation router - #3037
Conversation
Closes #2972 The continuation router had zero eval coverage: five workflow cases existed and none exercised the `continue` path, while one of its ordering invariants had already regressed once and been fixed inside the router's own creation PR (#1603, originating issue #1476). The 0.27.0 router evolution (#2971 / PR #3029) shipped without it by design; this is the safety net that item deferred. Nine cases (ids 6-14) pin the shipped behavior. Each grades the router's stated reason, not just its verdict, so a rewrite that reaches the right mechanism by the wrong edge still fails. - First-yes-wins: machine-going-away routes to clean-stop over a healthy-context continue; an explicit background request outranks question 3's zero-cost in-session exit (the regressed invariant). - Zone gating: a compaction marker overrides a green zone word and rejects in-session continue for judgment-heavy work; a healthy zone whose next stage consumes the reasoning verbatim prefers continue. - Post-evolution: the AFK edge hands the spawn brief to orchestrate, keeps asking (non-terminal), and launches nothing; suggest-by-default is graded on its full mechanism/evidence/next-step shape. - Autonomy gating: the opt-in counts only in a genuine user turn; `auto` cannot carry out /clear or /compact; clean-stop takes the literal `continue auto` token and nothing else. Schema-validated and quality-linted green (0 warnings). Co-authored-by: Claude <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 38s —— View job Security review — no security-relevant surface
Skip gate hit: item 2 — this change has no security-relevant surface. All three changed files are non-executable data/config:
No security findings. Nothing further to report. |
|
Last security-reviewed head: |
|
Claude finished @kyle-sexton's task in 2m 34s —— View job Code review —
|
|
Claude has reviewed this PR 1 time. The lane skips further automatic reviews after 5; deleting this comment resets the count. |
Closes #2972
Summary
Adds eval coverage for the
session-flow:workflowcontinuation router. The router had zero cases exercising itscontinuepath — five workflow evals existed and none touched it — while one of its ordering invariants had already regressed once and been fixed inside the router's own creation PR (#1603, originating issue #1476). The 0.27.0 router evolution (#2971 / PR #3029) deliberately shipped without eval coverage and stated it as out of scope; this is the safety net that item deferred, landing immediately after it.Fix
Nine cases (ids 6-14) join
plugins/session-flow/skills/workflow/evals/evals.json, covering all four bullets of the issue's "Proposed work". Each case grades the router's stated reason, not just its verdict, so a rewrite that reaches the right mechanism by the wrong edge still fails the case.clean-stop(question 0 outranks every cost question below it); and an explicit background request with healthy context must reachcontinue-in-backgroundrather than falling through to question 3's zero-cost in-session exit. That second case pins the invariant that regressed once: question 3 answers yes whenever context is healthy, so asking it first silently discards a user instruction.orchestrateand then keep asking (it is the router's one non-terminal edge) while launching nothing, leavingcontinue-in-background's explicit-intent gate untouched. Suggest-by-default is graded on its full three-part shape: the single mechanism, the evidence that selected it, and the literal next invocation.autocannot carry out/clearor/compact, which sit outside theSkill-invocable built-ins; andclean-stoptakes the literalcontinue autotoken and nothing else, because once invoked it pushes commits, opens PRs, and files issues without a further confirmation.The router itself is unchanged — its shipped design is the specification these cases test against, not something this PR revisits.
Also bumps session-flow to 0.28.0 with the matching CHANGELOG entry, per the repo's per-plugin semver parity gate and the precedent of the last eval-coverage-only change (#2989).
Verification
Full local gate, all green:
check-jsonschemavsplugins/skill-quality/reference/evals.schema.jsonok -- validation donecheck-evals-quality.shon the suitePASS (0 warnings)check-evals-quality.test.sh(the lint's own suite)0 failurescheck-orphaned-fixtures.sh --checkvalidate-plugins.sh(incl.--strictcatalog)Validation passedcheck-changelog-parity.sh --check/--check-bump/--check-preserved/--check-ordercheck-changed-skills.sh origin/mainCHECK-SKILL workflow: PASS — 0 errors, 0 warningsgenerate-catalog.mjs --check/generate-cheatsheet.mjs --checkmarkdownlint-cli2on the touched CHANGELOG0 issuestyposon all three touched files--check-preservedwas re-run after the commit so it compared real headings (79) rather than the pre-commit zero-diff case.Acceptance criteria from the issue: evals exercise the
continuepath (ids 6-14); the regressed-once ordering invariant is pinned (id 7, with id 6 covering the question-0 half); autonomy opt-in gating covered (ids 12, 13, 14); CI green pending this run.Related
REcompile() paniclines incheck-skill.shoutput are that known mawk artifact, not findings)🤖 Generated with Claude Code
https://claude.ai/code/session_01QbfCrj3X9FfGL7VRZYmrn4
Generated by Claude Code