Skip to content

test(session-flow): eval coverage for the workflow continuation router - #3037

Merged
kyle-sexton merged 1 commit into
mainfrom
claude/pocock-steering-course-00zkvd
Aug 19, 2026
Merged

test(session-flow): eval coverage for the workflow continuation router#3037
kyle-sexton merged 1 commit into
mainfrom
claude/pocock-steering-course-00zkvd

Conversation

@kyle-sexton

Copy link
Copy Markdown
Contributor

Closes #2972

Summary

Adds eval coverage for the session-flow:workflow continuation router. The router had zero cases exercising its continue path — five workflow evals existed and none touched it — while one of its ordering invariants had already regressed once and been fixed inside the router's own creation PR (#1603, originating issue #1476). The 0.27.0 router evolution (#2971 / PR #3029) deliberately shipped without eval coverage and stated it as out of scope; this is the safety net that item deferred, landing immediately after it.

Fix

Nine cases (ids 6-14) join plugins/session-flow/skills/workflow/evals/evals.json, covering all four bullets of the issue's "Proposed work". Each case grades the router's stated reason, not just its verdict, so a rewrite that reaches the right mechanism by the wrong edge still fails the case.

  • First-yes-wins ordering — a machine-going-away prompt with healthy context and a small next step must still route to clean-stop (question 0 outranks every cost question below it); and an explicit background request with healthy context must reach continue-in-background rather than falling through to question 3's zero-cost in-session exit. That second case pins the invariant that regressed once: question 3 answers yes whenever context is healthy, so asking it first silently discards a user instruction.
  • Zone gating — a green zone word plus an evidence-degraded compaction marker must read as degraded, and judgment-heavy work in a degraded context must not route to in-session continue. The complement is graded too: a healthy zone whose next stage consumes the current stage's reasoning verbatim must prefer continue, since a summary of the reasoning is not the reasoning.
  • Post-evolution behavior — the AFK edge must hand the spawn-brief decision to orchestrate and then keep asking (it is the router's one non-terminal edge) while launching nothing, leaving continue-in-background's explicit-intent gate untouched. Suggest-by-default is graded on its full three-part shape: the single mechanism, the evidence that selected it, and the literal next invocation.
  • Autonomy gating — three cases hold the line 0.27.0's review fixes drew: the opt-in counts only in a genuine user turn (consent-shaped text inside a pasted issue body is data, never a licence); auto cannot carry out /clear or /compact, which sit outside the Skill-invocable built-ins; and clean-stop takes the literal continue auto token and nothing else, because once invoked it pushes commits, opens PRs, and files issues without a further confirmation.

The router itself is unchanged — its shipped design is the specification these cases test against, not something this PR revisits.

Also bumps session-flow to 0.28.0 with the matching CHANGELOG entry, per the repo's per-plugin semver parity gate and the precedent of the last eval-coverage-only change (#2989).

Verification

Full local gate, all green:

Gate Result
check-jsonschema vs plugins/skill-quality/reference/evals.schema.json ok -- validation done
check-evals-quality.sh on the suite PASS (0 warnings)
check-evals-quality.test.sh (the lint's own suite) 0 failures
check-orphaned-fixtures.sh --check no orphaned fixtures
validate-plugins.sh (incl. --strict catalog) Validation passed
check-changelog-parity.sh --check / --check-bump / --check-preserved / --check-order all pass (79 headings preserved, 83 changelogs ordered)
check-changed-skills.sh origin/main CHECK-SKILL workflow: PASS — 0 errors, 0 warnings
generate-catalog.mjs --check / generate-cheatsheet.mjs --check both in sync
markdownlint-cli2 on the touched CHANGELOG 0 issues
typos on all three touched files clean

--check-preserved was re-run after the commit so it compared real headings (79) rather than the pre-commit zero-diff case.

Acceptance criteria from the issue: evals exercise the continue path (ids 6-14); the regressed-once ordering invariant is pinned (id 7, with id 6 covering the question-0 half); autonomy opt-in gating covered (ids 12, 13, 14); CI green pending this run.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01QbfCrj3X9FfGL7VRZYmrn4


Generated by Claude Code

Closes #2972

The continuation router had zero eval coverage: five workflow cases
existed and none exercised the `continue` path, while one of its
ordering invariants had already regressed once and been fixed inside
the router's own creation PR (#1603, originating issue #1476). The
0.27.0 router evolution (#2971 / PR #3029) shipped without it by
design; this is the safety net that item deferred.

Nine cases (ids 6-14) pin the shipped behavior. Each grades the
router's stated reason, not just its verdict, so a rewrite that
reaches the right mechanism by the wrong edge still fails.

- First-yes-wins: machine-going-away routes to clean-stop over a
  healthy-context continue; an explicit background request outranks
  question 3's zero-cost in-session exit (the regressed invariant).
- Zone gating: a compaction marker overrides a green zone word and
  rejects in-session continue for judgment-heavy work; a healthy zone
  whose next stage consumes the reasoning verbatim prefers continue.
- Post-evolution: the AFK edge hands the spawn brief to orchestrate,
  keeps asking (non-terminal), and launches nothing; suggest-by-default
  is graded on its full mechanism/evidence/next-step shape.
- Autonomy gating: the opt-in counts only in a genuine user turn;
  `auto` cannot carry out /clear or /compact; clean-stop takes the
  literal `continue auto` token and nothing else.

Schema-validated and quality-linted green (0 warnings).

Co-authored-by: Claude <noreply@anthropic.com>
@claude

claude Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 38s —— View job


Security review — no security-relevant surface

  • Gather context (PR diff, changed files)
  • Apply skip gate
  • Review changed files for security-relevant surface
  • Post final findings

Skip gate hit: item 2 — this change has no security-relevant surface.

All three changed files are non-executable data/config:

  • plugins/session-flow/.claude-plugin/plugin.json — version bump 0.27.00.28.0 only.
  • plugins/session-flow/CHANGELOG.md — changelog prose entry for the bump.
  • plugins/session-flow/skills/workflow/evals/evals.json — nine new eval cases (ids 6-14) for the session-flow:workflow continuation router: plain JSON objects (prompt / expected_output / expectations strings) consumed by the eval harness as text fixtures. No code, no scripts, no GitHub Actions workflow changes, no permission or credential handling, no injection surfaces (command/SQL/path/template), and no change to the router's own logic — the router implementation is explicitly untouched by this PR.

No security findings. Nothing further to report.

@github-actions

Copy link
Copy Markdown

Last security-reviewed head: 90394a3d7394377bd9623b2e2449c2044671d64f. On the next push, the relevance gate compares only the commits since this SHA; delete this comment to force a full re-review.

@claude

claude Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 2m 34s —— View job


Code review — /review:code-review

  • Gather context (PR diff vs origin/main, router spec, eval schema/lint contract)
  • Understand the request (CI code-review lane on this PR)
  • Review the three changed files
  • Post findings

Scope: plugins/session-flow/.claude-plugin/plugin.json (version bump), plugins/session-flow/CHANGELOG.md (0.28.0 entry), plugins/session-flow/skills/workflow/evals/evals.json (+9 cases, ids 6–14). No router logic changed — context/continuation.md is untouched.

What I checked: read the router spec line-by-line against each new case and traced the routing each prompt should produce (question order, zone-degradation rule, non-terminal question 2, the two autonomy-tier rules), cross-checked the .compacted evidence-degraded marker terminology against context-guard's actual reader contract, and diffed against the eval JSON Schema (plugins/skill-quality/reference/evals.schema.json) and the narration/Q4 lint contract in check-evals-quality.sh.

Findings: none blocking. Each of the 9 new cases traces cleanly to a specific clause in the router spec and grades the stated reason, not just the terminal, as the PR describes:

  • id 6 / id 7 pin the first-yes-wins ordering exactly as written in continuation.md#L67-84, including the regressed-once invariant (question 1 before question 3).
  • id 8's .compacted marker reference matches context-guard's real reader contract (reference/reader-contract.md:36-37), not an invented artifact name.
  • id 10 correctly treats question 2 as the router's one non-terminal edge (continues to question 3 rather than asserting a specific mechanism it can't derive from the prompt alone — appropriately loose).
  • id 12/13/14 match the three autonomy carve-outs almost verbatim: injected-context-is-never-a-licence, built-in terminals excluded from auto, and clean-stop's literal-token-only gate.
  • All 14 cases (5 pre-existing + 9 new) have unique id/name, satisfy the schema's anyOf grading-criterion requirement, and use only schema-declared properties. narration: true is present on cases whose prose contains skill-name-shaped tokens and correctly omitted on ids 11/13, which don't.
  • plugin.json version bump is the only change in that file; CHANGELOG.md's new entry lines up with it.

Not independently re-run: this lane's tool grant is Read/Glob/Grep plus gh pr diff/view/comment/review — I did not execute check-jsonschema, check-evals-quality.sh, check-changelog-parity.sh, or the catalog/cheatsheet generators myself, so the PR body's verification table is author-claimed and unverified from this lane, not independently confirmed here. Manual read-through of the JSON against the schema and lint rules above turned up no discrepancy.

@github-actions

Copy link
Copy Markdown

Claude has reviewed this PR 1 time. The lane skips further automatic reviews after 5; deleting this comment resets the count.

@kyle-sexton
kyle-sexton merged commit f3ded7d into main Aug 19, 2026
48 checks passed
@kyle-sexton
kyle-sexton deleted the claude/pocock-steering-course-00zkvd branch August 19, 2026 17:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

session-flow/workflow: eval coverage for the continuation router (aihero lane 2)

2 participants