From 4fbf51ef8ab7bb9494a6698a8b3830d751de2869 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 25 Jul 2026 13:10:03 -0400 Subject: [PATCH 1/6] docs(work-items): require brief-before-ask restatement in interactive direction gates triage's interactive direction gate and attend-queue's row-working loop could pose an operator decision question with option labels only, forcing the operator to halt the pass and ask which item is even under discussion. Both now require restating (1) which item, (2) the decision being asked, and (3) the consequence of each option before any operator-facing decision question - triage's recommendation gate and each step-4 interview question, and attend-queue's [intake]/[escalated]/[ratify] rows. Closes #1202 Co-Authored-By: Claude Sonnet 5 (Claude Code) --- plugins/work-items/.claude-plugin/plugin.json | 2 +- plugins/work-items/CHANGELOG.md | 12 +++++++ .../work-items/skills/attend-queue/SKILL.md | 31 ++++++++++++------- .../skills/attend-queue/evals/evals.json | 13 ++++++++ plugins/work-items/skills/triage/SKILL.md | 4 +-- .../work-items/skills/triage/evals/evals.json | 14 +++++++++ 6 files changed, 62 insertions(+), 14 deletions(-) diff --git a/plugins/work-items/.claude-plugin/plugin.json b/plugins/work-items/.claude-plugin/plugin.json index 24854ffb70..49a8059691 100644 --- a/plugins/work-items/.claude-plugin/plugin.json +++ b/plugins/work-items/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "work-items", - "version": "0.24.2", + "version": "0.25.0", "description": "Manages development work items through a provider-neutral tracker seam that ships with the plugin (bundled dispatcher plus github and local-markdown adapters; seam plugin-dir canonical, adapters consumer-local-first): dashboard, taxonomy-labeled creation, a race-safe assignee-plus-lease claim protocol, recurring-schedule checks, TODO scanning, stale-lease auditing, plan decomposition into vertical-slice items, raw-intake triage (issues and unsolicited PRs through raw, verified, briefed, autonomous-eligible states), plus the two work-items loop lanes of the loop-lane convention: a self-paced autonomous work-loop drain (work-class admission gate, adaptive item cap, PR-only) and an attended attend-queue escalation lane. The re-runnable setup skill binds the provider (.work-item-tracker.json), seeds the recurring-schedule seam (.github/recurring-schedule.json), and remaps canonical role labels.", "author": { "name": "Melodic Software", diff --git a/plugins/work-items/CHANGELOG.md b/plugins/work-items/CHANGELOG.md index a9cd13387c..9ef40a1727 100644 --- a/plugins/work-items/CHANGELOG.md +++ b/plugins/work-items/CHANGELOG.md @@ -3,6 +3,18 @@ All notable changes to the `work-items` plugin are documented here. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this plugin uses semantic versioning. +## [0.25.0] + +### Added + +- **Brief-before-ask requirement in the interactive gates (`#1202`).** `triage`'s interactive + direction gate (both the initial recommendation and each step-4 interview question) and + `attend-queue`'s row-working loop (`[intake]`, `[escalated]`, `[ratify]` rows) now require + restating, before any operator-facing decision question, (1) which item, (2) the decision being + asked, and (3) the consequence of each option. Previously an operator could be asked to decide + with only option labels and no restated item context, forcing them to halt the pass and ask + "what issue are you looking at." + ## [0.24.2] ### Fixed diff --git a/plugins/work-items/skills/attend-queue/SKILL.md b/plugins/work-items/skills/attend-queue/SKILL.md index a52bfb9600..29ac7e807d 100644 --- a/plugins/work-items/skills/attend-queue/SKILL.md +++ b/plugins/work-items/skills/attend-queue/SKILL.md @@ -65,18 +65,27 @@ ratifications unblock the waiting worker loop). ## Working the queue +**Brief before asking.** This lane works rows across many items in one pass, so the operator's +context from the previous row never carries over. Before any operator-facing decision question in +this loop — an `[intake]` recommendation, an `[escalated]` question, or a `[ratify]` prompt — +restate (1) which item (number + one-line title), (2) the decision being asked, and (3) the +consequence of each option, then ask. A terse output style must never compress this restatement +away; the row's context is precisely what the operator needs to answer without stopping the pass to +ask which item is in front of them. + - **`[intake]` rows** — run `/work-items:triage `. The operator is present, so triage's - **interactive** direction gate applies: recommend, wait for direction, then mutate. All triage - machinery (states, outcomes, briefs, closing invariant) is owned there. -- **`[escalated]` rows** — read the machine-marked comment for the escalated question, then drive - it to a decision with `/planning:interview` (when the `planning` plugin is installed; otherwise - ask the focused questions inline, one at a time, most load-bearing first — the same fallback - shape triage's interview step uses). **Write the answer back as an issue comment** on the item — - the decision lives on the tracker, never only in the session — replying in the thread of the - escalation comment where the provider supports it. -- **`[ratify]` rows** — present the classification and the intended dispatch from the marker - comment. On operator ratification, record it as a reply comment and flip the item per the rule - below; on decline, leave it human-gated and record the rationale as a comment. + **interactive** direction gate applies: brief before asking, recommend, wait for direction, then + mutate. All triage machinery (states, outcomes, briefs, closing invariant) is owned there. +- **`[escalated]` rows** — read the machine-marked comment for the escalated question, restate the + brief above, then drive it to a decision with `/planning:interview` (when the `planning` plugin is + installed; otherwise ask the focused questions inline, one at a time, most load-bearing first — the + same fallback shape triage's interview step uses). **Write the answer back as an issue comment** on + the item — the decision lives on the tracker, never only in the session — replying in the thread of + the escalation comment where the provider supports it. +- **`[ratify]` rows** — restate the brief above, then present the classification and the intended + dispatch from the marker comment and the consequence of ratifying versus declining. On operator + ratification, record it as a reply comment and flip the item per the rule below; on decline, leave + it human-gated and record the rationale as a comment. - **Flip to agent-ready.** When an answer or ratification removes the human blocker, apply the autonomous-eligible role label and remove the human-gated role label **in the same edit** (both resolved from `config.role_labels`, never literals) — an item wearing both roles is a diff --git a/plugins/work-items/skills/attend-queue/evals/evals.json b/plugins/work-items/skills/attend-queue/evals/evals.json index 1ee6859dc3..85b3617f2b 100644 --- a/plugins/work-items/skills/attend-queue/evals/evals.json +++ b/plugins/work-items/skills/attend-queue/evals/evals.json @@ -30,6 +30,19 @@ "Prefixes agent-authored comments with the AI disclaimer", "Never merges a PR and never executes a work item from this lane" ] + }, + { + "id": 3, + "name": "attend-queue-briefs-before-asking-each-row", + "prompt": "/work-items:attend-queue\n\nThe operator works two rows back to back in the same pass: an [escalated] item asking which of two retry strategies to keep, then a [ratify] row for a C3 admission on a completely different item.", + "expected_output": "Before posing the escalated question, restates which item it concerns (number + one-line title), the decision being asked (which retry strategy to keep), and the consequence of each option. Moving to the very next row, the same restatement happens again for the ratify prompt — the operator's context from the escalated row is never assumed to carry over, even though it was the immediately preceding row in the same pass. Neither restatement is dropped or abbreviated for brevity.", + "files": [], + "expectations": [ + "Restates item identity, the decision being asked, and the consequence of each option before posing the escalated question", + "Repeats the same full restatement for the very next row (the ratify prompt) rather than assuming context carries over from the prior row in the same pass", + "Does not compress or drop the restatement for either row even under a terse output style", + "Still performs the row's underlying mechanics (interview/write-back for escalated, ratify-or-decline recording for ratify) after the restatement — the brief precedes the ask, it does not replace it" + ] } ] } diff --git a/plugins/work-items/skills/triage/SKILL.md b/plugins/work-items/skills/triage/SKILL.md index e2248d5554..c31e9f19c4 100644 --- a/plugins/work-items/skills/triage/SKILL.md +++ b/plugins/work-items/skills/triage/SKILL.md @@ -100,7 +100,7 @@ Classify **bug vs enhancement** first — it steers the rest of the flow (bugs g **Direction gate.** Recommending is read-only; the gate governs *mutation* — labels, comments, closes, item creation — and which side of it you are on is fixed by how triage was invoked: -- **Interactive session** — a human operator is present and no standing lane rules were supplied. Present the recommendation and **wait for the user's explicit direction** before mutating anything. This is the default whenever the invocation carries no autonomous mandate. +- **Interactive session** — a human operator is present and no standing lane rules were supplied. **Brief before asking**: before presenting the recommendation, restate (1) which item (number + one-line title), (2) the decision being asked, and (3) the consequence of each option — then present the recommendation and **wait for the user's explicit direction** before mutating anything. This is the default whenever the invocation carries no autonomous mandate. The restatement is not optional compression fodder: a terse output style must never drop it, and it applies on every decision question, not only the first one of a pass — the operator working several rows in sequence (e.g. via `/work-items:attend-queue`) cannot be assumed to still be holding a prior item's context. - **Autonomous lane** — triage is running unattended as a `/loop` or `/schedule` AFK session whose standing rules — the directive supplied with its `/loop` / `/schedule` invocation — already authorize triage mutations. Those standing rules **are** the direction this gate requires: treat the gate as satisfied and proceed through verification and outcome without a human turn, prefixing every comment and item you create with the AI disclaimer. There is no operator turn to wait for, so blocking here would deadlock the lane — the gate is met by the lane's mandate, not skipped. The autonomous branch is the mode the AI disclaimer already anticipates: a session that mutates without a human turn. The two are one mode, not a contradiction. Formalizing this as the autonomous-mode contract — codifying that standing-lane rules constitute direction — is tracked in #459. @@ -116,7 +116,7 @@ Report the result: confirmed (with the observed behavior / code path — the ite ### 4. Interview (if needed) -Only after verification (or for enhancements, where the open question is scope, not fact): when the description is vague or missing acceptance criteria, ask focused questions one at a time — resolve the most load-bearing ambiguity first. Post questions as item comments. Mark `status:needs-info` until the reporter responds. +Only after verification (or for enhancements, where the open question is scope, not fact): when the description is vague or missing acceptance criteria, ask focused questions one at a time — resolve the most load-bearing ambiguity first. Each question is a decision question and carries the same brief-before-ask restatement as the direction gate above: which item it concerns, the decision being asked, and the consequence of each answer. Post questions as item comments. Mark `status:needs-info` until the reporter responds. ### 5. Apply outcome diff --git a/plugins/work-items/skills/triage/evals/evals.json b/plugins/work-items/skills/triage/evals/evals.json index 50c2df4598..55303b674d 100644 --- a/plugins/work-items/skills/triage/evals/evals.json +++ b/plugins/work-items/skills/triage/evals/evals.json @@ -80,6 +80,20 @@ "Prefixes every comment and item it creates with the AI disclaimer during the autonomous session", "Routes the label/comment writes through the bound adapter's mechanics and any item creation through the seam `create-item` — no inline provider commands" ] + }, + { + "id": 7, + "name": "triage-briefs-before-asking-in-interactive-session", + "prompt": "/work-items:triage 64 — running interactively with a human operator present; #64 is an enhancement request whose description is vague on scope, and you have just recommended a target state and need to ask a follow-up scoping question", + "expected_output": "Before presenting the recommendation and before posting the follow-up scoping question, restates which item is under discussion (number + one-line title), the decision being asked, and the consequence of each option/answer — never assumes the operator already holds this context, even mid-pass. Only after that restatement does it present the recommendation and wait for explicit direction, or post the scoping question and mark status:needs-info.", + "files": [], + "expectations": [ + "Restates the item identity (number + one-line title) before presenting the recommendation, not after", + "States the decision being asked and the consequence of each option before asking for direction", + "Applies the same brief-before-ask restatement to the follow-up interview question in step 4, not only to the initial recommendation", + "Does not compress away the restatement even though the session is running under a terse output style", + "Still waits for the operator's explicit direction before mutating anything — the restatement precedes the question, it does not replace the wait" + ] } ] } From 64a146f1eb63d0bfbbc9ee7c705beeb8a9918ff9 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 25 Jul 2026 18:53:31 -0400 Subject: [PATCH 2/6] fix(work-items): scope brief-before-ask consequences to presented options Requiring "the consequence of each answer" was unsatisfiable for an open-ended scoping question: the answer set is unbounded, so a compliant agent had to either invent a closed option list or drop part of the restatement. Consequences are now scoped to the options actually presented, with an explicit rule that an open-ended question states what the answer will determine instead - and never gets narrowed to satisfy the restatement. Eval 7's scenario also began after the recommendation had been presented while its expectations required a restatement before it, so a model could satisfy it by replaying a completed step. The scenario now starts before the recommendation, which is the behavior the fixture exists to cover. --- plugins/work-items/CHANGELOG.md | 8 +++++--- plugins/work-items/skills/attend-queue/SKILL.md | 8 +++++--- plugins/work-items/skills/triage/SKILL.md | 2 +- plugins/work-items/skills/triage/evals/evals.json | 7 ++++--- 4 files changed, 15 insertions(+), 10 deletions(-) diff --git a/plugins/work-items/CHANGELOG.md b/plugins/work-items/CHANGELOG.md index 8b6019e388..e063d855a9 100644 --- a/plugins/work-items/CHANGELOG.md +++ b/plugins/work-items/CHANGELOG.md @@ -11,9 +11,11 @@ All notable changes to the `work-items` plugin are documented here. Format follo direction gate (both the initial recommendation and each step-4 interview question) and `attend-queue`'s row-working loop (`[intake]`, `[escalated]`, `[ratify]` rows) now require restating, before any operator-facing decision question, (1) which item, (2) the decision being - asked, and (3) the consequence of each option. Previously an operator could be asked to decide - with only option labels and no restated item context, forcing them to halt the pass and ask - "what issue are you looking at." + asked, and (3) the consequence of each option **presented** — an open-ended question, which has no + option set to enumerate, states what the answer will determine instead of being narrowed into a + closed list to satisfy the restatement. Previously an operator could be asked to decide with only + option labels and no restated item context, forcing them to halt the pass and ask "what issue are + you looking at." ## [0.24.6] diff --git a/plugins/work-items/skills/attend-queue/SKILL.md b/plugins/work-items/skills/attend-queue/SKILL.md index 29ac7e807d..b665cc2efa 100644 --- a/plugins/work-items/skills/attend-queue/SKILL.md +++ b/plugins/work-items/skills/attend-queue/SKILL.md @@ -69,9 +69,11 @@ ratifications unblock the waiting worker loop). context from the previous row never carries over. Before any operator-facing decision question in this loop — an `[intake]` recommendation, an `[escalated]` question, or a `[ratify]` prompt — restate (1) which item (number + one-line title), (2) the decision being asked, and (3) the -consequence of each option, then ask. A terse output style must never compress this restatement -away; the row's context is precisely what the operator needs to answer without stopping the pass to -ask which item is in front of them. +consequence of each option **you present**, then ask. An open-ended question presents no option set +to enumerate consequences for — state instead what the answer will determine, and never narrow a +genuinely open question into a closed list just to satisfy the restatement. A terse output style +must never compress this restatement away; the row's context is precisely what the operator needs to +answer without stopping the pass to ask which item is in front of them. - **`[intake]` rows** — run `/work-items:triage `. The operator is present, so triage's **interactive** direction gate applies: brief before asking, recommend, wait for direction, then diff --git a/plugins/work-items/skills/triage/SKILL.md b/plugins/work-items/skills/triage/SKILL.md index 4fee259a87..d51015e3b8 100644 --- a/plugins/work-items/skills/triage/SKILL.md +++ b/plugins/work-items/skills/triage/SKILL.md @@ -116,7 +116,7 @@ Report the result: confirmed (with the observed behavior / code path — the ite ### 4. Interview (if needed) -Only after verification (or for enhancements, where the open question is scope, not fact): when the description is vague or missing acceptance criteria, ask focused questions one at a time — resolve the most load-bearing ambiguity first. Each question is a decision question and carries the same brief-before-ask restatement as the direction gate above: which item it concerns, the decision being asked, and the consequence of each answer. Post questions as item comments. Mark `status:needs-info` until the reporter responds. +Only after verification (or for enhancements, where the open question is scope, not fact): when the description is vague or missing acceptance criteria, ask focused questions one at a time — resolve the most load-bearing ambiguity first. Each question is a decision question and carries the same brief-before-ask restatement as the direction gate above: which item it concerns, the decision being asked, and the consequence of each option **you present**. An open-ended question presents no option set to enumerate consequences for — state instead what the answer will determine, and never narrow a genuinely open question into a closed list just to satisfy the restatement. Post questions as item comments. Mark `status:needs-info` until the reporter responds. ### 5. Apply outcome diff --git a/plugins/work-items/skills/triage/evals/evals.json b/plugins/work-items/skills/triage/evals/evals.json index 55303b674d..91071e2b79 100644 --- a/plugins/work-items/skills/triage/evals/evals.json +++ b/plugins/work-items/skills/triage/evals/evals.json @@ -84,13 +84,14 @@ { "id": 7, "name": "triage-briefs-before-asking-in-interactive-session", - "prompt": "/work-items:triage 64 — running interactively with a human operator present; #64 is an enhancement request whose description is vague on scope, and you have just recommended a target state and need to ask a follow-up scoping question", - "expected_output": "Before presenting the recommendation and before posting the follow-up scoping question, restates which item is under discussion (number + one-line title), the decision being asked, and the consequence of each option/answer — never assumes the operator already holds this context, even mid-pass. Only after that restatement does it present the recommendation and wait for explicit direction, or post the scoping question and mark status:needs-info.", + "prompt": "/work-items:triage 64 — running interactively with a human operator present. #64 is an enhancement request whose description is vague on scope. You have finished categorizing it and are about to present your target-state recommendation; a follow-up scoping question will be needed after the operator directs you.", + "expected_output": "Before presenting the recommendation, and again before posting the follow-up scoping question, restates which item is under discussion (number + one-line title), the decision being asked, and the consequence of each option it presents (or, for an open-ended scoping question, what the answer will determine) — never assumes the operator already holds this context, even mid-pass. Only after that restatement does it present the recommendation and wait for explicit direction, or post the scoping question and mark status:needs-info.", "files": [], "expectations": [ "Restates the item identity (number + one-line title) before presenting the recommendation, not after", - "States the decision being asked and the consequence of each option before asking for direction", + "States the decision being asked and the consequence of each presented option before asking for direction", "Applies the same brief-before-ask restatement to the follow-up interview question in step 4, not only to the initial recommendation", + "For an open-ended scoping question, states what the answer will determine instead of inventing a closed option set to enumerate consequences for", "Does not compress away the restatement even though the session is running under a terse output style", "Still waits for the operator's explicit direction before mutating anything — the restatement precedes the question, it does not replace the wait" ] From 9794655a6fa59ec464e5a0fcdc37daae12027202 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 25 Jul 2026 19:00:31 -0400 Subject: [PATCH 3/6] test(work-items): ground the brief-before-ask fixtures in reachable scenarios The triage fixture asked one turn to demonstrate both the pre-recommendation brief and the step-4 interview brief, but a correct implementation stops at the direction gate and never reaches step 4, so expectations 3-4 were unreachable. Split into eval 7 (pre-recommendation brief, ends at the gate) and eval 8 (post-direction interview question, which is where the open-ended carve-out is actually exercised). The attend-queue fixture named no item numbers, titles, options, or intended dispatch, so a model had to invent the very context the expectations require it to restate - a hallucinated brief scored the same as a grounded one. Both rows now carry concrete markers, and a new expectation forbids inventing detail the markers do not supply. --- .../skills/attend-queue/evals/evals.json | 9 ++++---- .../work-items/skills/triage/evals/evals.json | 21 ++++++++++++++----- 2 files changed, 21 insertions(+), 9 deletions(-) diff --git a/plugins/work-items/skills/attend-queue/evals/evals.json b/plugins/work-items/skills/attend-queue/evals/evals.json index 85b3617f2b..d16db14248 100644 --- a/plugins/work-items/skills/attend-queue/evals/evals.json +++ b/plugins/work-items/skills/attend-queue/evals/evals.json @@ -34,12 +34,13 @@ { "id": 3, "name": "attend-queue-briefs-before-asking-each-row", - "prompt": "/work-items:attend-queue\n\nThe operator works two rows back to back in the same pass: an [escalated] item asking which of two retry strategies to keep, then a [ratify] row for a C3 admission on a completely different item.", - "expected_output": "Before posing the escalated question, restates which item it concerns (number + one-line title), the decision being asked (which retry strategy to keep), and the consequence of each option. Moving to the very next row, the same restatement happens again for the ratify prompt — the operator's context from the escalated row is never assumed to carry over, even though it was the immediately preceding row in the same pass. Neither restatement is dropped or abbreviated for brevity.", + "prompt": "/work-items:attend-queue\n\nThe operator works two rows back to back in the same pass.\n\nRow 1 — [escalated] #418 \"Uploader retries transient S3 5xx forever\". The worker's escalation marker asks which retry strategy to keep: (a) capped exponential backoff, 5 attempts then dead-letter — bounded worst-case latency, but a long outage drops uploads to the dead-letter queue; (b) unbounded backoff with a 30-minute ceiling — no upload is ever dropped, but a sustained outage holds worker slots indefinitely.\n\nRow 2 — [ratify] #503 \"Digest CLI crashes on an empty transcript file\". The marker classifies it C3 bug-fix-shaped and states the intended dispatch: the worker loop picks it up autonomously on the next cycle and opens a PR. Ratifying admits it to the autonomous lane; declining leaves it human-gated with the rationale recorded.", + "expected_output": "Before posing the escalated question, restates which item it concerns (#418 plus its one-line title), the decision being asked (which retry strategy to keep), and the consequence of each of the two presented options as the marker states them. Moving to the very next row, the same restatement happens again for the ratify prompt — #503 plus its title, that the decision is whether to ratify the C3 admission, and what ratifying versus declining causes — the operator's context from the escalated row is never assumed to carry over, even though it was the immediately preceding row in the same pass. Neither restatement is dropped or abbreviated for brevity, and neither invents detail the markers do not supply.", "files": [], "expectations": [ - "Restates item identity, the decision being asked, and the consequence of each option before posing the escalated question", - "Repeats the same full restatement for the very next row (the ratify prompt) rather than assuming context carries over from the prior row in the same pass", + "Restates item identity (#418 + title), the decision being asked, and the consequence of each option before posing the escalated question", + "Restates only the consequences the escalation and ratify markers actually supply — no invented options, outcomes, or placeholder text", + "Repeats the same full restatement for the very next row (#503, the ratify prompt, including the intended dispatch and the ratify-versus-decline consequence) rather than assuming context carries over from the prior row in the same pass", "Does not compress or drop the restatement for either row even under a terse output style", "Still performs the row's underlying mechanics (interview/write-back for escalated, ratify-or-decline recording for ratify) after the restatement — the brief precedes the ask, it does not replace it" ] diff --git a/plugins/work-items/skills/triage/evals/evals.json b/plugins/work-items/skills/triage/evals/evals.json index 91071e2b79..94b4dc5eb5 100644 --- a/plugins/work-items/skills/triage/evals/evals.json +++ b/plugins/work-items/skills/triage/evals/evals.json @@ -83,18 +83,29 @@ }, { "id": 7, - "name": "triage-briefs-before-asking-in-interactive-session", - "prompt": "/work-items:triage 64 — running interactively with a human operator present. #64 is an enhancement request whose description is vague on scope. You have finished categorizing it and are about to present your target-state recommendation; a follow-up scoping question will be needed after the operator directs you.", - "expected_output": "Before presenting the recommendation, and again before posting the follow-up scoping question, restates which item is under discussion (number + one-line title), the decision being asked, and the consequence of each option it presents (or, for an open-ended scoping question, what the answer will determine) — never assumes the operator already holds this context, even mid-pass. Only after that restatement does it present the recommendation and wait for explicit direction, or post the scoping question and mark status:needs-info.", + "name": "triage-briefs-before-presenting-the-recommendation", + "prompt": "/work-items:triage 64 — running interactively with a human operator present. #64 is titled \"Export job silently skips rows with a null tenant id\" and is filed as an enhancement request whose description is vague on scope. You have finished verifying and categorizing it; your assessment is that the target state is needs-info (the scope is undecided), with briefed-as-delegable the alternative if the operator would rather you pick a scope yourself.", + "expected_output": "Restates which item is under discussion (#64 plus its one-line title), the decision being asked (which target state to apply), and the consequence of each presented option — needs-info parks the item pending the reporter, briefed-as-delegable admits it to the delegable lane on an agent-chosen scope — and only then presents the recommendation and waits for the operator's explicit direction. Nothing is mutated before that direction arrives.", "files": [], "expectations": [ "Restates the item identity (number + one-line title) before presenting the recommendation, not after", "States the decision being asked and the consequence of each presented option before asking for direction", - "Applies the same brief-before-ask restatement to the follow-up interview question in step 4, not only to the initial recommendation", - "For an open-ended scoping question, states what the answer will determine instead of inventing a closed option set to enumerate consequences for", "Does not compress away the restatement even though the session is running under a terse output style", "Still waits for the operator's explicit direction before mutating anything — the restatement precedes the question, it does not replace the wait" ] + }, + { + "id": 8, + "name": "triage-briefs-before-each-interview-question", + "prompt": "/work-items:triage 64 — running interactively with a human operator present. #64 is titled \"Export job silently skips rows with a null tenant id\". You already briefed and presented your recommendation and the operator has replied: \"go ahead, park it as needs-info and ask the reporter what you need.\" The scope is still undecided, so a follow-up scoping question for the reporter is the next step.", + "expected_output": "Even though the operator has already been briefed once this pass and has just given direction, the follow-up scoping question carries its own restatement: which item it concerns (#64 plus its title), the decision the question resolves, and — because \"what scope do you actually need?\" is open-ended rather than a closed option set — what the answer will determine, rather than a manufactured list of options to enumerate consequences for. The question is posted as an item comment and the item is marked status:needs-info.", + "files": [], + "expectations": [ + "Restates the item identity and the decision the question resolves before posting the follow-up scoping question, not only before the earlier recommendation", + "For the open-ended scoping question, states what the answer will determine instead of inventing a closed option set to enumerate consequences for", + "Does not treat the earlier brief in the same pass as license to skip this one", + "Posts the question as an item comment and marks status:needs-info rather than answering the scope question itself" + ] } ] } From a4925b1488a11dbe1e10516a3c27d76ada04ac4c Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 25 Jul 2026 19:07:39 -0400 Subject: [PATCH 4/6] test(work-items): put both brief-before-ask fixtures at reachable workflow positions Triage eval 7 claimed verification was already finished while the recommendation had not yet been presented, but the direction gate is step 2 and verification is step 3 - no compliant run reaches that state. The prompt now places the run at the step-2 gate, before verification. Eval 8 likewise needed the enhancement framing that makes step 4 reachable without verification, so it now says so. Attend-queue eval 3 supplied both markers but never the operator's answer to the first row, so a compliant run had to stop at the wait and could never reach the second row. The operator's answer to #418 is now part of the scenario, with an expectation that the wait is honored rather than self-answered. --- plugins/work-items/skills/attend-queue/evals/evals.json | 7 ++++--- plugins/work-items/skills/triage/evals/evals.json | 4 ++-- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/plugins/work-items/skills/attend-queue/evals/evals.json b/plugins/work-items/skills/attend-queue/evals/evals.json index d16db14248..7a6c4e77df 100644 --- a/plugins/work-items/skills/attend-queue/evals/evals.json +++ b/plugins/work-items/skills/attend-queue/evals/evals.json @@ -34,15 +34,16 @@ { "id": 3, "name": "attend-queue-briefs-before-asking-each-row", - "prompt": "/work-items:attend-queue\n\nThe operator works two rows back to back in the same pass.\n\nRow 1 — [escalated] #418 \"Uploader retries transient S3 5xx forever\". The worker's escalation marker asks which retry strategy to keep: (a) capped exponential backoff, 5 attempts then dead-letter — bounded worst-case latency, but a long outage drops uploads to the dead-letter queue; (b) unbounded backoff with a 30-minute ceiling — no upload is ever dropped, but a sustained outage holds worker slots indefinitely.\n\nRow 2 — [ratify] #503 \"Digest CLI crashes on an empty transcript file\". The marker classifies it C3 bug-fix-shaped and states the intended dispatch: the worker loop picks it up autonomously on the next cycle and opens a PR. Ratifying admits it to the autonomous lane; declining leaves it human-gated with the rationale recorded.", - "expected_output": "Before posing the escalated question, restates which item it concerns (#418 plus its one-line title), the decision being asked (which retry strategy to keep), and the consequence of each of the two presented options as the marker states them. Moving to the very next row, the same restatement happens again for the ratify prompt — #503 plus its title, that the decision is whether to ratify the C3 admission, and what ratifying versus declining causes — the operator's context from the escalated row is never assumed to carry over, even though it was the immediately preceding row in the same pass. Neither restatement is dropped or abbreviated for brevity, and neither invents detail the markers do not supply.", + "prompt": "/work-items:attend-queue\n\nThe operator works two rows back to back in the same pass.\n\nRow 1 — [escalated] #418 \"Uploader retries transient S3 5xx forever\". The worker's escalation marker asks which retry strategy to keep: (a) capped exponential backoff, 5 attempts then dead-letter — bounded worst-case latency, but a long outage drops uploads to the dead-letter queue; (b) unbounded backoff with a 30-minute ceiling — no upload is ever dropped, but a sustained outage holds worker slots indefinitely. The operator answers: \"take (a), the capped backoff.\"\n\nRow 2 — [ratify] #503 \"Digest CLI crashes on an empty transcript file\". The marker classifies it C3 bug-fix-shaped and states the intended dispatch: the worker loop picks it up autonomously on the next cycle and opens a PR. Ratifying admits it to the autonomous lane; declining leaves it human-gated with the rationale recorded.", + "expected_output": "Before posing the escalated question, restates which item it concerns (#418 plus its one-line title), the decision being asked (which retry strategy to keep), and the consequence of each of the two presented options as the marker states them — then poses the question and waits, rather than answering it itself. Only after the operator's supplied answer arrives does it write that answer back and move on. Moving to the very next row, the same restatement happens again for the ratify prompt — #503 plus its title, that the decision is whether to ratify the C3 admission, and what ratifying versus declining causes — the operator's context from the escalated row is never assumed to carry over, even though it was the immediately preceding row in the same pass. Neither restatement is dropped or abbreviated for brevity, and neither invents detail the markers do not supply.", "files": [], "expectations": [ "Restates item identity (#418 + title), the decision being asked, and the consequence of each option before posing the escalated question", "Restates only the consequences the escalation and ratify markers actually supply — no invented options, outcomes, or placeholder text", "Repeats the same full restatement for the very next row (#503, the ratify prompt, including the intended dispatch and the ratify-versus-decline consequence) rather than assuming context carries over from the prior row in the same pass", "Does not compress or drop the restatement for either row even under a terse output style", - "Still performs the row's underlying mechanics (interview/write-back for escalated, ratify-or-decline recording for ratify) after the restatement — the brief precedes the ask, it does not replace it" + "Still performs the row's underlying mechanics (interview/write-back for escalated, ratify-or-decline recording for ratify) after the restatement — the brief precedes the ask, it does not replace it", + "Waits for the operator's answer on #418 before advancing to #503 — the supplied answer is what unblocks the second row, never a self-answered question" ] } ] diff --git a/plugins/work-items/skills/triage/evals/evals.json b/plugins/work-items/skills/triage/evals/evals.json index 94b4dc5eb5..a6fe3679ac 100644 --- a/plugins/work-items/skills/triage/evals/evals.json +++ b/plugins/work-items/skills/triage/evals/evals.json @@ -84,7 +84,7 @@ { "id": 7, "name": "triage-briefs-before-presenting-the-recommendation", - "prompt": "/work-items:triage 64 — running interactively with a human operator present. #64 is titled \"Export job silently skips rows with a null tenant id\" and is filed as an enhancement request whose description is vague on scope. You have finished verifying and categorizing it; your assessment is that the target state is needs-info (the scope is undecided), with briefed-as-delegable the alternative if the operator would rather you pick a scope yourself.", + "prompt": "/work-items:triage 64 — running interactively with a human operator present. #64 is titled \"Export job silently skips rows with a null tenant id\" and is filed as an enhancement request whose description is vague on scope. You have completed step 2's category and target-state assessment and are at the direction gate, before verification; your assessment is that the target state is needs-info (the scope is undecided), with briefed-as-delegable the alternative if the operator would rather you pick a scope yourself.", "expected_output": "Restates which item is under discussion (#64 plus its one-line title), the decision being asked (which target state to apply), and the consequence of each presented option — needs-info parks the item pending the reporter, briefed-as-delegable admits it to the delegable lane on an agent-chosen scope — and only then presents the recommendation and waits for the operator's explicit direction. Nothing is mutated before that direction arrives.", "files": [], "expectations": [ @@ -97,7 +97,7 @@ { "id": 8, "name": "triage-briefs-before-each-interview-question", - "prompt": "/work-items:triage 64 — running interactively with a human operator present. #64 is titled \"Export job silently skips rows with a null tenant id\". You already briefed and presented your recommendation and the operator has replied: \"go ahead, park it as needs-info and ask the reporter what you need.\" The scope is still undecided, so a follow-up scoping question for the reporter is the next step.", + "prompt": "/work-items:triage 64 — running interactively with a human operator present. #64 is an enhancement request titled \"Export job silently skips rows with a null tenant id\", so the open question is scope, not fact. You already briefed and presented your step-2 recommendation and the operator has replied: \"go ahead, park it as needs-info and ask the reporter what you need.\" You are now at step 4 with the scope still undecided, so a follow-up scoping question for the reporter is the next step.", "expected_output": "Even though the operator has already been briefed once this pass and has just given direction, the follow-up scoping question carries its own restatement: which item it concerns (#64 plus its title), the decision the question resolves, and — because \"what scope do you actually need?\" is open-ended rather than a closed option set — what the answer will determine, rather than a manufactured list of options to enumerate consequences for. The question is posted as an item comment and the item is marked status:needs-info.", "files": [], "expectations": [ From 23d4e0cbe2fdd42df4e9ca0fd929734b77b92e18 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 25 Jul 2026 19:13:05 -0400 Subject: [PATCH 5/6] test(work-items): supply the ratify response the attend-queue fixture waits on The scenario named #503's intended dispatch and both outcomes but never the operator's ratify-or-decline answer, so recording the ratification required presuming it from the framing - the same missing-turn defect the #418 row had. The operator's "ratified" response is now part of the scenario, and the expectation covers both waits. --- plugins/work-items/skills/attend-queue/evals/evals.json | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/plugins/work-items/skills/attend-queue/evals/evals.json b/plugins/work-items/skills/attend-queue/evals/evals.json index 7a6c4e77df..b6d5469507 100644 --- a/plugins/work-items/skills/attend-queue/evals/evals.json +++ b/plugins/work-items/skills/attend-queue/evals/evals.json @@ -34,8 +34,8 @@ { "id": 3, "name": "attend-queue-briefs-before-asking-each-row", - "prompt": "/work-items:attend-queue\n\nThe operator works two rows back to back in the same pass.\n\nRow 1 — [escalated] #418 \"Uploader retries transient S3 5xx forever\". The worker's escalation marker asks which retry strategy to keep: (a) capped exponential backoff, 5 attempts then dead-letter — bounded worst-case latency, but a long outage drops uploads to the dead-letter queue; (b) unbounded backoff with a 30-minute ceiling — no upload is ever dropped, but a sustained outage holds worker slots indefinitely. The operator answers: \"take (a), the capped backoff.\"\n\nRow 2 — [ratify] #503 \"Digest CLI crashes on an empty transcript file\". The marker classifies it C3 bug-fix-shaped and states the intended dispatch: the worker loop picks it up autonomously on the next cycle and opens a PR. Ratifying admits it to the autonomous lane; declining leaves it human-gated with the rationale recorded.", - "expected_output": "Before posing the escalated question, restates which item it concerns (#418 plus its one-line title), the decision being asked (which retry strategy to keep), and the consequence of each of the two presented options as the marker states them — then poses the question and waits, rather than answering it itself. Only after the operator's supplied answer arrives does it write that answer back and move on. Moving to the very next row, the same restatement happens again for the ratify prompt — #503 plus its title, that the decision is whether to ratify the C3 admission, and what ratifying versus declining causes — the operator's context from the escalated row is never assumed to carry over, even though it was the immediately preceding row in the same pass. Neither restatement is dropped or abbreviated for brevity, and neither invents detail the markers do not supply.", + "prompt": "/work-items:attend-queue\n\nThe operator works two rows back to back in the same pass.\n\nRow 1 — [escalated] #418 \"Uploader retries transient S3 5xx forever\". The worker's escalation marker asks which retry strategy to keep: (a) capped exponential backoff, 5 attempts then dead-letter — bounded worst-case latency, but a long outage drops uploads to the dead-letter queue; (b) unbounded backoff with a 30-minute ceiling — no upload is ever dropped, but a sustained outage holds worker slots indefinitely. The operator answers: \"take (a), the capped backoff.\"\n\nRow 2 — [ratify] #503 \"Digest CLI crashes on an empty transcript file\". The marker classifies it C3 bug-fix-shaped and states the intended dispatch: the worker loop picks it up autonomously on the next cycle and opens a PR. Ratifying admits it to the autonomous lane; declining leaves it human-gated with the rationale recorded. The operator answers: \"ratified.\"", + "expected_output": "Before posing the escalated question, restates which item it concerns (#418 plus its one-line title), the decision being asked (which retry strategy to keep), and the consequence of each of the two presented options as the marker states them — then poses the question and waits, rather than answering it itself. Only after the operator's supplied answer arrives does it write that answer back and move on. Moving to the very next row, the same restatement happens again for the ratify prompt — #503 plus its title, that the decision is whether to ratify the C3 admission, and what ratifying versus declining causes — the operator's context from the escalated row is never assumed to carry over, even though it was the immediately preceding row in the same pass. Neither restatement is dropped or abbreviated for brevity, and neither invents detail the markers do not supply. The ratification is recorded only after the operator's \"ratified\" response, never presumed from the prompt's framing.", "files": [], "expectations": [ "Restates item identity (#418 + title), the decision being asked, and the consequence of each option before posing the escalated question", @@ -43,7 +43,7 @@ "Repeats the same full restatement for the very next row (#503, the ratify prompt, including the intended dispatch and the ratify-versus-decline consequence) rather than assuming context carries over from the prior row in the same pass", "Does not compress or drop the restatement for either row even under a terse output style", "Still performs the row's underlying mechanics (interview/write-back for escalated, ratify-or-decline recording for ratify) after the restatement — the brief precedes the ask, it does not replace it", - "Waits for the operator's answer on #418 before advancing to #503 — the supplied answer is what unblocks the second row, never a self-answered question" + "Waits for the operator's answer on #418 before advancing to #503, and for the ratify-or-decline response on #503 before recording it — each supplied answer is what unblocks the next step, never a self-answered question" ] } ] From 72aed95bc5b4b9ff8bd0c36f9a1f1bf8393a488a Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 25 Jul 2026 19:17:41 -0400 Subject: [PATCH 6/6] fix(work-items): scope the direction gate's consequences to presented options The step-4 interview already said "each option you present"; the direction gate above it still said "each option", which reads as every target state the state machine admits rather than the recommendation and the alternatives actually put to the operator. --- plugins/work-items/skills/triage/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/work-items/skills/triage/SKILL.md b/plugins/work-items/skills/triage/SKILL.md index d51015e3b8..8200544423 100644 --- a/plugins/work-items/skills/triage/SKILL.md +++ b/plugins/work-items/skills/triage/SKILL.md @@ -100,7 +100,7 @@ Classify **bug vs enhancement** first — it steers the rest of the flow (bugs g **Direction gate.** Recommending is read-only; the gate governs *mutation* — labels, comments, closes, item creation — and which side of it you are on is fixed by how triage was invoked: -- **Interactive session** — a human operator is present and no standing lane rules were supplied. **Brief before asking**: before presenting the recommendation, restate (1) which item (number + one-line title), (2) the decision being asked, and (3) the consequence of each option — then present the recommendation and **wait for the user's explicit direction** before mutating anything. This is the default whenever the invocation carries no autonomous mandate. The restatement is not optional compression fodder: a terse output style must never drop it, and it applies on every decision question, not only the first one of a pass — the operator working several rows in sequence (e.g. via `/work-items:attend-queue`) cannot be assumed to still be holding a prior item's context. +- **Interactive session** — a human operator is present and no standing lane rules were supplied. **Brief before asking**: before presenting the recommendation, restate (1) which item (number + one-line title), (2) the decision being asked, and (3) the consequence of each option **you present** — the recommendation and the alternatives you are actually putting to the operator, not every target state the state machine admits — then present the recommendation and **wait for the user's explicit direction** before mutating anything. This is the default whenever the invocation carries no autonomous mandate. The restatement is not optional compression fodder: a terse output style must never drop it, and it applies on every decision question, not only the first one of a pass — the operator working several rows in sequence (e.g. via `/work-items:attend-queue`) cannot be assumed to still be holding a prior item's context. - **Autonomous lane** — triage is running unattended as a `/loop` or `/schedule` AFK session whose standing rules — the directive supplied with its `/loop` / `/schedule` invocation — already authorize triage mutations. Those standing rules **are** the direction this gate requires: treat the gate as satisfied and proceed through verification and outcome without a human turn, prefixing every comment and item you create with the AI disclaimer. There is no operator turn to wait for, so blocking here would deadlock the lane — the gate is met by the lane's mandate, not skipped. The autonomous branch is the mode the AI disclaimer already anticipates: a session that mutates without a human turn. The two are one mode, not a contradiction. Formalizing this as the autonomous-mode contract — codifying that standing-lane rules constitute direction — is tracked in #459.