diff --git a/.changes/evals/chore-dispatch-timeout-and-judge-calibration-63a982bb.md b/.changes/evals/chore-dispatch-timeout-and-judge-calibration-63a982bb.md new file mode 100644 index 0000000000..31a2c993aa --- /dev/null +++ b/.changes/evals/chore-dispatch-timeout-and-judge-calibration-63a982bb.md @@ -0,0 +1,7 @@ +--- +bump: patch +--- + +### Changed + +- `plugin-eval`: `## Run` now calibrates judges before the full-suite run, and on a miss fixes the rubric or moves the run to a stronger `--judge-model`. The rule covers every LLM-judge grader the calibration harness can reproduce (labeled samples, judging text), treats the default judge as uncalibrated until it passes, and reports graders it cannot reproduce as uncalibrated. Observed twice in #6669's eval gate: the default Haiku judge marked uc3 below baseline in a full run, then agreed with its labels on only 66.7% of calibration runs; a Sonnet judge calibrated at 100% and the next full run passed. diff --git a/.changes/implementation/chore-dispatch-timeout-and-judge-calibration-9d3fea74.md b/.changes/implementation/chore-dispatch-timeout-and-judge-calibration-9d3fea74.md new file mode 100644 index 0000000000..9863909d02 --- /dev/null +++ b/.changes/implementation/chore-dispatch-timeout-and-judge-calibration-9d3fea74.md @@ -0,0 +1,7 @@ +--- +bump: patch +--- + +### Changed + +- `implement-dispatch`: every worker brief now carries a clause to wrap each command that can block in a timeout, record a timeout as a FAIL, and never run an interactive command. Observed once, while implementing #6306 (PR #6669): a Phase 1 worker hung 2.5 hours on one unbounded Sanity Check command and had to be stopped; later briefs carried the clause and no phase hung again. Shipped on the maintainer's decision despite a single observation. diff --git a/plugins/evals/skills/plugin-eval/SKILL.md b/plugins/evals/skills/plugin-eval/SKILL.md index 5708eb5850..0486cbf14c 100644 --- a/plugins/evals/skills/plugin-eval/SKILL.md +++ b/plugins/evals/skills/plugin-eval/SKILL.md @@ -219,24 +219,24 @@ committed `mocks/.replay/` so agent mocks replay without a model call. invoking the `evals:validate` skill through the Skill tool for the finding detail; exit 2 means the eval dir is unreadable or the arguments are wrong, which is also a stop. -2. Invoke the CLI. Confirm the target comes first, before any list-taking flag: +2. Run "Calibrating a judge" first unless every `llm` grader it can reproduce has passed with this + run's judge model. On a miss, fix the rubric or calibrate a stronger `--judge-model` and use it. + +3. Invoke the CLI. Confirm the target comes first, before any list-taking flag: ```bash claude plugin eval --trust-plugin --keep-temp --json results.json --threshold 0.8 --max-cost-usd --no-publish ``` `--keep-temp` keeps each run's trace, which the validity gate under "Reading the delta" reads. - Run it in the foreground with a tool timeout that covers the estimate (a three-case pass took - about six minutes here) and wait. Never background the CLI from a headless `-p` session: the - session ends and takes the run with it. - - The run is finished when the process exits and `results.json` exists; read the exit code and the - JSON together, never one alone. Write the file somewhere git ignores (the CLI's own copy lands - under `/results//`, and a repo that ignores that tree can take `--json` - there too); a run result is evidence to distill, not a file to commit. + about six minutes here) and wait. Never background the CLI from a headless `-p` session: the run + ends with the session. The run is finished when the process exits and `results.json` exists; read + the exit code and the JSON together, never one alone. Write the file somewhere git ignores (the + CLI's own copy lands under `/results//`, and a repo that ignores that tree + can take `--json` there too); a run result is evidence to distill, not a file to commit. -3. Read the JSON with the `read` action below. With `--json ` there is no terminal table, so +4. Read the JSON with the `read` action below. With `--json ` there is no terminal table, so the JSON is the only record of what happened. | Fact | Basis and as-of | Recheck trigger, and what to do when it fires | @@ -348,10 +348,9 @@ What the number means: case, and confirm by re-running that one case until the grader passes. A passing fired grader shows the trigger works; it is not evidence that the skill improved, and any claim of a gain still needs a VALID run and its noise report. -3. If that grader passes and the delta is negative, suspect the judge before the plugin. A small - judge marks a correct answer wrong on formatting. Re-run with a larger `--judge-model` and - tighten the rubric so formatting cannot decide the verdict; the step is settled when the verdict - survives a rubric that says nothing about form. +3. If that grader passes and the delta is negative, suspect the judge before the plugin: a small + judge marks a correct answer wrong on formatting. Re-run with a larger `--judge-model` and tighten + the rubric so formatting cannot decide the verdict; settled when it survives a rubric silent on form. 4. Iterate on one case with `--case --runs 1 --ablation none`, which reports `SCORE` and `PASS%` instead of `WITH`, `W/OUT`, and delta. One run is noisy, so confirm any change at 3 runs per case and read the confirm's noise report before trusting it. @@ -369,9 +368,12 @@ What the number means: ## Calibrating a judge -Trust an `llm` grader's scores only after its judge agrees with labeled answers on at least 90% of -runs. The labels are the must-pass and must-fail answers in the case's `samples/.json`; -three agents label them independently and the user settles every disagreement. Then: +Trust an `llm` grader's scores only after its judge, the model the suite uses, agrees with labeled +answers on at least 90% of runs; a pass says nothing about another model. Before the first full +run, calibrate every LLM-judge grader the harness can reproduce (one with labeled samples, judging +text); the default judge is uncalibrated until it passes. Report any other judge grader as +uncalibrated, never trusted silently. Labels are the must-pass and must-fail answers in +the case's `samples/.json`, set by three agents independently with the user settling every disagreement: ```bash python3 "${CLAUDE_PLUGIN_ROOT}/skills/plugin-eval/scripts/calibrate-judge.py" build --suite --out @@ -380,23 +382,21 @@ python3 "${CLAUDE_PLUGIN_ROOT}/skills/plugin-eval/scripts/calibrate-judge.py" sc ``` `build` writes one case per sample into an empty plugin: the source case's prompt goes out -unchanged, and an appended system prompt has the agent reply with the sample word for word, so the -judge grades that sample as the answer to that question. No generated file carries the label; a -must-pass and a must-fail case differ only in the sample text. The appended prompt presents the -sample as fixed test material to output byte for byte even when it is wrong or incomplete, with no -commentary added. `build` skips a grader that judges a file or mock calls, and prints one line for -each empty or whitespace-only sample it skips: Claude Code answers an empty reply with an injected -user turn, so it cannot be reproduced, and an empty answer is a deterministic failure that needs no -judge. `--threshold 0` keeps the CLI's exit code about errors, since must-fail cases are -meant to score 0. Calibrate with the judge model the real suite uses; the result says nothing -about another. +unchanged, and an appended system prompt presents the sample as fixed test material for the agent +to output byte for byte, even when wrong or incomplete, with no commentary, so the judge grades that +sample as the answer to that question. No generated file carries the label; a must-pass and a +must-fail case differ only in the sample text. `build` skips a grader that judges a file or mock +calls, and prints one line for each empty or whitespace-only sample it skips: Claude Code answers an +empty reply with an injected user turn, so it cannot be reproduced, and an empty answer is a +deterministic failure that needs no judge. `--threshold 0` keeps the CLI's exit code about errors, +since must-fail cases are meant to score 0. `score` prints a `FAIL grader` line for each grader under 90% and exits 1; fix that rubric, or move -to a stronger judge, and calibrate again before reading its scores. Its false positives and -negatives name the samples to read first. A run whose reply was not the sample is left out of the -agreement, and so is one with neither a kept trace nor judge evidence, which is also reported -unchecked. A sample with no reproduced run is listed as `untested`, counts toward no agreement, -and shows in the verdict line; raise `--runs` or tighten the prompt before reading the grader's score. +the whole suite to a stronger judge, and calibrate again before reading its scores. Its false +positives and negatives name the samples to read first. Agreement leaves out a run whose reply was +not the sample, and one with neither a kept trace nor judge evidence (also reported unchecked). A +sample with no reproduced run is listed `untested`, counts toward no agreement, and shows in the +verdict line; raise `--runs` or tighten the prompt before reading the grader's score. - **Pointer**: what a judge reads for each `focus`, see ; the 90% bar, see diff --git a/plugins/implementation/skills/implement-dispatch/SKILL.md b/plugins/implementation/skills/implement-dispatch/SKILL.md index e0a7556249..1dd3dfdc53 100644 --- a/plugins/implementation/skills/implement-dispatch/SKILL.md +++ b/plugins/implementation/skills/implement-dispatch/SKILL.md @@ -88,7 +88,7 @@ Because the orchestrator stays on the default branch, **every source-touching op 7. Any model routing the plan specifies. 8. **When the worker edits in a dedicated worktree** (an out-of-tree sibling or any checkout other than the session's default): that worktree's absolute path, written literally, plus the anchoring rule in the Gotchas bullet "A worker's worktree cwd does not persist across tool calls": `git -C `, one plain command per call, and no `VAR=` prefixes, loops, `$( )` or `cd`. A command that reads its configuration from the working directory (a repository's own build, test or lint wrapper) runs through its own directory flag (`make -C `, `npm --prefix `) or by the wrapper's absolute path when the wrapper finds the repository from its own location; a command with neither is left to the orchestrator's main-side gate. The interactive default is a pre-existing worktree path the brief supplies. 9. **When provisioning is worker-side** (the autonomous work-lane; see Commit authority): materializing that isolated worktree is the worker's **first step**. The orchestrator cannot itself invoke `/source-control:worktree create`, whose `EnterWorktree` terminal would transition the orchestrator's session. The worker invokes `/source-control:worktree` via the Skill tool for its non-entering creation seam when installed, or a plain `git worktree add` otherwise, and works in it under the item 8 anchoring, never entering it. The brief tells the worker, before its first edit, to fetch and confirm the new branch starts from the intended base (`git -C merge-base HEAD /` equals `/`, where `` is the remote provisioning based the branch on, not always `origin`), and on a mismatch to STOP and report. Provisioning happens **once per item, on the first dispatched phase**: the worktree persists across the item's phases, so every **later** phase of the same item is handed that same worktree path and works in it. Never re-provision the already-checked-out item branch; both `git worktree add -b ` and attaching the branch fail while it is checked out in the persisted worktree. The first phase's brief also instructs the worker to bring the branch current with the default branch, commit, and push before returning, then **return the worktree's absolute path plus the branch name** so the orchestrator can open the PR against the pushed branch. A worker that cannot provision an isolated worktree STOPs and reports rather than editing the default checkout. - 10. **The three CI-hygiene clauses, front-loaded**: the Gotchas bullets on issue-number comments, new shebang files and early push, the last two under `worker`. + 10. **The four hygiene clauses, front-loaded**: the Gotchas bullets on bounded commands, issue-number comments, new shebang files and early push, the last two under `worker`. 11. **The phase's design excerpt**: the part of PLAN.md's `## Design` section the phase touches (its module layout, contracts, variation verdicts, and the conventions they cite), quoted verbatim, never paraphrased. The worker's worktree holds no memory slice, so the quote is the only copy of the design it sees. When the plan has no `## Design` section or the phase touches none of it, the brief says `Design: none`. 12. **Existing test files are FORBIDDEN** unless the phase is a test-authoring phase (the plan names writing or changing tests as the phase's work) or the plan names that test file as changing with the behavior it pins. New test files stay allowed under the phase's TDD cadence, but a new or changed runner configuration or test setup file (`conftest.py`, a pytest ini or `pyproject.toml` pytest table, a Jest or Vitest config or its setup files, `package.json` test scripts) is a phase-verifier finding, like an existing-test edit, because it changes what every test run executes. A worker that finds an existing test must change to pass STOPs per item 2 instead of editing it. Why: in one benchmark study (ImpossibleBench, arXiv 2510.20270), read-only tests stopped agents modifying tests and kept legitimate performance, but did not stop special-casing the tested inputs, so this fence is partial cover, and the holdout option below and the phase-verifier remain the rest. 2. **Dispatch** each worker by its phase's routing row, the `Model` column. Never rely on root @@ -244,6 +244,7 @@ We treat the phases of one approved plan as one task, so the orchestrator's wind - **A worker STOP is a divergence signal, not a failure.** Route it through `/implementation:implement`'s Step 3 severity ladder; revising the brief is the cheap fix, a plan review the escalation - **Surface subagent results before ending the turn.** Results left unsurfaced at turn end are lost to the user - **A worker's worktree cwd does not persist across tool calls.** Brief every dedicated-worktree worker to anchor every command, edits AND git status/add/commit/diff/log, on the worktree's literal absolute path: `git -C ` for git, absolute paths for every other command, one plain command per call. No `VAR=` prefixes, loops, `$( )` or `cd`: a one-time `cd` does not persist, and those shapes leave the permission fast path, where a headless worker is blocked. The ban covers a `cd` into the worktree itself: we grant it no exception (Basis: . As of: 2026-10-09. Recheck: that section changes which `cd` targets run without a prompt). A command that reads its configuration from the working directory runs through its own directory flag (`make -C `, `npm --prefix `) or by the wrapper's absolute path when the wrapper finds the repository from its own location, since absolute file paths alone would run it against the wrong checkout; a command with neither is left to the orchestrator's main-side gate (see Gates) +- **Bound every command that can block.** Brief every worker to prefix each command that can block (tests, scripts, a nested `claude -p`, network calls) with a timeout such as `timeout `, to record a timeout as a FAIL, and never to run an interactive command. Nothing else in a brief bounds command time, so one hung command stalls the phase indefinitely - **No issue-number back-references in code comments.** Brief every worker that a comment citing an issue number (`# Issue #NNN ...`, `(issue #NNN obs #N)`) trips the `comment-hygiene` check; `TODO(#issue)` is the sanctioned exception - **New shebang files need `chmod`, then `git add`, then `git update-index --chmod=+x`. In that order.** Brief every `worker`-authority worker (for `orchestrator`, see Commit authority): `chmod +x `, then `git add ` (a not-yet-tracked file fails `git update-index --chmod=+x` outright. It can't override the index mode of a path that isn't staged yet), then `git update-index --chmod=+x ` to force the index mode explicitly (skip symlinks, staged `120000`, they fail the same command). A shebang file staged non-executable trips the `exec-bit` check - **Push early, before waiting on CI. But never the PR.** Brief every `worker`-authority worker to commit and push as early as practical rather than deferring until its fix-and-verify loop is done, so a mid-session death never orphans unpushed work. This is a source-only checkpoint commit. The phase-boundary ritual (Step 4) still runs separately, orchestrator-side, once the phase's acceptance criteria are verified. PR creation stays out of every worker brief. It happens in the orchestrator's post-verification flow (`/implementation:implement` Step 5) after every return is verified and the build/test gate passes. The early push holds only before the pull request exists. Once it is open, every push starts a full CI run and cancels the one in flight, so brief a worker dispatched onto an open pull request (a later phase, a fix) to commit as it goes and push once, after its whole change set passes its checks. Commit authority `orchestrator` forgoes the early push: uncommitted work lives only on local disk until the orchestrator commits. A worker that waits on CI waits with `run_in_background` or Monitor, never a foreground `sleep` or `until` loop or `--watch`.