Skip to content
Merged
11 changes: 11 additions & 0 deletions .changes/evals/upstream-pointers-plugin-quality-4e1a7c20.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
bump: patch
---

### Fixed

- `/evals:plugin-eval` references no longer say a `Write` or `Edit` grant needs a sandbox backend; which grants run sandboxed is now read live from the plugin-evals Grant tools section.

### Changed

- `/evals:plugin-eval` and `/evals:methodology` references point at the exact upstream sections (prompt.md fields, grader targets, exit codes and options, JSON result, requirements, and the platform evaluation sections) with an as-of date and recheck trigger instead of restating their defaults, bounds and exit-code meanings.
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
bump: patch
---

### Changed

- `/mcp-tools:audit` grades C17 and C18 against the result-size ceiling and the effective approval value read live from the Claude Code MCP page, instead of a copied ceiling number, and the duplicate statements in the checklist and discovery reference cite that one record.
- `/mcp-tools:audit-posture` and the README point at the MCP scope, precedence and managed-server sections for where configuration lives and which entries load, instead of restating file locations, precedence and entry checks; the organization-provided removal note is a pointer record.
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
bump: patch
---

### Changed

- `/plugin-quality:audit` and the `auditor` agent point at the exact Claude Code docs sections (hook exit codes, `PostToolUse` and matchers, placeholder resolution, `plugin validate --json`, monitors, mod calls and events, skill and command name precedence, forked and cloud skill loading, subagent model order, settings scopes) with an as-of date and recheck trigger instead of restating them. The `PostToolUse`, placeholder and report-filename-guardrail records now live once in `evidence-packet.md`, and the agent, skill and `packet-seal.sh` cite them.
6 changes: 4 additions & 2 deletions plugins/evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,10 @@ cookbook (`anthropics/claude-cookbooks` `misc/building_evals.ipynb`), fetched 20

`claude plugin eval` is what runs and scores a plugin's eval cases: it loads the plugin, replays
each case with the plugin and again with nothing loaded, grades both arms, and reports `WITH`,
`W/OUT`, and the delta between them. It needs Claude Code v2.1.269 or later, and every run and every
judge grader is a metered model call on your own account.
`W/OUT`, and the delta between them. It needs a recent Claude Code; for the minimum version, see
[Requirements](https://code.claude.com/docs/en/plugin-evals#requirements) (as of 2026-10-11;
recheck when that section's version floor changes). Every run and every judge grader is a metered
model call on your own account.

This plugin is the practice around that command rather than a second runner. `/evals:plugin-eval`
preflights the machine and the target, validates the case files, prices the suite before it spends,
Expand Down
12 changes: 10 additions & 2 deletions plugins/evals/skills/methodology/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,16 @@ metadata:
# LLM evaluation methodology

Distilled from a cover-to-cover reading of Anthropic's "Define success criteria and build
evaluations" (<https://platform.claude.com/docs/en/test-and-evaluate/develop-tests>) and its linked
evals cookbook (`anthropics/claude-cookbooks` `misc/building_evals.ipynb`), fetched 2026-08-08.
evaluations" and its linked evals cookbook (`anthropics/claude-cookbooks`
`misc/building_evals.ipynb`).
Comment thread
kyle-sexton marked this conversation as resolved.

- **Pointer**: when a specific must be current, fetch
[Define your success criteria](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests#define-your-success-criteria)
and [Build evaluations](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests#build-evaluations)
live.
- **As of**: 2026-08-08 (distilled); pointer anchors verified 2026-10-11
- **Recheck trigger**: either section is retitled, moves, or changes the guidance this page's
rules rest on.

This page states the rules; answer from it. The reference files below are background for a human
reader, each with its own source stamp. For runnable code, or a specific that must be current,
Expand Down
13 changes: 9 additions & 4 deletions plugins/evals/skills/methodology/reference/eval-design.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,14 @@
# Eval design

Distilled from Anthropic's "Define success criteria and build evaluations"
(<https://platform.claude.com/docs/en/test-and-evaluate/develop-tests>) and the evals cookbook
(`anthropics/claude-cookbooks` `misc/building_evals.ipynb`), both fetched 2026-08-08. Re-fetch the
sources before treating any specific here as current.
Distilled from Anthropic's "Define success criteria and build evaluations" and the evals cookbook
(`anthropics/claude-cookbooks` `misc/building_evals.ipynb`).

- **Pointer**: when a specific here must be current, fetch
[Build evaluations](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests#build-evaluations)
and [Eval design principles](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests#eval-design-principles)
live.
- **As of**: 2026-08-08 (distilled); pointer anchors verified 2026-10-11
- **Recheck trigger**: either section is retitled, moves, or changes its design principles.

This repository's own defaults and source-conflict records: [local-decisions.md](local-decisions.md).

Expand Down
13 changes: 9 additions & 4 deletions plugins/evals/skills/methodology/reference/grading.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,14 @@
# Grading methods

Distilled from Anthropic's "Define success criteria and build evaluations"
(<https://platform.claude.com/docs/en/test-and-evaluate/develop-tests>) and the evals cookbook
(`anthropics/claude-cookbooks` `misc/building_evals.ipynb`), both fetched 2026-08-08. Re-fetch the
sources before treating any specific here as current.
Distilled from Anthropic's "Define success criteria and build evaluations" and the evals cookbook
(`anthropics/claude-cookbooks` `misc/building_evals.ipynb`).

- **Pointer**: when a specific here must be current, fetch
[Grade your evaluations](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests#grade-your-evaluations)
and [Tips for LLM-based grading](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests#tips-for-llm-based-grading)
live.
- **As of**: 2026-08-08 (distilled); pointer anchors verified 2026-10-11
- **Recheck trigger**: either section is retitled, moves, or changes its grading guidance.

This repository's own defaults and source-conflict records: [local-decisions.md](local-decisions.md).

Expand Down
13 changes: 8 additions & 5 deletions plugins/evals/skills/methodology/reference/recipes.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
# Eval recipes: one per criteria dimension

Distilled from Anthropic's "Define success criteria and build evaluations"
(<https://platform.claude.com/docs/en/test-and-evaluate/develop-tests>, fetched 2026-08-08). The
source page carries full runnable code for every recipe in Python, TypeScript, C#, Go, Java, PHP,
and Ruby. Fetch it for implementation; this file carries the design of each recipe. Re-fetch the
source before treating any specific here as current.
Distilled from Anthropic's "Define success criteria and build evaluations". This file carries the
design of each recipe; the source page carries the runnable code.

- **Pointer**: when implementing a recipe, or when a specific here must be current, fetch
[Example evals](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests#example-evals)
live.
- **As of**: 2026-08-08 (distilled); pointer anchor verified 2026-10-11
- **Recheck trigger**: that section is retitled, moves, or adds, drops or renames a recipe.

This repository's own defaults and source-conflict records: [local-decisions.md](local-decisions.md).

Expand Down
11 changes: 8 additions & 3 deletions plugins/evals/skills/methodology/reference/success-criteria.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,13 @@
# Success criteria

Distilled from Anthropic's "Define success criteria and build evaluations"
(<https://platform.claude.com/docs/en/test-and-evaluate/develop-tests>, fetched 2026-08-08).
Re-fetch the source before treating any specific here as current.
Distilled from Anthropic's "Define success criteria and build evaluations".

- **Pointer**: when a specific here must be current, fetch
[Define your success criteria](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests#define-your-success-criteria)
and [Common success criteria](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests#common-success-criteria)
live.
- **As of**: 2026-08-08 (distilled); pointer anchors verified 2026-10-11
- **Recheck trigger**: either section is retitled, moves, or changes its criteria list.

This repository's own defaults and source-conflict records: [local-decisions.md](local-decisions.md).

Expand Down
65 changes: 49 additions & 16 deletions plugins/evals/skills/plugin-eval/reference/case-authoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,16 +88,23 @@ unless the case routes it in explicitly.
| Type | Fields | Passes when |
|---|---|---|
| `regex` | `pattern`, `flags`, `match`, `target` | The JS regex is found in the target; `match: not_contains` requires absence, `match: "count:N"` exactly N |
| `tool_used` | `tool`, `input_match`, `min`, `max` | Matching calls fall in range; `min` defaults to 1 and `max` to unlimited |
| `tool_used` | `tool`, `input_match`, `min`, `max` | Matching calls fall in range; the range defaults are in the `tool_used` row of [Grader types](https://code.claude.com/docs/en/plugin-evals#grader-types), as of 2026-10-11. Recheck trigger: that row or the grader's fields change |
| `tool_order` | `before`, `after` | The `tool_order` row of [Grader types](https://code.claude.com/docs/en/plugin-evals#grader-types), as of 2026-10-07. Recheck trigger: that row or the grader's fields change |
| `file_exists` | `path`, `exists` | The `file_exists` row of [Grader types](https://code.claude.com/docs/en/plugin-evals#grader-types), as of 2026-10-07. Recheck trigger: that row or the grader's fields change. Scaffold output and files the model only edits do not count |
| `llm` | `criteria`, `focus` | The judge votes PASS in at least 2 of 3. In the `.md` layout the body is the criteria |
| `baseline` | `baseline_file`, `criteria` | The judge finds the run at least as good as the reference `.jsonl` in the case dir |

Targets for `regex.target` and `llm.focus`: `last_message` (the default), `trace` (JSON per line,
where a judge sees only the first 12 and last 12 messages and quotes are JSON-escaped, so match
`\"`), `files` (the **list of created paths**, not their contents), `{ source: file, path: <p> }`
(the contents), and `mock_calls`.
Targets for `regex.target` and `llm.focus`: `last_message`, `trace` (JSON per line, where quotes
are JSON-escaped, so match `\"`), `files` (the **list of created paths**, not their contents),
`{ source: file, path: <p> }` (the contents), and `mock_calls`. Pick the target explicitly rather
than relying on the default, and assume a judge given `trace` sees only part of a long one.

- **Pointer**: when choosing a target, or judging a long trace, fetch
[What a grader can look at](https://code.claude.com/docs/en/plugin-evals#what-a-grader-can-look-at)
live for the default target and the judge's trace window.
- **As of**: 2026-10-11
- **Recheck trigger**: that section adds or renames a target, changes the default, or changes how
much of a trace the judge reads.

## Mocks

Expand All @@ -112,30 +119,56 @@ where a judge sees only the first 12 and last 12 messages and quotes are JSON-es

## Frontmatter bounds

| Fact | Basis and as-of | Recheck trigger, and what to do when it fires |
|---|---|---|
| The complete `prompt.md` frontmatter key set is `schema_version` (currently `"1.1"`), `name`, `description`, `tags`, `plugins`, `runs` (default 3, range 1 to 50), `expected_outcome`, `model`, `max_turns` (default 10, maximum 200), `timeout_seconds` (default 300, maximum 3600), `allowed_tools`, `append_system_prompt`, and `env`. An unknown key is an error (`prompt.md: unknown frontmatter key`), each grader type's option set is strict, `weight` must be positive, and a case whose `schema_version` major exceeds what the binary supports is refused | <https://code.claude.com/docs/en/plugin-evals> and the binary's own validator messages, verified 2026-09-12 | Recheck trigger: the page's frontmatter table gains or loses a key, or a bound changes. Then re-read the page, re-derive this row and the validator's FAIL tier together, and refresh this record with the outcome |
The `prompt.md` key set, each key's default and bound, and the schema version are the CLI's,
read live; the validator's FAIL tier mirrors them in its own constants, so a change there is
re-derived in the validator too. The binary's own load-time messages, read 2026-09-12, showed
it refusing an unknown key, a non-positive `weight`, and a newer schema major; the validator
reports the same as FAIL.

- **Pointer**: when writing or checking a case's frontmatter, fetch
[`prompt.md` fields](https://code.claude.com/docs/en/plugin-evals#prompt-md-fields) live.
Comment thread
kyle-sexton marked this conversation as resolved.
- **As of**: 2026-10-11
- **Recheck trigger**: that section gains or loses a key, a default or bound changes, or the
binary's validator rejects a case the static validator passed. Then re-derive the validator's
FAIL tier with it.

Which grants need a sandbox backend is the CLI's rule too, and this checklist never lists the
tools.

- **Pointer**: when a case grants a tool outside the read-only set, fetch
[Grant tools](https://code.claude.com/docs/en/plugin-evals#grant-tools) live for which grants
run sandboxed and what each platform needs for a sandbox backend.
- **As of**: 2026-10-11
- **Recheck trigger**: that section changes which grants run under the sandbox or what a platform
needs for the backend, or a run is refused for a grant that section does not name.

- [ ] `runs`, `max_turns`, and `timeout_seconds` are inside their bounds. Hitting `max_turns` is a
run error, not a low score.
- [ ] `allowed_tools` requests only what the case needs. Read-only tools (`Read`, `Glob`, `Grep`,
`NotebookRead`, `Skill`, `AskUserQuestion`, `Agent`, `TodoWrite`, `TaskCreate`, `TaskGet`,
`TaskList`, `TaskUpdate`, `TaskStop`; see the record in the validator's `READ_ONLY_TOOLS`)
need no operator grant;
anything else needs `--allow-tools` and, for `Bash`, `PowerShell`, `Write`, or `Edit`, a sandbox backend.
- [ ] `allowed_tools` requests only what the case needs. Read-only tools (the set in the
validator's `READ_ONLY_TOOLS`, which mirrors the Grant tools pointer above) need no operator
grant;
anything else needs `--allow-tools`, and a grant that runs sandboxed also needs a sandbox
backend on the machine (the Grant tools record above, in
[Frontmatter bounds](#frontmatter-bounds); it has no heading of its own. As of 2026-10-11.
Recheck trigger: that record moves or stops naming the backend).
- [ ] `tags` are set when the suite will ever be filtered. A case runs if **any** of its tags match.

Run the static validator before every paid pass; it checks the bounds above with no model call.

## Common mistakes

- `target: files` when the contents were meant. `files` is the list of created paths.
- Assuming `target` defaults to `trace`. It defaults to `last_message`.
- Assuming `target` defaults to `trace`. It does not; read the default at the targets
record's pointer in [Graders](#graders) (the record has no heading of its own). As of 2026-10-11.
Recheck trigger: that record moves or its pointer stops covering the default target.
- Inline `(?i)` in a pattern. Case-insensitivity is `flags: i`.
- `file_exists` with the default `exists` in a read-only suite. Nothing is created, so the grader
can never pass.
- A gated tool in `allowed_tools` with no matching `--allow-tools` grant. The tool is removed from
the session and reported on stderr as `not granted`, and the case quietly measures a model without
it.
- Taking `claude plugin eval init --bare`'s scaffold as-is: its default grader is `llm`, the paid
type.
- Taking `claude plugin eval init --bare`'s scaffold as-is without checking which grader it
writes. Replace a paid `llm` grader with a deterministic one wherever one can decide the case.
When checking what the scaffold writes, fetch
[Write a case manually](https://code.claude.com/docs/en/plugin-evals#write-a-case-manually)
live. As of 2026-10-11. Recheck trigger: that section changes what `init --bare` writes.
Loading
Loading