docs: record why this marketplace's eval format is not claude plugin eval's - #3102
Conversation
… eval`'s The Evals section states that no first-party command executes model-graded evals, but never says which format the shipped suites are in or how it relates to the `claude plugin eval` command, so a reader who meets that command reasonably reads 203 evals.json files as drift. They are not. The shipped format is skill-creator's, which is why the schema notes upstream's `assertions` naming, and it is the ecosystem-wide shape. The command consumes a different layout and is early access — it refuses to run — so adopting it would trade a corpus CI checks for one nobody here can execute. Adoption stays deferred behind the same tracker as the runner. Also reconciles write-for-agents-brief.md, whose acceptance criterion named a `claude plugin eval` suite as its settling artifact while the skill shipped in #3003 with the house format. The criterion now describes what actually gates it. Found while diagnosing #3063, where three rounds of work built machinery to make an eval prompt literally executable before establishing that nothing executes one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b965819795
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Claude finished @kyle-sexton's task in 2m 24s —— View job Claude finished the code-review lane for this PR
SummaryReviewed both changed files against No blocking or high-signal findings. This is a well-substantiated documentation change, and I independently checked its central factual claims rather than restating the author's own verification:
I could not independently verify the Given the high-signal bar for this lane (exclude linter-catchable issues, pre-existing issues, style nits, and unverifiable generic advice), I found nothing to flag as a blocking or important finding, so no inline comments were posted. |
|
Claude has reviewed this PR 2 times. The lane skips further automatic reviews after 5; deleting this comment resets the count. |
…enerally The paragraph asserted that nothing executes an eval prompt, and told authors not to assume a runner would stage anything. Both overreach. This repo's own surfaces are scoped more carefully: plugins/evals/skills/design/SKILL.md and that plugin's README say no MARKETPLACE command runs model-graded evals, and name Anthropic's skill-creator as a runner a consumer may have installed — the runner belonging to this very format. Left as written, the guidance would have pushed authors toward cases that are incompatible with skill-creator's staging, which is the opposite of the intent. The corrected text keeps what is true and load-bearing — nothing here runs these, so a case must be followable by hand — and replaces the false half with the actual rule the #3063 apparatus violated: declare fixtures in files[] and reference them by their documented path, rather than hand-rolling staging in the prompt string. A prompt that builds its own workspace is neither followable by hand nor compatible with the runner that would otherwise stage it. Found by review on #3102. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 3m 20s —— View job Claude finished the code-review lane for this PR
SummaryReviewed One finding, not carried by any changed line, so it's here rather than inline:
Verified independently (not just restating the author's claims):
Could not verify (no network/gh access to confirm against GitHub, consistent with the earlier automated review's own caveat): the No other correctness or maintainability issues found in the diff. Security lane exists in this repo ( |
2 participants
No linked issue.
Summary
docs/MIGRATION-PLAYBOOK.md"Evals" already states that no first-party command executes model-gradedevals and names the tracker for the deferred runner. What it never says is which eval format the
203 shipped suites are in, or how that relates to the
claude plugin evalcommand — so a reader whomeets that command reasonably concludes the repo has drifted from it.
That conclusion is wrong, and it cost real work: three rounds on #3063 built machinery to make an
eval prompt literally executable — a scratch git repo, a pinned config cascade,
unset GIT_DIR—before anyone established that nothing executes an eval prompt here. All of it was deleted. This
records the fact that would have prevented it.
Fix
Two Anthropic-owned eval formats exist and they are not the same.
The shipped one is
skill-creator's:evals/evals.jsoninside the skill directory, casescarrying
id/prompt/expected_output/files/expectations— which is whyevals.schema.json's owndescriptionnotes that upstream names that last fieldassertions. It isthe ecosystem-wide shape; a public code search returns thousands of
evals.jsonfiles in that formagainst a handful in any other.
claude plugin evalconsumes a different layout —<eval dir>/**/case.yaml, orprompt.mdplusgraders/*.md, withexperimental.evalsnaming the directory. This repo has none of it,deliberately: the command is early access and refuses to run.
Adopting its format today would trade a corpus CI checks on every PR for one nobody here can execute.
Adoption stays deferred behind the same
melodic-software/medley#1418tracker as the runner.The paragraph closes with the consequence authors actually need: because nothing executes a prompt,
a prompt must be readable and followable by a human or an agent working by hand, and must not assume
a runner will stage anything for it. That is the sentence #3063 needed and did not have.
Also reconciles a stale acceptance criterion.
docs/specs/write-for-agents-brief.mdnamed a"shipped
claude plugin evalsuite" as the artifact settling its trigger-families criterion, whilethe skill actually shipped in #3003 with the house format. The criterion now describes what gates it
in fact — a written specification checked by
check-evals-quality.shand exercised by hand per theplaybook's recipe — rather than a pass/fail command run that never happened.
Verification
The determination behind this was produced by a read-only investigation and then re-checked directly:
MIGRATION-PLAYBOOK.md"Evals — warrant policy and consumer-verify recipe": "no first-party command that executes model-graded evals today"plugins/evals/skills/design/SKILL.md— "does not execute evals", in its always-loaded descriptionevals.schema.jsondescriptioncitesskill-creator'sassertionsnamingplugin eval is currently in early access— reproduced aboveplugins/changedgit diff --name-only | grep -c '^plugins/'→ 0, so no version bumpmarkdownlint-cli2,typos, editorconfigDocs-only, two files, +21/−2.
Related
melodic-software/medley#1418— owns the deferred eval runner.What this does not settle
Whether
claude plugin evalpredates this repo's format could not be established: the publicanthropics/claude-codechangelog has zero entries for it through 2.1.238, and dating the npmpackage means bisecting a 331 MB binary. It does not matter for this decision, because the format's
provenance turned out to be a different Anthropic surface rather than that command — but the timing
claim is not made here, and should not be inferred from this text. Early-access availability may also
differ per account; this was tested on one.
Generated by Claude Code