Repository navigation
test(playgrounds): the eval suite runs without the plugin (dependency-unsatisfied) #12191
Description
Observed
The playgrounds plugin does not load in the claude plugin eval sandbox, so its eval suite measures nothing. plugins/playgrounds/.claude-plugin/plugin.json declares a cross-marketplace dependency, {"name": "playground", "marketplace": "claude-plugins-official"}. On 2026-11-01 lane 4's smoke runs of two converted playgrounds cases (#8635 group 1) showed plugin_errors: dependency-unsatisfied for playground@claude-plugins-official in every trace's startup record, and playgrounds was not in the loaded plugins. Both arms ran without the plugin: use-absent-emits-install-commands scored 0.00 with and without, use-cloud-session-feature-detects-delivery 0.67 with and without.
Fix wanted
Make the suite measurable, for example by staging the playground dependency in the case (a local stand-in plugin listed in the case's plugins:, the way songwriting's datamuse-path stand-in is), or by another way the eval sandbox accepts for a cross-marketplace dependency. Then one --runs 1 smoke run that shows the plugin loaded.
Related
- feat(evals): convert other plugins' evals.json files to claude plugin eval suites #8635
- run-validity issue filed alongside this one (a run like these should be INVALID)
Activity
github-actions commentedon Nov 2, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
#12191 (comment)
This decision settles the open questions. I read the issue as asking for eval-only
playgroundstand-ins so theplaygroundseval suite loads the plugin. The--runs 1smoke run is a host step and is routed separately. I found no duplicates.Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Nov 2, 2026
Not admitted to the issue lane: it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (
.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 2, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: confirmed; its comment says why.melodic-automation-lanes commentedon Nov 2, 2026
Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.
Intended outcome: Add a local stand-in plugin named
playgroundto the playgrounds eval cases, so theplaygroundsdependency is met and the plugin loads, then record one--runs 1smoke run showing it loaded.Related work
datamuse-pathstand-in is the model the issue cites.Open questions
playgroundplugin make the upstream plugin look present in every playgrounds case, or only in some? The existing cases assume it is absent, so a stand-in namedplaygroundwould defeat them.RECOMMENDED: Use two arms. Add an eval-only stand-in named
playground(a stub skill) to the cases that test the present path, such as use-routes-natural-explorer. Keep the use-absent-* cases as absent-path checks. Since the plugin cannot load while the dependency is unsatisfied, give those cases a way to run with the plugin loaded and the upstream skill absent. The owner has to choose how, for example a stand-in whose skill listing does not read as the upstream skill. Basis: plugins/playgrounds/evals/use-absent-install-scope/prompt.md:2 says the upstream plugin is absent in every eval run, and plugins/playgrounds/skills/use/SKILL.md:65-75 prints the install commands only when it is absent.dependenciesentry inplugin.json(for example a patched eval-only copy)? Editing the shipped manifest would change real installs.RECOMMENDED: Use the stand-in and leave the shipped manifest alone. The docs say a local plugin satisfies a dependency entry, including one that names a marketplace, on Claude Code v2.1.242 or later. Confirm the eval sandbox's Claude Code version is at least that. Basis: https://code.claude.com/docs/en/plugins/dependencies, section 'Test a plugin and its dependency locally'; plugins/playgrounds/.claude-plugin/plugin.json:17-22.
plugins:front matter, as songwriting does, and does the 1.1 case.yaml format used by the playgrounds cases support that field?RECOMMENDED: Yes. Use the
plugins: \["../..", "<stand-in dir>"\]front matter that the songwriting prompt.md files use. The playgrounds cases arecase.yamlfiles with a separateprompt.md, so the author must check that the field works the same way there. Basis: plugins/songwriting/evals/dg-audit-identity-rhyme/prompt.md:4 (plugins: \["../..", "datamuse-path"\]); plugins/playgrounds/evals/use-absent-install-scope/prompt.md:1-8 has nopluginsfield.RECOMMENDED: Require the run's startup record to show no
dependency-unsatisfiedplugin error and to listplaygroundsamong the loaded plugins, and record that in the PR. Do not wait for #12190. Basis: judgment: the issue's own evidence is the startup record'splugin_errorsand loaded-plugin list.Sources read
pluginsfieldclaude plugin list, then a skill-listing fallbackplugins: \["../..", "datamuse-path"\], the stand-in modelTo answer, post a comment that starts with this line:
Automated refinement by the lanes App. Answer with a comment whose first line is
**Decision recorded.**.kyle-sexton commentedon Nov 3, 2026
Decision recorded.
The suite gets eval-only stand-ins: a
playgroundstub with a skill for present-path cases, and aplaygroundstub with no skill for absent-path cases, listed in each case'splugins:; the shippedplugin.jsondependency is unchanged. Proof of load is a startup record with nodependency-unsatisfiederror andplaygroundsamong the loaded plugins. Basis: the body's Fix;plugins/playgrounds/.claude-plugin/plugin.json:17-22; the songwriting stand-in pattern atplugins/songwriting/evals/dg-audit-identity-rhyme/prompt.md:4. Written by the issue-triage session.playgrounddependency is staged in the case (a local stand-in plugin listed in the case'splugins:), or met another way the eval sandbox accepts for a cross-marketplace dependency.--runs 1smoke run shows the plugin loaded.github-actions commentedon Nov 3, 2026
The triage sweep dispatched intake triage for this issue (attempt 1 of 2).
kyle-sexton commentedon Nov 3, 2026
Agent choice, not an owner answer. The decision record above was written by the issue-triage session under the owner's standing direction §1 (decide from evidence). The owner did not answer this question; the comment appears under the owner's login only because the session posts with that account.
kyle-sexton commentedon Nov 3, 2026
Claim. Lane 3 host run is building this (hub overnight order).
kyle-sexton commentedon Nov 3, 2026
Smoke proof (Done-when item 2). One
--runs 1smoke ofuse-routes-natural-exploreron Claude Code 2.1.317, Sonnet: the startup record listsplaygroundsandplaygroundwithplugin_errorsempty, so nodependency-unsatisfiedand the plugin loaded. Stand-ins (Done-when item 1): #12653, all five cases. The run failedno-write(it invokedplaygrounds:use, then wrote the HTML itself); one run, a real suite signal, not a load failure.kyle-sexton commentedon Nov 3, 2026
Closed by #12653 (merge dc61e51, 14:40Z). Done-when (1): all five playgrounds cases list an eval-only
playgroundstand-in, and the shippedplugin.jsonis unchanged. Done-when (2): the--runs 1smoke showsplaygroundsloaded withplugin_errorsempty, see #12191 (comment)