Repository navigation
test(visualization): fix the frozen suite's graders and re-measure present and visualize with VALID runs #11628
Description
Context
#11623 (concise-skills slice for visualization, #6595) ships a shorter present description and a frozen eval suite. Its fresh-context code review found that the evidence behind the description cut, and several graders in the frozen suite, do not meet the repository's measurement rules. The suite is frozen for that PR, so the fixes and the re-measure land here.
- Validity.
run-validity.pyjudged all 8 trigger passes and 3 of 8 outcome passes INVALID.plugins/playbooks/skills/skill-authoring/reference/skill-criteria.mdMeasurement step 4 says to read a number only from a run/evals:plugin-evaljudges valid, andrewrite-a-plugin.mdsteps 4-8 require VALID results and passes launched together. The passes ran back to back. One INVALID reason is theskill firedcheck that fix(evals): let trigger-measurement probe runs pass run-validity #11618 covers; the other is sandbox Glob denials outside the plugin directory with no denial-free comparison runs. - Graders in
plugins/visualization/evals/that reject outcome-meeting replies or accept failing ones:viz-ambiguous-coderecommended-first: themflag lets^anchor at any bullet, so a-/*bullet holding "recommend" passes in any position.viz-ambiguous-codeasks-one-question: only checks for a?;renders-nothingmisses an indented tree without box characters.pres-k2-instructiontitle-not-hijacked: misses- **Title:** ...,Title slide: ...and numbered bold titles; fails a reply that quotes the injection under a heading.pres-outline-recordspeaker-notes: needs 2 matches whereexpected_outcomeasks for notes on every slide (at least 3), and counts prose like "your notes:"; the skill says "any speaker notes".viz-comparison-tablenames-rungandviz-pr-diff-terminalreason-given: still reject replies the PR's "Grader-miss rulings" list as meeting the outcome.pres-missing-figure: no grader catches an invented customer quote;blank-markedpasses on one[__].
- Sample.
pres-k2-instruction/samples/reports-instruction.jsonfirst fail answer starts mid-sentence (" is available here, ...").
Proposed work
- Rewrite the graders above against each case's
expected_outcome, add the missing pass and fail samples, and fix the truncated sample. - Once fix(evals): let trigger-measurement probe runs pass run-validity #11618 lands, re-run the
presentandvisualizetrigger and outcome passes concurrently, with denial-free runs or the denial cause removed, on a fresh held-out set. - Keep the
presentdescription only if the VALID re-measure shows no loss; otherwise revert it. Re-decide the reverted bodies and thevisualizedescription on the new numbers.
Acceptance criteria
-
validate-cases.py plugins/visualization/evalsexits 0 with a sample for each shape listed above. - Every trigger and outcome pass the decision reads from is judged VALID by
/evals:plugin-eval. - The
presentdescription on main is backed by that VALID measurement, or reverted.
References
- PR #11623, sections "Grader-miss rulings" and "Validity".
- Concise skills rollout: visualization (2 skills, 2 model-invoked) #6595, Concise skills: one skill-creation standard, evals, and a fleet-wide rollout #6476 (umbrella), fix(evals): let trigger-measurement probe runs pass run-validity #11618 (
run-validity.pyand emitted probe suites).
Metadata
Metadata
Assignees
Labels
Type
Fields
No fields configured for Task.
Activity
github-actions commentedon Oct 31, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
#11628 (comment)
The decision settles the open questions. The graders, samples and truncated sample are done, and #11618 no longer blocks. What remains is the concurrent VALID re-measure of
presentandvisualize, which needs hostclaude plugin evalruns. That is host-run work, not a decision for a person.Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Oct 31, 2026
Not admitted to the issue lane: it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (
.github/actions/resolve-config/admission-rules.json).fix(visualization): keep the present description until a valid trigge…
kyle-sexton commentedon Oct 31, 2026
Two Codex findings on #11623 deferred here (the suite is frozen in that PR), with the samples to add:
viz-ambiguous-codeasks-one-question(thread):\?[\s\S]*$passes a multi-question reply such asWhich form do you want? What should it focus on?followed by a recommended-first menu. Add that as a fail sample and enforce a single question.viz-ambiguous-coderenders-nothing(thread): a fenced ASCII call tree (PostalCarrier.quote/+-- RateTable.services_for/+-- RateTable.price) after the clarification menu passes all three graders. Cover ASCII trees and the other code-shape forms the case offers, with fail samples.melodic-automation-lanes commentedon Oct 31, 2026
This records a re-fire of an earlier refusal, not a new refusal. The lanes kill switch stopped run 41226504820, the work-resolve-issue run that
agent-readystarted, before it did any work; that run ended at 2026-10-31T18:06:14Z. work-unblock has now removed and addedagent-readyback to start a fresh run. A kill-switch refusal is no fault in this issue or the lanes, so it never counts against later re-fires; after limits.limit-stops-per-issue of them since a person last addedagent-readyor a run last reported, work-unblock waits longer before each one, up to 24 hours.8 remaining items
melodic-automation-lanes commentedon Nov 1, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: confirmed; its comment says why.test(debugging): keep only the frozen debug probes; defer the body re…
kyle-sexton commentedon Nov 1, 2026
Decision recorded. The grader and sample work is on
mainthrough #11811, and theno-invented-quotefix has been on main since #11936 merged it (7d60541). Thepresentdescription onmainis the reverted before text. What is left is a host run ofclaude plugin eval: the concurrent VALID re-measure ofpresentandvisualize.This is the Issue Triage session's decision on the evidence. It answers what #11811 left, which the admission comment of 2026-10-31T16:31:04Z asked for.
Are the graders, samples and truncated sample done? Yes. test(visualization): fix the frozen suite's graders against each case's expected outcome #11811 rewrote every grader the body lists, added their pass and fail samples, added the
no-invented-quotegrader topres-missing-figure, and fixed the truncatedreports-instruction.jsonsample. Both shapes from the 2026-10-31T16:35Z comment are onmainas samples: the two-question reply (Which form do you want? What should it focus on?) and the fenced+--ASCII call tree.asks-one-questionnow requires exactly one?.Basis:
git show --stat dfeb40cf17(test(visualization): fix the frozen suite's graders against each case's expected outcome #11811,Refs: #11628, touches every case and sample the body names);plugins/visualization/evals/viz-ambiguous-code/case.yaml:6-7(pattern: '\?',match: 'count:1');viz-ambiguous-code/samples/asks-one-question.json:18andsamples/renders-nothing.json:18on origin/main;python3 -I plugins/evals/skills/validate/scripts/validate-cases.py --json plugins/visualization/evalson an origin/main export (a175b78) printed[], exit 0.Is the first acceptance criterion met? Yes, as of origin/main a175b78, by the validator run above. test(visualization): read bracketed quote placeholders as placeholders #11936 (7d60541, merged 2026-11-01T14:30Z) changed
no-invented-quoteagain, after 12 real runs at Claude Code 2.1.316 showed it failing 5 bracketed placeholders. 4 of those 5 were in the without-plugin arm, which inflated the delta. The validator still exits 0 on main after that merge.Basis: the validator run above, on origin/main a175b78, which includes 7d60541;
git show -s 7d60541fec(Refs: #11628, "Summary").Is fix(evals): let trigger-measurement probe runs pass run-validity #11618 still a blocker? No. fix(evals): judge invocation-probe runs by their measured rate in run-validity #11639 closed it on 2026-10-31.
Trigger passes. The invocation-probe rules fix(evals): judge invocation-probe runs by their measured rate in run-validity #11639 and its successors added are on
main:run-validity.py:55-57;check_firedat:1014, probe tally at:1035-1058).:609-654).measure-invocation.sh:600).That removes both INVALID causes test(visualization): outcome eval cases and trigger probes for present and visualize #11623 reported for the trigger passes.
Outcome passes. The cause is not removed for these.
~,~/.claude,/usr/binand the sandbox root. A pass holds more runs than that, so (by inference) no such denial was shared by every run.in_every_run,:888-906) warns only when the same named call was denied in every run of the case, in both arms.:901). The sandbox-root denial is such an ancestor.The body's condition, "with denial-free runs or the denial cause removed", therefore still applies to the outcome passes.
Basis:
git show -s --format=%B 97a86a879a(Closes #11618);gh api repos/melodic-software/claude-code-plugins/issues/11618(state: closed,completed, 2026-10-31T17:10:58Z);git show -s --format=%B 41164a9628(fix(skill-quality): run emitted invocation probes without the read-only search tools #11755,Closes #11624: "Probe runs now pass--disallow-tools Read Glob Grep");git show -s --format=%B b8d717429c(test(visualization): outcome eval cases and trigger probes for present and visualize #11623 "Validity": the denied paths, "1 to 3 per pass, in both versions");python3 -I -m unittest discover -s plugins/evals/skills/plugin-eval/scripts -p test_run_validity.pyon the origin/main export: 99 tests OK.Is the third acceptance criterion met? Yes, by its "or reverted" branch. test(visualization): outcome eval cases and trigger probes for present and visualize #11623 committed the shorter
presentdescription and reverted it before merge, somaincarries the before text. That revert followed from INVALID passes, not from a measured keep decision, so it closes nothing on its own.Basis:
git show -s --format=%B b8d717429c(test(visualization): outcome eval cases and trigger probes for present and visualize #11623: "the shorterpresentdescription was committed, then reverted in3e4f215e66");git log origin/main -- plugins/visualization/skills/present/SKILL.mdshows no change after 4b9aada (feat(adhd): clarify resolves the rendered-views medium before it publishes #11454, 2026-10-31T06:00Z), which predates test(visualization): outcome eval cases and trigger probes for present and visualize #11623.What is left? The second acceptance criterion, and the re-decision the body's Proposed work names. Both need
claude plugin evalruns, which the issue lane cannot make, so this issue is host-run work with no lane criterion. Steps, in order, for the host session:plugins/visualization/evals/on amainthat includes 7d60541, and freeze it.presentandvisualizeoutside the repository (plugins/playbooks/skills/skill-authoring/reference/rewrite-a-plugin.mdOrder of work item 3, "Probes"). Write it in a session without the visualization plugin installed, so the writer's skill listing does not show the current descriptions (fix(evals): stop outcome passes failing validity on a required user-scope read and on forked skills reporting 0 turns #11727, 2026-11-01T14:07Z comment, item 2).--runs, judge and threshold, and--keep-tempfor every pass;/skill-quality:check measure-invocationemit-plugin-eval, and run with the line its README writes./evals:plugin-evaljudge every pass. A decision reads only from VALID passes. If an outcome pass fails the denial check, remove the denial's cause or rerun until each case has denial-free runs, as the body says. Launch any rerun together with its before control.presentdescription only if the VALID numbers show no loss. Re-decide thepresentandvisualizebodies and thevisualizedescription on the same numbers, and record the per-skill decision table in the pull request.Basis:
rewrite-a-plugin.md:39-55(items 3 and 5); the body's second and third Proposed work bullets.No
- [ ]criterion is added here: every remaining item is a run the issue lane cannot make.kyle-sexton commentedon Nov 1, 2026
kyle-sexton commentedon Nov 1, 2026
Host re-measure done (lane 1, melo-lap-001 WSL, 2026-11-01):
presentkeeps main's before description; nothing else changes, so no PR.Frozen suite:
plugins/visualization/evalstreecc8ed9f83b0cca3302a5193da56e9ce774fc4733on mainbbdfccecd2(includes 7d60541;validate-cases.pyexit 0), Claude Code 2.1.316. Fresh held-out trigger set, 16 queries per skill, written by a tool-lessclaude -p --safe-modesession with no plugin loaded (measure-invocation.sh validatePASS). The plan was written before any number. Every pass launched together per group with the same flags (--runs 3 --judge-model opus --threshold 0.8 --keep-temp).Read ~/.claude/rendered-views.md(1-3 runs)Decision table (trigger fired / 24 runs):
presentdescriptionvisualizedescriptionpresent,visualizebodiesThe decision reads only from VALID passes (criterion 2). The one VALID outcome pass (after Sonnet, skill text equal to main except the description line) shows the plugin delta +0.32 [+0.11, +0.52] vs no plugin. No before/after outcome pair is VALID, and the decision does not read one. Criterion 3:
presenton main is the before text, backed by the trigger measurement above.Follow-up: the outcome-pass INVALID cause,
visualizereading the user-scope~/.claude/rendered-views.mdinside the eval sandbox, belongs to #11727 ("a required user-scope read"), which lane 1 holds. Sonnet's lowvisualizefiring rate (4/24, 3/24) matches #11623.Closing as complete.