[kyle-sexton]

Closes #6595
Refs: #6476

Summary

This is the visualization slice of the concise-skills rollout, run through the per-plugin runbook (/playbooks:skill-authoring, reference/rewrite-a-plugin.md). A cut ships only with eval evidence that nothing was lost.

• No skill text changes. The shorter present description scored the same held-out trigger counts on both models, but every trigger pass behind it was judged INVALID by /evals:plugin-eval's validity check, and skill-criteria.md Measurement step 4 reads a number only from a run judged valid. So its before description ships. The re-measure is test(visualization): fix the frozen suite's graders and re-measure present and visualize with VALID runs #11628.
• Both bodies and the visualize description stay as they were. Each shorter version failed the pre-registered rule on Opus.
• Six new outcome eval cases under evals/, three per skill (pres-*, viz-*). All are graded deterministically (regex), and each grader has offline pass and fail samples.
• Frozen trigger-probe sets under probes/, 18 queries per skill.
• No run artifacts are committed: no held-out trigger set, emitted trigger cases, run results, traces, plugin copies or run scripts.

Fix

• No skill file changes: the shorter present description was committed, then reverted in 3e4f215e66 (see Summary).

• plugins/visualization/evals/: six cases written by a fresh agent from the before contract, frozen before any after-version run. A smoke pass on the before version classified three failures:

  • Two were grader defects. names-rung and reason-given rejected replies that met the outcome. Their regexes were widened, with the observed replies added as pass samples and near-misses as fail samples.
  • One was a case defect. viz-ambiguous-code's paste had one dominant form, so the skill was right not to ask. Its prompt was rewritten so three code-shape forms fit, and a re-smoke confirmed it.

• The meaning diff (/docs-hygiene:compress compare, two independent labelers per skill) found three losses. Each was restored before any after-version run:

  • present's Genre: decks. identifier.
  • the visualize body's trigger examples ("show me …", bare /visualization:visualize).
  • "legend and dark-mode contrast" in the chart-craft boundary.

  None of the restored text ships, because both bodies reverted.

• Changelog fragment .changes/visualization/feat-6595-concise-visualization-36723930.md, patch bump.

Verification

Per-skill decision table

Description: held-out positives that fired (out of 8) and negatives that stayed quiet (out of 8), one run per query. Body: mean outcome-case score over 3 runs, scored with the plugin loaded. The decision rule was written before any run: after ships only if after >= before on trigger counts, and after >= before - 0.05 on every case, on both models (claude-opus-5-5, claude-sonnet-5-5).

Skill	Description	Opus pos b / a	Sonnet pos b / a	Neg quiet b / a (Opus, Sonnet)	Body	Case	Opus body b / a	Sonnet body b / a
present	before (counts tied; passes INVALID)	7/8 / 7/8	7/8 / 7/8	8/8 / 8/8, 8/8 / 8/8	before	pres-k2-instruction	0.778 / 0.889	0.778 / 0.778
						pres-missing-figure	1.000 / 1.000	1.000 / 1.000
						pres-outline-record	0.889 / 0.778	1.000 / 1.000
visualize	before	6/8 / 4/8	0/8 / 0/8	7/8 / 8/8, 8/8 / 8/8	before	viz-ambiguous-code	1.000 / 1.000	0.667 / 1.000
						viz-comparison-table	0.889 / 0.778	1.000 / 1.000
						viz-pr-diff-terminal	0.889 / 0.778	1.000 / 1.000

• present description: before ships. The counts tied (the same held-out query, pos-val-02, missed on both versions and both models), but these trigger passes were judged INVALID, so under step 4 they cannot carry a ship decision.
• visualize description reverts. On Opus the shorter description missed pos-val-01 and pos-val-04, which the original fired on. Sonnet fired on none of the eight visualize positives with either description: it answered directly, with no Skill call and no denial. So Sonnet cannot separate the two versions here, and the decision rests on Opus.
• present body reverts.
  • pres-outline-record fell from 0.889 to 0.778 on Opus: the speaker-notes grader failed in 2 of 3 after runs against 1 of 3 before runs.
  • Section 1 (the outline record) is unchanged between the versions, so no restore applies.
  • The two-version noise report says present Opus is "not shown non-inferior at margin 0.05" (lower end -0.13) and present Sonnet is "non-inferior at margin 0.05".
• visualize body reverts.
  • On Opus, viz-comparison-table and viz-pr-diff-terminal each fell from 0.889 to 0.778, one more grader miss each.
  • The two directives they measure are the same in substance in both versions: "name the deciding rung … shipped default" and "say in one line why it was not produced". No restored text could close the gap.
  • The noise report says "not shown non-inferior at margin 0.05" on Opus (-0.15) and Sonnet (-0.11), with no Sonnet case dropping.

Grader-miss rulings

None were applied: the decisions stand on the pre-registered scores. A post-run read found misses, in both versions, on replies that meet the case's expected_outcome:

• names-rung rejected "That's the default when nothing else is set", "I used the default for a small comparison", and before run 3's "That was the default choice".
• reason-given rejected one-line reasons naming the other repository's PR diff and the escape helper, in both versions.
• pres-outline-record's expected_outcome asks for speaker notes on every slide, but the skill says "any speaker notes", so speaker-notes passes or fails by chance.

With 3 runs, one grader flip moves a case by 0.111, more than the 0.05 margin, so these graders cannot tell noise from loss. A follow-up round should fix them and re-measure the bodies with a fresh held-out set.

Validity

16 measured passes (8 trigger, 8 outcome) plus 2 smoke passes, run one at a time on Claude Code 2.1.316. No pass hit a run error, timeout, cost ceiling, or session or usage limit, and none was partial.

run-validity.py judged 5 outcome passes VALID. It judged 3 outcome passes and all 8 trigger passes INVALID, for one of two reasons:

• sandbox Glob denials on paths outside the plugin directory (~/.claude, ~, /usr/bin, the sandbox root), 1 to 3 per pass, in both versions;
• should-trigger queries that did not fire, which is the quantity being measured.

Under the rule set before the runs, only run errors and timeouts invalidate a pass.

Passes ran back to back in before/after pairs, not concurrently, because this lane allows one claude plugin eval at a time. All 16 ran within 49 minutes.

After the passes, two eval prompts each got one blank line so markdownlint passes (MD003, MD032). This is whitespace only. The passes ran on suite digest 774bd872…; the committed suite is 2e3185c1….

The held-out trigger sets label one negative per skill train, only to satisfy the probe validator. All 16 queries per skill were scored.

Unmeasured

• present: the Artifact quickstart call and type_url, the check-deck.mjs gate, publish and share, and the line naming which layer supplied the medium. The eval sandbox has no claude.ai Artifact tool, and the cases grant only Skill and Read. The cases grade only the outline-only fallback.
• visualize: these need Bash, Write or tools the sandbox lacks, so the meaning diff covers them:
  • the mktemp local-file placement;
  • the mermaid-gate.mjs call and the mermaid-source form;
  • the call-tree and file-tree picks;
  • the rendered-views cascade layer verdicts;
  • Artifact publishing and the three-row-table rule under artifact;
  • the design canvas, ClaudeDesign routing, and the playground boundary.
• skills/*/evals/evals.json (the skill-creator suites) is unchanged and was not run.

Checks on this branch

• scripts/check-affected.sh --tests: 14 passed, 2 skipped over 34 changed files.
• /evals:validate (validate-cases.py) on plugins/visualization/evals: exit 0, every sample sorts correctly.
• /skill-quality:check plugins/visualization/skills: PASS, with no FAIL on either side.
  • Before: present 1 warning (description 498 chars), visualize 5.
  • After: present 0, visualize 5 (unchanged text).
• measure-invocation.sh validate on probes/: PASS, 0 warnings.
• check-changelog-fragments.sh --check-required origin/main: pass.
• Diff scanned for secrets and machine-specific paths: none found.

Left out: the Artifact, publish and Bash-dependent paths listed under Unmeasured (the eval sandbox cannot run them); the skill-creator evals.json suites (not part of this protocol).

Related

• Umbrella: Concise skills: one skill-creation standard, evals, and a fleet-wide rollout #6476 (concise-skills rollout). Precedent: feat(writing): concise be-concise body measured by evals, plus outcome cases and probes #8459 (writing).
• Follow-up: test(visualization): fix the frozen suite's graders and re-measure present and visualize with VALID runs #11628 (grader fixes and a VALID re-measure).

Ready review findings

A fresh-context code review ran at ready in place of the CI code-review lane (venue local). Every finding is VALID (defer) to #11628, because the suite is frozen and measured and skill text changes only with new eval evidence:

• The present description cut reads trigger passes run-validity.py judged INVALID (see Validity), against skill-criteria.md Measurement step 4. Resolved in this PR: the cut is reverted (3e4f215e66), and test(visualization): fix the frozen suite's graders and re-measure present and visualize with VALID runs #11628 re-measures it.
• Grader defects: recommended-first, asks-one-question and renders-nothing, title-not-hijacked, speaker-notes, names-rung and reason-given, and no invented-quote grader in pres-missing-figure.
• One truncated fail sample in pres-k2-instruction/samples/reports-instruction.json.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VcuponPYh1riXtUL5WmMiY

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-31T16:33:54.963183Z	e0cee8f	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

claude-security-review reviewed 3e4f215e66e4f91bcf19b22d4489d708786b579a: 0 finding(s), 0 as inline comments.

Incremental security review of .changes/visualization/feat-6595-concise-visualization-36723930.md since e0cee8f. This pull request's remaining 32 files are unchanged since that commit and were reviewed then.

The change is a changelog fragment reword. The heading goes from ### Changed to ### Added, and the text now says the descriptions and bodies of both skills are unchanged. The bump: patch front matter is untouched. The file is plain Markdown. It is not executable, not a trusted path, not a plugin manifest and not a standing instruction. It touches no token, trust control, hold or userConfig key.

The skip gate's second condition applies: the change has no security-relevant surface. I did not run a fan-out.

No spend: no billing, plan, or model/effort/turn-limit change.

critical-findings: 0

[github-actions]

claude-review: 1 inline finding(s) on e0cee8ff2498a4c0e5cac044f435d50b31ec35d5; the summary is in this lane's pull request comment.

claude-lane: claude-review

[github-actions]

claude-review reviewed 3e4f215e66e4f91bcf19b22d4489d708786b579a: 0 finding(s), 0 as inline comments.

Incremental review of .changes/visualization/feat-6595-concise-visualization-36723930.md: no findings.

The fragment now says ### Added and no longer claims a shorter present description. That matches the PR body: the description cut was reverted in 3e4f215e, and the skills' descriptions and bodies are unchanged. The text accurately describes the six eval cases and the probes. The patch bump is consistent with a change that adds only evals and probes. This change doesn't touch any trusted path, and it has no cost or security impact.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e0cee8ff24

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".