docs(rules): exempt grader samples on synthetic prompts from the transcript rule

[kyle-sexton]

Refs: #6476

Summary

Adds one sentence to .claude/rules/eval-case-transcripts.md: output an eval run captures from a model answering a synthetic case prompt, such as a grader sample, is not a transcript under the rule, provided it holds no machine-specific path or identifying detail.

Fix

The rule as written could be read to bar committing grader samples (samples/*.json) that the #6476 rollout cases carry. The owner decided those are not transcripts; this records the decision where the rule lives.

Observed-stumble evidence (repeated, current models):

• feat(multi-agent): concise skill text measured by evals, plus outcome cases and probes #7402: a review bot raised a P1 asking to rewrite committed grader samples as "raw session transcripts" under this rule (thread).
• feat(animation): concise skill text measured by evals, plus outcome cases and probes #7427: the authoring session classified the same reading as VALID and rewrote seven grader samples from scratch on that basis (thread).

Verification

• Docs-only change to one rule file; no code or skill text changes.
• The rule's paths: globs are unchanged.

Related

• Concise skills: one skill-creation standard, evals, and a fleet-wide rollout #6476 (skill concision rollout, whose eval cases carry grader samples)

🤖 Generated with Claude Code

[claude]

Claude finished @kyle-sexton's task in 1m 29s —— View job

---

Security review

• Gather context (PR diff, changed file)
• Apply skip gate / instruction-surface lens
• Report findings

Scope: 1 file, .claude/rules/eval-case-transcripts.md, +4/-0. Adds a sentence exempting model
output generated against a synthetic eval case prompt (e.g. a grader sample) from the "no raw
transcript" rule, conditioned on holding no machine-specific path or identifying detail.

Instruction-surface check: This narrows a standing rule, so I checked it against the
instruction exception register.
The rule's core protection (real user chat logs / support tickets / session transcripts are never
committed to this public repo, in prompt, expected output, grader, or fixture) maps to the
secret-handling (PII) and external-publication Gate 0 classes — but this diff leaves that core
ban fully intact at

claude-code-plugins/.claude/rules/eval-case-transcripts.md

Lines 10 to 12 in 8f43688

	This repository is public. No eval case here holds a raw Claude Code session transcript or a raw
	product transcript (a chat log, support ticket, or log line from a real user), in a prompt, an
	expected output, a grader, or a fixture.

.
The new carve-out at

claude-code-plugins/.claude/rules/eval-case-transcripts.md

Lines 14 to 16 in 8f43688

	Output an eval run captures from a model answering a synthetic case prompt, such as a grader
	sample, is not a transcript under this rule, as long as it holds no machine-specific path or
	identifying detail.

only exempts model-generated output produced against a prompt that must itself still be
synthetic (unchanged by this PR) — it does not relax the ban on real prompts/content, and it
carries its own compensating condition (no machine-specific path or identifying detail). No
scenario in this diff reintroduces real-user data into the exempted path, so this reads as a
scope clarification of an existing owner decision, not a weakening of the protected class.

No security issues found. No inline comments posted.

claude-lane-summary: lane=claude-security-review sha=8f4368813f44a572047ddb2c41c9db4b2e0070db run=38507314986

[claude]

Claude finished @kyle-sexton's task in 1m 22s —— View job

---

CI code review

• Gather context (diff, full rule file, related eval samples/*.json usage)
• Review the one-line docs addition against code-review criteria
• Report findings

Summary

This is a 4-line, docs-only addition to .claude/rules/eval-case-transcripts.md: it exempts model output captured from answering a synthetic eval case prompt (e.g. a grader sample) from the "no raw transcript" rule, as long as it carries no machine-specific or identifying detail.

Checked:

• The new sentence doesn't widen the paths: frontmatter glob, matching the PR's own claim.
• The "grader sample" term lines up with the repo's existing evals/*/samples/*.json convention (e.g. plugins/evals/evals/design-criteria-first/samples/criteria-first.json), which already stores synthetic model answers to synthetic prompts — so the exemption describes real, pre-existing practice rather than inventing new terminology.
• PR body follows the pr-body-contract shape (Refs:, Summary/Fix/Verification/Related) and the title is valid Conventional Commits.

No correctness or maintainability issues found in the changed lines — nothing to block on.

claude-lane-summary: lane=claude-review sha=8f4368813f44a572047ddb2c41c9db4b2e0070db run=38507314930

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8f4368813f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

claude-review has reviewed this pull request through 63717c4; a later push is reviewed from there.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-14T03:40:52.608934Z	8f43688	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

claude-security-review has reviewed this pull request through 63717c4; a later push is reviewed from there.