Concise skills: one skill-creation standard, evals, and a fleet-wide rollout #6476

Activity

1. github-actions

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 7, 2026
2. github-actions commented on Oct 7, 2026

   

   github-actionsbot

   on Oct 7, 2026 – with GitHub Actions

   Contributor

   More actions

   Thanks for the report. Automated triage handed this issue to a maintainer.

   _{Automated triage. A maintainer reviews it.}
3. github-actions

   added

   needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

   on Oct 7, 2026
4. kyle-sexton commented on Oct 7, 2026

   

   kyle-sexton

   on Oct 7, 2026 – with Claude

   ContributorAuthor

   More actions

   Overlap map (Phase 0)

   There is one owner per rule. Every other site that states the rule becomes a pointer edit, made in the phase that owns its plugin. Built from git grep on 2026-10-07 at 311b50704. /playbooks:skill-authoring refers to its new reference/skill-criteria.md.

   #	Rule	Single owner (rule / enforcement)	Other sites that state it (work item)
   1	Description shape (third person, "Use when", one trigger per branch, user-invoked one-liner)	/playbooks:skill-authoring ## Descriptions / /skill-quality:check	docs/conventions/invocation-mode/README.md (Phase 1); /docs-hygiene:write-for-agents (Phase 4); naming:name-it-better context/sources.md (fleet: naming)
   2	Size: description and body caps, Pocock max	/playbooks:skill-authoring ## Descriptions, ## Body / /skill-quality:check constants	docs/migration-playbook.md (Phase 1); harness-config audit and audit-instructions criteria (Phase 2); docs-hygiene audit-progressive-disclosure and extract-ssot (Phase 4)
   3	Metadata keys need a consumer	/playbooks:skill-authoring ## Metadata / /skill-quality:check registry	none that states the rule (keys are only used); unconsumed keys are cleaned up per plugin in the fleet
   4	History notes stay out of instruction files	/playbooks:skill-authoring ## History and provenance / SKILL.md: /skill-quality:check; agents/*.md: /harness-config:audit-instructions	/docs-hygiene:audit-noise ticket-pr-residue (Phase 4 routes SKILL.md and agents to the owners above); /code-tidying:audit-comment-residue covers code comments, a separate domain and not a duplicate
   5	Reference files one level deep	/playbooks:skill-authoring ## Reference files	docs-hygiene extract-ssot and its context/, audit-progressive-disclosure context/tier-model.md (Phase 4)
   6	Progressive disclosure (what goes in SKILL.md vs references)	rule: /playbooks:skill-authoring ## Body; audit: /docs-hygiene:audit-progressive-disclosure	plugin-quality:audit reference/component-types (fleet: plugin-quality); playbooks:boris is vendored and exempt
   7	Skill listing budget (per-entry truncation, total budget)	/harness-ops:audit-skill-visibility	harness-ops:audit-native-overlap, harness-memory:audit criteria, context-budget:audit, harness-config audit and audit-pass (fleet, per plugin); /skill-quality:check keeps the per-entry constant as enforcement
   8	Deletion test ("could the model already know this?")	/playbooks:skill-authoring ## Body, citing the claude-api prompt-audit guide	harness-memory:audit and instruction-placement:realign apply it to other files: these become pointers in the fleet. architecture:improve's "deletion test" is about modules, not prose, so it is not a duplicate

   Also from Phase 0:

   • Fleet before-snapshot: taken, kept locally. It is the baseline the "no new FAIL" check reads.
   • /playbooks:update drift check for skill-authoring: frontmatter version 1.0.0 matches upstream, but the vendor content has drifted (SHA256 7b0923d8… locally vs 05b4e6b7… upstream). This needs a separate sync PR and is not part of this rollout.

   ---

   Generated by Claude Code
5. kyle-sexton

   added 3 commits that reference this issue on Oct 7, 2026

   feat(skill-quality): skill-criteria standard, WARN checks and rewrite…

   ...

   13a1967

   feat(playbooks): let evals decide skill cuts in the skill criteria (#…

   ...

   cc3c4f1

   feat(code-tidying): concise skill text measured by evals, plus outcom…

   ...

   f5915e6
6. kyle-sexton

   added sub-issues on Oct 7, 2026

   • Concise skills rollout: discipline (18 skills, 16 model-invoked) #6548
   • Concise skills rollout: session-flow (17 skills, 15 model-invoked) #6549
   • Concise skills rollout: harness-ops (15 skills, 12 model-invoked) #6550
   • Concise skills rollout: architecture (12 skills, 11 model-invoked) #6551
   • Concise skills rollout: planning (12 skills, 11 model-invoked) #6552
   • Concise skills rollout: songwriting (11 skills, 10 model-invoked) #6553
   • Concise skills rollout: docs-hygiene (10 skills, 9 model-invoked) #6554
   • Concise skills rollout: harness-config (10 skills, 9 model-invoked) #6555
7. 54 remaining items

   Load more
8. kyle-sexton

   added sub-issues on Oct 7, 2026

   • Concise skills rollout: naming (1 skills, 1 model-invoked) #6611
   • Concise skills rollout: playgrounds (1 skills, 1 model-invoked) #6612
   • Concise skills rollout: repo-fleet-hygiene (4 skills, 1 model-invoked) #6613
   • Concise skills rollout: repo-hygiene (2 skills, 1 model-invoked) #6614
   • Concise skills rollout: tdd (1 skills, 1 model-invoked) #6615
   • Concise skills rollout: user-interface (1 skills, 1 model-invoked) #6616
   • Concise skills rollout: writing (1 skills, 1 model-invoked) #6617
   • Concise skills rollout: x (1 skills, 1 model-invoked) #6618
   • Concise skills rollout: the 15 check-and-setup plugins #6619
9. kyle-sexton

   added the

   Task

   issue type on Oct 12, 2026
10. kyle-sexton

    removed

    needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

    on Oct 12, 2026
11. github-merge-queue

    added 2 commits that reference this issue on Oct 14, 2026

    docs(rules): exempt grader samples on synthetic prompts from the tran…

    ...

    bdcdc6a

    feat(session-flow): concise skill text measured by evals, plus outcom…

    ...

    ea0f9aa
12. kyle-sexton commented on Oct 14, 2026

    

    kyle-sexton

    on Oct 14, 2026

    ContributorAuthor

    More actions

    Batch 1 status (2026-10-14)

    Batch 1 of the rollout is complete. Each child shipped through a PR backed by Opus and Sonnet before/after outcome runs and held-out trigger runs. Any cut that lost meaning or measured a drop it could not explain was restored or reverted to main.

    Child	PR	Result
    #6550 harness-ops	#7434	merged; 5 bodies cut, 6 reverted, --root fix; grader follow-ups in #7435
    #6548 discipline	#7441	merged; 13 of 16 skills cut, 3 keep their old description; follow-ups in #7440
    #6549 session-flow	#7442	merged; 13 skills cut; recall graders tightened after measurement, re-run pending
    #6551 architecture	#7437	merged; 11 descriptions cut, 3 bodies cut, 8 bodies at main; follow-ups in #7438
    #6552 planning	#7465	in the merge queue; 10 skills cut, interview and setup at main

    Rule change: #7433 merged, so grader samples on synthetic prompts are not transcripts.

    Lessons for the remaining children

    • Audit graders before the freeze for these patterns:
      • negations right before a name ("don't use X" passing)
      • one match where several facts are required
      • write guards that miss cp, mv, install, dd or interpreter writes
      • Read-tool access to files the case forbids
      • wording too narrow for the skill's own text
    • Pass --keep-temp so failing traces can be read. Delete only the eval dirs your own results name; they are read-only until chmod u+rwx.
    • Never commit credential-shaped fixture strings, not even in an intermediate commit: GitGuardian scans every commit in the PR. Use CANARY-1234 markers.
    • Relative links in fixtures must resolve (lychee-offline). Don't leave TODO markers in fixtures either: the comment-marker scan fails lint-repo.
    • Every review thread must be resolved before merge. After two rounds of regex-depth nits on a grader, decline with the reason and track it in an issue.
    • The display-redaction timing test (display-redaction: linear-time mask test times out at 5000 ms in CI #7273) fails some queue runs. Re-queue when it does.
    • Docker Desktop recreates the ~/.docker symlinks, which blocks evals that grant Bash. Check before each batch.

    Remaining

    • 50 children not started (listed below this issue). Run at most two at a time.
    • feat(evals): compress plugin-eval/SKILL.md (v2) with a same-day confirm run #5956 and test(education): close eval-case gaps found in the concise-skills review #7416 run inside the rollout.
    • Optional: re-run feat(session-flow): concise skill text measured by evals, plus outcome cases and probes #7442's two recall cases under the tightened graders.
13. github-merge-queue

    added a commit that references this issue on Oct 14, 2026

    feat(planning): concise skill text measured by evals, plus outcome ca…

    ...

    39d3e65