Concise skills rollout: wizard (2 skills, 2 model-invoked) #6596

Activity

1. kyle-sexton

   added a parent issue on Oct 7, 2026

   • Concise skills: one skill-creation standard, evals, and a fleet-wide rollout #6476
2. github-actions

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 7, 2026
3. github-actions commented on Oct 7, 2026

   

   github-actionsBot

   on Oct 7, 2026 – with GitHub Actions

   Last edited by melodic-automation-lanes

   Contributor

   More actions

   Proposed work class: work-class: scoped
   Basis: automated intake triage; the lanes App applied this class.

   Size: Medium
   Complexity: Medium
   Uncertainty: Known
   Flags: none
   Blocked by: none

   Understood: rewrite the two model-invoked wizard skills (generate, unattended) to the skill criteria in one draft PR, following the per-plugin runbook. That includes adding probe files, running the skill-quality and eval checks, a per-skill decision table in the PR body, and a version or changelog entry. The scope and done-when items are clear. Eval spend is recorded as pre-approved within prepaid credit. No likely duplicates found in the open issues. This is part of #6476.

   _{Automated triage, applied by the lanes App.}
4. github-actions

   added

   needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

   on Oct 7, 2026
5. kyle-sexton

   added the

   Task

   issue type on Oct 12, 2026
6. kyle-sexton

   removed

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 12, 2026
7. kyle-sexton

   removed

   needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

   on Oct 24, 2026
8. kyle-sexton commented on Oct 24, 2026

   

   kyle-sexton

   on Oct 24, 2026

   MemberAuthor

   More actions

   Released by the owner's HITL session (2026-10-24, C12): it waited only on a removed gate. Intake triage re-runs on it to set its fields, then admission takes it.
9. github-actions commented on Oct 25, 2026

   

   github-actionsBot

   on Oct 25, 2026 – with GitHub Actions

   Contributor

   More actions

   The triage sweep dispatched intake triage for this issue (attempt 1 of 2).
10. github-actions

    added

    priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

    on Oct 25, 2026
11. melodic-automation-lanes

    added

    status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

    work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

    admission-deferred

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

    and removed

    admission-deferred

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

    on Oct 25, 2026
12. melodic-automation-lanes commented on Oct 25, 2026

    

    melodic-automation-lanesBot

    on Oct 25, 2026

    Contributor

    More actions

    The work-resolve-issue lane stopped partway through this issue and needs a person.

    Why, in the run's own words:

    The issue requires the rewrite protocol in plugins/playbooks/skills/skill-authoring/reference/rewrite-a-plugin.md and skill-criteria.md 'Measurement', which ships only parts that were measured: a before/after /skill-quality:check run, eval cases written through /evals:design and checked with /evals:validate, probe sets checked with /skill-quality:check measure-invocation, paid /evals:plugin-eval passes for each version on each target model, and a /docs-hygiene:compress compare meaning diff. This CI lane loads only the work-items plugin, so none of those skills is available. Bash runs only the named wrappers, so it cannot run claude plugin eval or spend API credit. Rewriting without measurement would break the protocol's 'ship only parts that were measured' rule. Writing the probe files here would also skip the required checks and break the rule that keeps probes away from the rewriter. Nothing was changed. The issue needs a session that has the evals, skill-quality and docs-hygiene plugins plus eval API credit.

    What the run says it did not do:

    • /skill-quality:check plugins/wizard before/after comparison: the skill-quality plugin is not loaded in this lane
    • /evals:validate on plugins/wizard eval suite and outcome cases via /evals:design: the evals plugin is not loaded and paid eval passes cannot run here
    • Probe files plugins/wizard/probes/generate.json and unattended.json: the protocol needs a fresh-agent probe set checked with /skill-quality:check measure-invocation, which is unavailable here
    • Per-skill decision table with before/after numbers: no measurement possible in this lane
    • Plugin release record (version or changelog fragment): nothing shipped, so none added

    Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/40190522123

    To retry, remove needs-human and add agent-ready again.
13. kyle-sexton

    marked this as blocked on Oct 27, 2026

    • feat(ci): validate, gate and summarize dispatched plugin eval runs #9704
14. kyle-sexton commented on Oct 27, 2026

    

    kyle-sexton

    on Oct 27, 2026

    MemberAuthor

    More actions

    This was generated by AI.

    Parked for capacity, not for the owner. This rollout's remaining work is the measured eval passes of the batch-1 protocol, which the issue lane cannot run: it cannot start claude or claude plugin eval (see the lane's escalation on #6585, https://github.com/melodic-software/claude-code-plugins/actions/runs/40186335064). It waits on the plugin eval dispatch work in #9704, now a blocked-by edge, so admission holds it until #9704 closes.

    Under the owner's 2026-10-26 stop rule, a person is needed only for a change in intent or vision, an implementation finding that changes the intent or primary design, or a high or critical security issue; host, attended and eval runs are agent work. So needs-human comes off, and the issue moves to host-run with status: confirmed, which keeps the issue lane from taking it when #9704 closes: #9704 adds an eval dispatch workflow, not eval capability for the issue lane.
15. kyle-sexton

    added

    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.Needs an unattended run on a host; the host runner takes it, not a person or the issue lane.

    status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.A maintainer confirmed the item is real and in scope; ready to be worked.

    and removed

    status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

    on Oct 27, 2026
16. kyle-sexton

    self-assigned this

    on Oct 31, 2026
17. kyle-sexton commented on Oct 31, 2026

    

    kyle-sexton

    on Oct 31, 2026

    Last edited by kyle-sexton

    MemberAuthor

    More actions

    No description provided.
18. kyle-sexton

    closed this as completedin #11672on Oct 31, 2026
19. kyle-sexton commented on Oct 31, 2026

    

    kyle-sexton

    on Oct 31, 2026

    MemberAuthor

    More actions

    Result: landed in #11672.

    • Shipped: 6 outcome eval cases under plugins/wizard/evals/ (regex and tool_used graders) and frozen trigger probes probes/generate.json and probes/unattended.json.
    • No skill text changed. A concise hub-and-spoke rewrite of both skills was measured (40 passes), and every part reverted on the validity rule in skill-criteria.md Measurement step 4: 7 of 8 trigger passes were judged INVALID for sandbox denials the probes themselves trigger, and each body had one INVALID outcome sub-pass.
    • After review, three graders were corrected (test(wizard): outcome eval cases and trigger probes; concise rewrite measured, not shipped #11672 body, "Graders changed after review").
    • The measured rewrite is kept off-repo pending fix(evals): let trigger-measurement probe runs pass run-validity #11618 and test(visualization): fix the frozen suite's graders and re-measure present and visualize with VALID runs #11628, which hold the re-measure.