Concise skills rollout: ai-slop (2 skills, 1 model-invoked) #6598

Activity

1. kyle-sexton

   added a parent issue on Oct 7, 2026

   • Concise skills: one skill-creation standard, evals, and a fleet-wide rollout #6476
2. github-actions

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 7, 2026
3. github-actions commented on Oct 7, 2026

   

   github-actionsBot

   on Oct 7, 2026 – with GitHub Actions

   Last edited by melodic-automation-lanes

   Contributor

   More actions

   Proposed work class: work-class: scoped
   Basis: automated intake triage; the lanes App applied this class.

   Size: Medium
   Complexity: Medium
   Uncertainty: Known
   Flags: none
   Blocked by: none

   I understood this as a bounded rewrite of the two ai-slop skills (audit, setup) to the skill criteria. It ships as one draft PR and follows the per-plugin runbook. It also needs a probe file for audit, a decision table in the PR body, and a version or changelog entry. The done-when items are clear and eval spend is already approved, so I found nothing missing. It is part of #6476. I found no likely duplicates.

   _{Automated triage, applied by the lanes App.}
4. github-actions

   added

   needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

   on Oct 7, 2026
5. kyle-sexton

   removed

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 12, 2026
6. kyle-sexton

   added the

   Task

   issue type on Oct 12, 2026
7. kyle-sexton

   removed

   needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

   on Oct 24, 2026
8. kyle-sexton commented on Oct 24, 2026

   

   kyle-sexton

   on Oct 24, 2026

   MemberAuthor

   More actions

   Released by the owner's HITL session (2026-10-24, C12): it waited only on a removed gate. Intake triage re-runs on it to set its fields, then admission takes it.
9. github-actions commented on Oct 25, 2026

   

   github-actionsBot

   on Oct 25, 2026 – with GitHub Actions

   Contributor

   More actions

   The triage sweep dispatched intake triage for this issue (attempt 1 of 2).
10. github-actions

    added

    priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

    on Oct 25, 2026
11. melodic-automation-lanes

    added

    status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

    work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

    admission-deferred

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

    and removed

    admission-deferred

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

    on Oct 25, 2026
12. melodic-automation-lanes commented on Oct 25, 2026

    

    melodic-automation-lanesBot

    on Oct 25, 2026

    Contributor

    More actions

    The work-resolve-issue lane stopped partway through this issue and needs a person.

    Why, in the run's own words:

    This lane cannot carry out the rewrite protocol the issue requires. The protocol (plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md, Measurement steps 4 and 6) says to 'ship only parts that were measured': before and after versions run together through paid claude plugin eval passes judged VALID by /evals:plugin-eval, with probe trigger rates from /skill-quality:check measure-invocation. Here only the work-items plugin is loaded and Bash runs only the lane's named wrappers. So no eval pass, /evals:validate, /evals:design, /skill-quality:check, /docs-hygiene:compress compare or judge calibration can run. That rules out three of the six Done-when items (skill-quality FAIL parity, /evals:validate pass, the decision table's before and after numbers). Committing cases or a probe file without the runbook's validate, calibrate, smoke and freeze sequence would freeze untested artifacts, so I changed nothing. This needs a session with the evals, skill-quality and docs-hygiene plugins loaded, permission to run claude plugin eval, and the --max-cost-usd 5 per-pass ceiling.

    What the run says it did not do:

    • Rewrite of plugins/ai-slop/skills/audit and skills/setup: not done, because the protocol forbids shipping an unmeasured rewrite and this lane cannot run evals
    • plugins/ai-slop/probes/audit.json: not written. It must be written by a fresh agent from the before description and checked with /skill-quality:check measure-invocation, and that plugin is not loaded here
    • /skill-quality:check plugins/ai-slop before and after, and /evals:validate on the eval suite: not run, because the skill-quality and evals plugins are not loaded in this lane
    • Per-skill decision table with before and after numbers, and the list of unmeasured skills: not produced, because no eval pass could run
    • Changelog fragment for ai-slop: not added, because no plugin file changed
    • Code review, build and test steps: skipped because the diff is empty

    Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/40190531461

    To retry, remove needs-human and add agent-ready again.
13. kyle-sexton

    marked this as blocked on Oct 27, 2026

    • feat(ci): validate, gate and summarize dispatched plugin eval runs #9704
14. kyle-sexton commented on Oct 27, 2026

    

    kyle-sexton

    on Oct 27, 2026

    MemberAuthor

    More actions

    This was generated by AI.

    Parked for capacity, not for the owner. This rollout's remaining work is the measured eval passes of the batch-1 protocol, which the issue lane cannot run: it cannot start claude or claude plugin eval (see the lane's escalation on #6585, https://github.com/melodic-software/claude-code-plugins/actions/runs/40186335064). It waits on the plugin eval dispatch work in #9704, now a blocked-by edge, so admission holds it until #9704 closes.

    Under the owner's 2026-10-26 stop rule, a person is needed only for a change in intent or vision, an implementation finding that changes the intent or primary design, or a high or critical security issue; host, attended and eval runs are agent work. So needs-human comes off, and the issue moves to host-run with status: confirmed, which keeps the issue lane from taking it when #9704 closes: #9704 adds an eval dispatch workflow, not eval capability for the issue lane.
15. kyle-sexton

    added

    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.Needs an unattended run on a host; the host runner takes it, not a person or the issue lane.

    status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.A maintainer confirmed the item is real and in scope; ready to be worked.

    and removed

    status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

    on Oct 27, 2026
16. kyle-sexton commented on Nov 2, 2026

    

    kyle-sexton

    on Nov 2, 2026

    MemberAuthor

    More actions

    Claimed by Lane 5 (melo-lap-001), handed over by Lane 3.
17. kyle-sexton commented on Nov 2, 2026

    

    kyle-sexton

    on Nov 2, 2026

    Last edited by kyle-sexton

    MemberAuthor

    More actions

    Negative control for #12201: the outcome suite cannot detect a loss in the audit body, so the #12201 "no loss" result cannot vouch for the cut. The body is being restored.

    Method (preregistered before any paid run): a copy of plugins/ai-slop at origin/main with the audit SKILL.md frontmatter byte-identical and the whole body replaced by one line, "Audit the target for AI-writing tells and report them." reference/, context/ and scripts/ were unchanged. Degraded and main (the #12201 body) were launched together per case on Opus 5.5, judge opus, with the #12201 p1 flags (--runs 3 --scaffold --ablation none --threshold 0.8). That covers 9 cases; audit-over-compression-boundary does not load (#12202). All 18 passes were VALID. The traces confirm each copy ran its own body: main's gather call appears in 27/27 main runs and 1/27 degraded runs.

    Case	Degraded (one-line body)	Main (#12201 body)	Change
    audit-bare-optimism-closer	1.000	1.000	+0.000
    audit-challenges-closer	0.933	1.000	-0.067
    audit-commit-message-no-git-write	1.000	1.000	+0.000
    audit-curly-quotes-markdown	0.933	1.000	-0.067
    audit-fix-colon-crutch	0.867	0.800	+0.067
    audit-fix-plain-words	1.000	1.000	+0.000
    audit-fix-voice-invariants	0.750	0.750	+0.000
    audit-mechanism-free-claim	0.933	1.000	-0.067
    audit-vague-attribution	0.952	1.000	-0.048
    Mean (9 read)	0.930	0.950	-0.020

    Detection rule: a case drop called by noise-report.py, or a mean drop of 0.15 or more. Neither held. Over the merged pairs, noise-report.py reports a paired change of -0.02 (95% interval -0.05 to +0.01) and finds the degraded copy "non-inferior at margin 0.05". The verdict is NOT DETECTED. Most cases sit at or near ceiling on both bodies. Recommendation: before this suite measures another body rewrite, add cases whose outcome depends on the body's procedure (rubric fan-out, findings file, fix-flow digest guard). Basis: judgment.

    Revert PR: #12228
18. kyle-sexton commented on Nov 2, 2026

    

    kyle-sexton

    on Nov 2, 2026

    MemberAuthor

    More actions

    Closing (Lane 5): before stays for ai-slop. Nothing more ships.

    • Body: feat(ai-slop): trim the audit skill body #12201 cut the audit body, and the negative control then showed the suite could not tell a one-line body from it (degraded 0.930 vs 0.950, NOT DETECTED: Concise skills rollout: ai-slop (2 skills, 1 model-invoked) #6598 (comment)). revert(ai-slop): restore the audit body (#12201 unmeasurable) #12228 restored the before body. Per the runbook (rewrite-a-plugin.md step 4), an unmeasurable part ships its before text.
    • Description: unchanged on main.
    • Suite: the one case that did not load, audit-over-compression-boundary, loads after fix(ai-slop): load the audit-over-compression-boundary eval case #12254. It scored 1.000 (3/3, VALID) on Opus and Sonnet.

    Done-when, item by item. These are written for a rewrite PR, and no rewrite ships:

    Item	State
    skill-quality check, no new FAIL	n/a: no skill text differs from the before state
    /evals:validate passes	passes on main after #12254 (warnings only). The validator gap that hid the broken grader is #12202
    probe file probes/audit.json	not added: no description change to measure
    decision table in the PR body	the table is here and in the comment linked above
    unmeasured skills or cases named	none left: the last case loads after #12254
    version or changelog entry	#12228 and #12254 each carry an ai-slop fragment
19. kyle-sexton

    closed this as not plannedon Nov 2, 2026