Initiative: Eval-gated, human-reviewed self-improving skills (SkillOpt-style)

Stand up an eval-gated, human-reviewed self-improvement loop for agent skills/prompts, modeled on Microsoft SkillOpt and Anthropic skill-creator (Discussion #572). The shippable scope is weight-free, fully version-controlled, and human-gated: an agent may propose a bounded edit to a skill markdown file, an offline eval gate decides on a held-out set, and the winner rides the existing idea->initiative->pr-review pipeline as a normal, CODEOWNER-reviewed PR. Nothing self-promotes; no release-tag ruleset changes.

Sequencing follows the idea's own deliverables. Phase 1 lands the keystone with no loop: a versioned held-out eval set for prompts/triage.md, a deterministic scorer, and a NON-BLOCKING CI report. Phase 2 generalizes the harness to prompts/deep-review.md (LLM-judge) and proves a strict-improvement gate rejects regressions (Deliverable 2). Phase 3 automates the scheduled proposer, but only behind the safe-rollout prerequisites (health-gated promotion #501 and shadow-mode dual-run) and is hands_off.

Reward-hacking guard by construction: eval cases and judge prompts are owned separately (CODEOWNERS) and never writable by the proposer; the gate promotes only strict winners (no ties). See docs/initiatives/idea-to-initiative-pipeline.md for the human gates this reuses unchanged.

Planned from idea discussion #572 by the BMAD Scrum Master initiative-planner. Inert until a maintainer adds initiative:auto.

🦆 Rubber-duck review of this plan

Strong baseline — every story is grounded in real repo files, the phasing is right, and #587 is correctly hands_off. Findings below are where the plan should be tightened before a maintainer adds initiative:auto. Ordered by load-bearing-ness.

1. The reward-hacking guard blocks editing the cases, not overfitting to them ⚠️ (deepest)

Stories #582/#585/#587 protect the held-out set with CODEOWNERS so the proposer "can never edit" it — but the proposer reads the whole repo, so it can see every case and tune the skill to pass exactly those with no real generalization ("teaching to the test"). SkillOpt's discipline relies on a held-out set the optimizer never sees; nothing here hides or rotates the cases.
Fix: add a held-out-hygiene story — a hidden/rotating holdout or a dev/test split where the proposer only sees the dev subset and the gate scores against the unseen one.

2. CODEOWNERS is not a hard block — the real enforcement is under-specified

CODEOWNERS only requests review and only bites if branch protection's "require review from code owners" is enabled (an unstated dependency). #587 AC3's "plus a check enforces this" is the actual guarantee and deserves its own test-covered AC: a CI gate that fails any proposer-authored PR touching skills/evals/ (keyed on PR author + changed paths).

3. Strict `>` gate + noisy LLM-judge = promotion on noise

#586 encodes candidate_score > incumbent_score. Fine for deterministic triage; but #585's deep-review uses an LLM-judge whose score varies run-to-run, so a single-sample strict > can promote a lucky draw. Fix: for judge-scored skills, require n runs and improvement beyond a noise-band margin, not a one-shot >.

4. A contested decision is baked into Story #582's ACs

The plan's own open-questions flag that the repo has no skills/ dir (skills live in prompts//frameworks/) — yet #582's ACs hard-code skills/evals/<skill>/cases.jsonl as fact, so a dev builds the contested tree and finds the conflict later. Resolve the path question first, or mark those ACs blocked-pending-decision.

5. No success metric for the initiative

The epic describes the mechanism but never defines a win condition — no target like "% of proposer PRs merged," "measured quality lift on the held-out set," or "zero reward-hack escapes." Add measurable exit/expansion criteria.

6. No cost ceiling

#584 (cron eval) + #585 (per-case LLM-judge) scale as cases × skills × judge-calls × cadence, and the idea cited the daily-cap concern (#206). Add a token/case/cadence cap as an explicit AC.

Recommended before initiative:auto: resolve the skills/evals/ path (finding 4), and add two stories — held-out-set hygiene (1) and a hard case-immutability CI check (2) — since those are the trust guarantees the whole "self-improving" claim rests on. Planner/automation-level feedback (dry-run≠real, no dup-epic idempotency, open-questions-as-gate) is being tracked separately as workflow enhancements.

Posted by Claude Code after planning this initiative; advisory, not a gate.

✅ Maintainer decisions on the open questions

A1 — Eval directory: use evals/<skill>/cases.jsonl at repo root (NOT a new skills/ tree — skills live in prompts//frameworks/). Story #582's ACs and target-surface paths should be updated from skills/evals/... to evals/... accordingly.

A2 — Shadow-mode prerequisite: promoted Discussion #566 to a tracked issue #605 and wired it as a real blocked_by edge on #587 (which now correctly shows blocked_by #586, #501, #605). The Phase 3 prerequisite is no longer prose-only.

A3 — Held-out set sizing/sourcing: approved — seed cases from real historical triage/deep-review decisions, de-identified (strip repo/author/secret material; keep the decision-relevant diff shape). Start at ~20–30 cases per skill and grow. Stories #582/#585 should note the de-identification requirement as an AC.

A4 — Reuse Safe Release Strategy machinery: confirmed — Phase 3 consumes the ring/health-gating machinery (#495/#501), it does not re-implement it. It's the standard.

These resolve the planning open-questions; the epic remains inert until initiative:auto is added.

refactor(evals): relocate held-out eval set to repo-root evals/ per m…

⚠️ Two native `blocked_by` edges still need manual wiring

The Phase 3 proposer (#587) should not activate until the two reward-hacking guardrails exist. Those guardrails were recommended in this epic's plan review but added after planning (#691 held-out hygiene, #692 case-immutability CI check), so they are not in the DAG yet. The initiative-driver only honors native GitHub issue-dependencies — a blocked_by written in issue prose is ignored — so these edges have to be added by hand:

How: on #587, use Add dependency → Blocked by in the GitHub UI (or the planner's add_blocked_by mutation / gh api repos/$REPO/issues/587/dependencies/blocked_by). I couldn't wire these from automation — this session has no path to the issue-dependency API (no gh, no token, no MCP tool for it).

Meanwhile sequencing is safe: #587 carries initiative:hold + dev-lead:hands-off, so the driver won't release it regardless; #691/#692 have no blockers and remain releasable so they can be built first. The prerequisites are also now recorded in #587's AC4 / Dev Notes.

The planner gap that caused this (review findings are gated but never materialized into the DAG) is tracked at #706 under epic #597.

Generated by Claude Code

feat: implement issue #582 — [Phase 1] Define held-out eval-case form…

Navigation Menu

Repository navigation

Initiative: Eval-gated, human-reviewed self-improving skills (SkillOpt-style) #581

🦆 Rubber-duck review of this plan

1. The reward-hacking guard blocks editing the cases, not overfitting to them ⚠️ (deepest)

2. CODEOWNERS is not a hard block — the real enforcement is under-specified

3. Strict `>` gate + noisy LLM-judge = promotion on noise

4. A contested decision is baked into Story #582's ACs

5. No success metric for the initiative

6. No cost ceiling

✅ Maintainer decisions on the open questions

⚠️ Two native `blocked_by` edges still need manual wiring

20 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Initiative: Eval-gated, human-reviewed self-improving skills (SkillOpt-style) #581

Description

Activity

don-petry commented on Jun 11, 2026

🦆 Rubber-duck review of this plan

1. The reward-hacking guard blocks editing the cases, not overfitting to them ⚠️ (deepest)

2. CODEOWNERS is not a hard block — the real enforcement is under-specified

3. Strict > gate + noisy LLM-judge = promotion on noise

4. A contested decision is baked into Story #582's ACs

5. No success metric for the initiative

6. No cost ceiling

don-petry commented on Jun 11, 2026

✅ Maintainer decisions on the open questions

don-petry commented on Jun 14, 2026

⚠️ Two native blocked_by edges still need manual wiring

20 remaining items

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

3. Strict `>` gate + noisy LLM-judge = promotion on noise

⚠️ Two native `blocked_by` edges still need manual wiring