[Phase 1] Make the Skill Eval Report visible — surface regression/error outcomes (no longer silent) #747

Activity

1. don-petry

   added

   dev-leadFor dev-lead agent pickupFor dev-lead agent pickup

   initiativeEpic / initiative tracking issueEpic / initiative tracking issue

   on Jun 15, 2026
2. don-petry

   added a parent issue on Jun 15, 2026

   • Initiative: Eval-gated, human-reviewed self-improving skills (SkillOpt-style) #581
3. github-actions

   added

   enhancementFeature requestsFeature requests

   on Jun 15, 2026
4. github-actions commented on Jun 15, 2026

   

   github-actionsBot

   on Jun 15, 2026 – with GitHub Actions

   Contributor

   More actions

   Thanks for the thorough write-up and for connecting this back to the epic — the visibility gap is well-articulated! Before we start on implementation: for the tracking-issue approach, should the issue be opened in this same repo (petry-projects/.github-private) or in a separate ops/observability repo, and is there an existing eval-health label to reuse or should the workflow create it on first use?
5. don-petry commented on Jun 15, 2026

   

   don-petry

   on Jun 15, 2026

   CollaboratorAuthor

   More actions

   Dev-Lead Implementation Plan

   Issue: #747 — [Phase 1] Make the Skill Eval Report visible — surface regression/error outcomes (no longer silent)

   Scope

   Add a non-blocking visibility layer to skill-eval-report.yml: on outcome=regression|error, open-or-update a single de-duplicated eval-health tracking issue; on recovery (pass with a prior open issue), comment + close it. Healthy runs stay silent. Logic lives in a stub-testable script with offline bats coverage.

   Implementation Checklist

   • scripts/evals/notify-eval-health.sh — new: find-or-create/update/close the single eval-health tracking issue (idempotent, de-duped by stable title+label), rendering the score table + regressed-case detail + run link. Modeled on scripts/idea-triage/upsert-queue.sh. Honors DRY_RUN; uses gh from PATH (stub-injectable).
   • .github/workflows/skill-eval-report.yml — add issues: write to the job permissions (keep contents: read); add an if: always() "Notify eval health" step that invokes the script with OUTCOME / SKILL / REPORT_PATH / RUN_URL. continue-on-error on the eval step is preserved (no gating added).
   • tests/test_eval_health_notify.bats — new offline test (stub gh on PATH, no network) covering open / update / close / de-dupe / no-noise-on-pass.
   • .github/workflows/lint.yml — add the new bats file to the repo-specific test list.
   • tests/test_skill_eval_report.bats — extend with workflow-YAML assertions for the new issues: write scope and the notify step.

   Tests to Write

   • regression with no existing issue → creates exactly one issue (label eval-health)
   • regression with an existing open issue → edits in place, creates none (de-dupe)
   • error outcome → opens/updates the tracking issue (not silent)
   • pass with a prior open issue → comments + closes it
   • pass with no existing issue → no create, no comment (zero noise)
   • body contains score table + regressed-case detail + run link
   • workflow YAML declares issues: write and wires the notify step

   Test Command

   bats tests/test_eval_health_notify.bats tests/test_skill_eval_report.bats tests/test_skill_evals.bats (full suite: the bats list in .github/workflows/lint.yml)
6. don-petry commented on Jun 15, 2026

   

   don-petry

   on Jun 15, 2026 – with Claude

   CollaboratorAuthor

   More actions

   Confirming the defaults so you're unblocked (matches your 00:29 plan):

   • Same repo — open the tracking issue here in petry-projects/.github-private, next to the workflow and CODEOWNERS. No separate ops/observability repo.
   • Label — there is no existing eval-health label; have the workflow create-it-on-first-use (idempotent find-or-create) so there's no manual setup step. Single issue, de-duped by stable title + the eval-health label, updated in place.

   Proceed on that basis. One real-world note to fold in: the first baseline dispatch of skill-eval-report.yml just came back outcome=regression (run 27516878017) — so this notify path will light up immediately on the first scheduled run. Please make sure the issue body renders the per-case expected-vs-actual detail for regressed cases (your checklist covers it) — that's the first thing we'll need to read.

   ---

   Generated by Claude Code
7. don-petry commented on Jun 15, 2026

   

   don-petry

   on Jun 15, 2026

   CollaboratorAuthor

   More actions

   Dev-Lead: Implementation Complete

   Plan Execution

   • New scripts/evals/notify-eval-health.sh — idempotent open/update/close of a single de-duplicated eval-health tracking issue (modeled on idea-triage/upsert-queue.sh)
   • .github/workflows/skill-eval-report.yml — added issues: write (kept contents: read) and an if: always() "Notify eval health" step; continue-on-error on the eval step preserved (no gating)
   • tests/test_eval_health_notify.bats — offline stub-gh tests for open / update / close / de-dupe / no-noise + guards
   • tests/test_skill_eval_report.bats — extended with workflow-YAML assertions for the new scope + notify step
   • .github/workflows/lint.yml — added the new bats file to the repo-specific test list

   Behavior (visibility only, non-blocking)

   • outcome=regression|error → opens or updates in place a single eval-health tracking issue (score table + regressed-case detail + run link). Never spams a new issue per day.
   • outcome=pass with a prior open tracking issue → comments Recovered and closes it.
   • outcome=pass with no open issue → zero noise (no create/comment/close).
   • Least-privilege: only issues: write added; contents: read retained for checkout.

   Test Results

   # bats (full lint.yml list, 23 files): 376 passing, 0 failing
   ok 366 regression with no existing issue creates exactly one eval-health issue
   ok 367 regression with an existing open issue updates in place (no new issue — de-dupe)
   ok 368 the tracking issue is looked up de-duped by label and open state
   ok 369 regression body carries the score table, regressed case detail and run link
   ok 370 error outcome opens the tracking issue even with an unparseable report
   ok 371 error outcome with an existing issue updates it (de-dupe, not silent)
   ok 372 pass with a prior open issue comments and closes it (recovery)
   ok 373 pass with no existing issue makes zero noise (no create/comment/close)
   ok 374 missing REPO is a hard error
   ok 375 an unknown OUTCOME is a hard error
   ok 376 DRY_RUN never invokes gh mutations
   
   # .dev-lead/scripts/dev-lead-lint.sh
     [lint] shellcheck on 55 file(s)...
     [lint] validating 4 agent profile(s)...
     [lint] all checks passed
   

   Files Changed

   • scripts/evals/notify-eval-health.sh (new): notifier logic — find-or-create/update/close.
   • .github/workflows/skill-eval-report.yml: issues: write + notify step (always()).
   • tests/test_eval_health_notify.bats (new): offline notifier tests.
   • tests/test_skill_eval_report.bats: +4 YAML assertions for [Phase 1] Make the Skill Eval Report visible — surface regression/error outcomes (no longer silent) #747.
   • .github/workflows/lint.yml: registered new bats file.

   Notes

   • Chose the tracking-issue approach (primary AC path); the daily-pr-review-health digest fold-in was explicitly "alternative/complementary" and is left out to keep scope minimal.
   • De-dup is scoped to open issues by label + stable per-skill title. A human-closed issue is intentionally not reopened; a fresh regression then opens a new one.
   • Acknowledged out-of-scope items untouched: no gating/blocking, no holdout case-set growth, no nondeterminism work.
8. don-petry

   added a commit that references this issue on Jun 15, 2026

   feat: implement issue #747 — [Phase 1] Make the Skill Eval Report vis…

   ...

   5fd164e
9. don-petry

   closed this as completedin #754on Jun 15, 2026