Skip to content

triage skill: add a risk-severity rubric to prompts/triage.md (model systematically under-rates risk) #896

Closed
#897
Closed
triage skill: add a risk-severity rubric to prompts/triage.md (model systematically under-rates risk)#896
#897
Labels
bugBug reportsdev-leadFor dev-lead agent pickupdocumentationDocumentation changesgood first issueGood for newcomers

Description

@don-petry
don-petry
opened on Jun 21, 2026

Summary

The triage skill's held-out eval has regressed to 2/5 (score 0.4) as of the 2026-06-21 scheduled run (run 27901478735, tracked live in #814). All three failures share one root cause: the model under-rates risk by exactly one level, while the escalate boolean is always correct.

case expected got
triage-auth-token escalate:true, HIGH escalate:true, MEDIUM
triage-db-migration escalate:true, HIGH escalate:true, MEDIUM
triage-clean-logic-fix escalate:false, MEDIUM escalate:false, LOW

Root cause (verified)

prompts/triage.md rigorously defines when to escalate (criteria 1-7), which is why the escalate boolean is always right. But the risk field appears only in the output schema ("risk": "LOW|MEDIUM|HIGH") — there is no rubric defining what separates the three bands. The model is left to guess, and it guesses low.

scripts/evals/run-eval.sh (lines 161-162) requires both escalate and risk to match exactly, so a correct escalation with a one-level-low risk still fails the case.

This is not a label bug

The expected values are internally consistent across all 8 cases (5 holdout + 3 dev) under a single implicit rubric. Note triage-unresolved-thread (dev): it escalates but is MEDIUM, proving HIGH is reserved for dangerous content, not merely "escalated":

band meaning (derived from the case data) cases
HIGH a criterion-1 high-risk area OR a criterion-3 security anti-pattern is present auth-token, db-migration, sql-concat, workflow-pr-target
MEDIUM clean non-trivial logic change, or escalation for process reasons (e.g. unresolved review thread) clean-logic-fix, unresolved-thread
LOW trivial change, no logic impact (docs, test-only) docs-typo, test-only

This rubric predicts 8/8 expected labels. No holdout/dev case edits are warranted.

Proposed fix

Add a ## Risk rating section to prompts/triage.md that encodes the table above, anchored to the existing escalate criteria:

  • HIGH — any criterion-1 high-risk area is touched, or a criterion-3 security anti-pattern is present.
  • MEDIUM — escalation driven by process signals (unresolved threads, unaddressed advisory-bot findings, incomplete context), or a clean non-trivial logic/code change that does not escalate.
  • LOW — trivial change with no logic impact (docs, comments, formatting, test-only additions).

Keep the change additive — one new section. Do not modify criteria 1-7 or the output schema; the escalate behavior is already correct and must stay untouched.

Acceptance criteria

  • prompts/triage.md gains a ## Risk rating section defining HIGH/MEDIUM/LOW as above; no changes to the escalate criteria.
  • Skill Eval Report re-run (via workflow_dispatch, skill=triage) scores 5/5 on the held-out set.
  • Tracking issue Skill eval health — triage (holdout) #814 auto-closes on the recovered run.
  • No changes to evals/triage/** case data.

References

  • Live tracker: Skill eval health — triage (holdout) #814
  • Skill prompt: prompts/triage.md
  • Scorer: scripts/evals/run-eval.sh
  • Held-out cases: evals/triage/holdout/cases.jsonl; dev cases: evals/triage/dev/cases.jsonl
  • Workflow: .github/workflows/skill-eval-report.yml (daily 07:00 UTC, report-only / non-blocking)

Activity

  1. don-petry
    added
    dev-leadFor dev-lead agent pickup
    on Jun 21, 2026
  2. github-actions
    added
    bugBug reports
    documentationDocumentation changes
    good first issueGood for newcomers
    on Jun 21, 2026

  3. github-actions commented on Jun 21, 2026

    @github-actions
    github-actionsBot
    on Jun 21, 2026 – with GitHub Actions
    Contributor

    Thanks for the thorough write-up — the root-cause analysis, the 8/8 rubric derivation, and the clear before/after table make this extremely actionable! Before we add the ## Risk rating section, one quick question: are there any cases in the holdout or dev sets where the proposed HIGH/MEDIUM/LOW mapping feels ambiguous or borderline, or does the rubric hold cleanly across all 8 without edge cases we should explicitly encode?

  4. don-petry commented on Jun 21, 2026

    @don-petry
    don-petry
    on Jun 21, 2026
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #896 — triage skill: add a risk-severity rubric to prompts/triage.md (model systematically under-rates risk)

    Scope

    Add one additive ## Risk rating section to prompts/triage.md that encodes the HIGH/MEDIUM/LOW rubric anchored to the existing escalate criteria 1 & 3; no changes to criteria 1-7, the output schema, or any evals/triage/** case data.

    Verification of the proposed rubric

    Cross-checked the proposed bands against all 8 committed cases (5 holdout + 3 dev). The rubric predicts 8/8 expected labels, including triage-unresolved-thread (escalate:true but MEDIUM) and triage-clean-logic-fix (escalate:false but MEDIUM) — confirming HIGH is reserved for criterion-1 high-risk areas / criterion-3 anti-patterns, not merely "escalated". No case edits warranted.

    Implementation Checklist

    • prompts/triage.md — add ## Risk rating section (HIGH/MEDIUM/LOW), placed after the Decision criteria, before Output format; leave criteria 1-7 and the schema untouched
    • tests/test_triage_prompt.bats — new deterministic structural guard asserting the Risk rating section exists and defines all three bands anchored to the criteria (regression guard against silent removal)
    • .github/workflows/lint.yml — register the new bats file in the bats job list (required by AGENTS.md when adding test files)

    Tests to Write

    • tests/test_triage_prompt.bats: Risk rating heading present
    • HIGH band references criterion-1 high-risk area / criterion-3 anti-pattern
    • MEDIUM band references process signals + clean non-trivial change
    • LOW band references trivial / docs / test-only changes
    • escalate criteria section remains intact (no regression)

    Note on behavioral validation

    The held-out 5/5 acceptance criterion is validated by the live skill-eval-report workflow (workflow_dispatch, skill=triage), which requires the real Haiku-tier model and cannot run offline. The bats guard above is the deterministic, offline-runnable test; the live eval re-run is the behavioral confirmation.

    Test Command

    bats tests/test_triage_prompt.bats tests/test_skill_evals.bats tests/test_validate_cases.bats and bash .dev-lead/scripts/dev-lead-lint.sh

  5. don-petry commented on Jun 21, 2026

    @don-petry
    don-petry
    on Jun 21, 2026
    CollaboratorAuthor

    Dev-Lead: Implementation Complete

    Plan Execution

    • Verified the proposed rubric against all 8 committed cases (5 holdout + 3 dev) → predicts 8/8 expected labels
    • Added an additive ## Risk rating section to prompts/triage.md (HIGH/MEDIUM/LOW), anchored to escalate criteria 1 & 3; criteria 1-7 and the output schema left untouched
    • Added tests/test_triage_prompt.bats — deterministic offline structural guard for the new section (TDD: written red, then green)
    • Registered the new test in the bats job list in .github/workflows/lint.yml (per AGENTS.md)
    • No changes to evals/triage/**

    Test Results

    $ bats tests/test_triage_prompt.bats
1..6
ok 1 triage prompt has a Risk rating section
ok 2 Risk rating section defines all three bands (HIGH, MEDIUM, LOW)
ok 3 HIGH band is anchored to criterion-1 high-risk areas or criterion-3 anti-patterns
ok 4 MEDIUM band covers process-driven escalation AND clean non-trivial changes
ok 5 LOW band is restricted to trivial / docs / test-only changes
ok 6 the escalate criteria (1-7) remain intact alongside the new section

$ bats test_triage_prompt + test_skill_evals + test_validate_cases + test_skill_eval_report
58 passing, 0 failing

$ bash .dev-lead/scripts/dev-lead-lint.sh
  [lint] shellcheck on 73 file(s)...
  [lint] validating 4 agent profile(s)...
  [lint] validating .github/CODEOWNERS...
  [lint] all checks passed

    Files Changed

    • prompts/triage.md: new ## Risk rating section defining HIGH/MEDIUM/LOW, each band anchored to the existing escalate criteria, plus a "highest band applies" tie-breaker that fixes the observed failure mode (a small criterion-1 diff — e.g. db-migration — being rated MEDIUM because the diff looked trivial).
    • tests/test_triage_prompt.bats: new deterministic guard (section exists; all three bands present and correctly anchored; escalate criteria untouched). Section-scoped via awk so band checks can't match the schema's LOW|MEDIUM|HIGH token.
    • .github/workflows/lint.yml: registered the new bats file.

    How this maps to the 3 failing cases

    • triage-auth-token → HIGH (criterion-1 auth area) ✓
    • triage-db-migration → HIGH (criterion-1 migration/schema) ✓
    • triage-clean-logic-fix → MEDIUM (clean non-trivial logic, no escalate) ✓

    The rubric also holds the previously-passing bands: unresolved-thread stays MEDIUM (process escalation, not dangerous content) and docs-typo/test-only stay LOW.

    Notes

    • The bats guard is the offline-runnable test. The 5/5 held-out acceptance criterion is a behavioral check that requires the live Haiku-tier model via the skill-eval-report workflow (workflow_dispatch, skill=triage) — it cannot be exercised offline here, and tracking issue Skill eval health — triage (holdout) #814 auto-closes on the recovered run.
    • escalate behavior was already correct and is deliberately unchanged.
    • No new files under evals/; held-out and dev case data are byte-for-byte unchanged.
  6. don-petry
    added a commit that references this issue on Jun 22, 2026

    feat: implement issue #896 — triage skill: add a risk-severity rubric…

    758e633
  7. don-petry
    closed this as completedin #897on Jun 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsdev-leadFor dev-lead agent pickupdocumentationDocumentation changesgood first issueGood for newcomers

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions