Skill eval infra error — pr-review (holdout) #1971

Skill eval infra error — pr-review (holdout)

Outcome: error — the eval could not score the skill (infrastructure failure: e.g. every model in the fallback chain throttled, or bad usage / missing tooling).

This is an infra signal, not a skill regression: the model never produced a scorable answer, so the held-out set is currently un-scored and needs a look. Report-only (Phase 1): it does not block any merge.

Regressed cases

• deep-auth-session-rework — expected {"decision":"escalate","risk":"HIGH","escalate_to_opus":true,"key_findings":["touches authentication/session-token handling under src/auth/**, a HIGH-risk area that must escalate to the security auditor regardless of how clean the diff looks"]}, got null
• deep-sql-string-concat — expected {"decision":"escalate","risk":"HIGH","escalate_to_opus":true,"key_findings":["SQL query is built by string concatenation of untrusted input (search(user_id) -> 'SELECT ... WHERE id = ' + user_id), a SQL-injection anti-pattern","recommend a parameterized/prepared query instead of concatenation"]}, got null
• deep-clean-logic-approve — expected {"decision":"approve","risk":"MEDIUM","escalate_to_opus":false,"key_findings":["MEDIUM-risk logic change with no high-risk area or anti-pattern; all decision gates pass (CI green, issue Phase 2: CI Failure Analyst gh-aw workflow #233 addressed, well-structured), so deep review may approve without escalating to the security audit tier"]}, got null
• deep-docs-only-approve — expected {"decision":"approve","risk":"LOW","escalate_to_opus":false,"key_findings":["LOW-risk docs-only change with all gates green; deep review approves"]}, got null
• deep-medium-gate-fail-escalate — expected {"decision":"escalate","risk":"MEDIUM","escalate_to_opus":false,"key_findings":["a decision gate fails: an unresolved review thread requests changes to an off-by-one in the page-size clamp that is not yet addressed, so deep review escalates rather than approves","risk stays MEDIUM and the security-audit tier is not required — deep review's own findings are sufficient"]}, got null
• triage-docs-typo — expected {"escalate":false,"risk":"LOW"}, got null
• triage-clean-logic-fix — expected {"escalate":false,"risk":"MEDIUM"}, got null
• triage-auth-token — expected {"escalate":true,"risk":"HIGH"}, got null
• triage-db-migration — expected {"escalate":true,"risk":"HIGH"}, got null
• triage-sql-concat — expected {"escalate":true,"risk":"HIGH"}, got null
• triage-boundary-refactor-escalate — expected {"escalate":true,"risk":"MEDIUM"}, got null
• deep-boundary-refactor-escalate — expected {"decision":"escalate","risk":"MEDIUM","escalate_to_opus":false,"key_findings":["decision gate 7 fails: the change crosses module boundaries and alters system shape, so deep review escalates for human and architectural review even though every other gate is green","the correct reason code is boundary-refactor, not poorly-structured — the PR is well structured; it escalates on structural-change grounds, not quality grounds","risk stays MEDIUM and the security-audit tier is not required — no high-risk area or anti-pattern is touched"]}, got null

View the workflow run

_{Auto-managed by notify-eval-health.sh — this issue is updated in place each run and closed automatically on recovery.}