An empty diagnostics row set summarizes as scoring_ready #354

Summary

summarizePublicEvalDiagnosticsRows([]) returns scoring_ready: true. A diagnostics document describing zero rows — a benchmark that planned nothing, or ran nothing — reports itself as ready to score.

$ node -e "const {summarizePublicEvalDiagnosticsRows}=require('./packages/evals/dist/index.js');
           console.log(JSON.stringify(summarizePublicEvalDiagnosticsRows([])))"
{"planned":0,"launched":0,"launch_failed":0,"run_records_missing":0,"workflow_succeeded":0,
 "workflow_failed":0,"workflow_nonterminal":0,"genuine_task_failure_rows":0,
 "terminal_reports_present":0,"scoring_ready":true}

Cause

packages/evals/src/public-diagnostics.ts:

scoring_ready: rows.every((row) => row.scoring_ready) && publicEvalDiagnosticsFailedTargetCount(rows) <= 1

[].every(…) is true and publicEvalDiagnosticsFailedTargetCount([]) is 0, so both conjuncts hold vacuously. The same shape appears in summarizeLegacyPublicEvalDiagnosticsRows.

parsePublicEvalDiagnostics recomputes the summary and compares it to the stored one, so a zero-row document is not merely accepted — it validates as internally consistent and scoring-ready.

Reachability — latent, not live

The only caller is packages/modal/src/public-eval-diagnostics.ts:253, and it is currently protected by z.array(matrixRowInputSchema).min(1) on the matrix in the same function. So no zero-row document can be produced today through that path.

The reason to fix it anyway is that the guard and the function are in different packages: the invariant "there is at least one row" is enforced in @ultrafuzz/modal while the assumption that depends on it lives in @ultrafuzz/evals, exported for anyone to call. I hit this while working on #165 — when I relaxed the record-level .min(1) in #353 I had to keep the matrix-level one specifically to avoid this, which is a guard held in place by a constraint two layers away rather than by anything local.

Suggested fix

scoring_ready: rows.length > 0 && rows.every((row) => row.scoring_ready) && publicEvalDiagnosticsFailedTargetCount(rows) <= 1

in both summarizePublicEvalDiagnosticsRows and summarizeLegacyPublicEvalDiagnosticsRows, with a test asserting an empty row set is not scoring-ready. That would also let the matrix-level .min(1) be reconsidered on its own merits rather than as a load-bearing workaround.

Related: #353 relaxes the record-level bound and explains why it deliberately left the matrix-level one alone.