CI: eval-history publisher treats soft-failed Modal smoke diagnostics as publishable results #560

Failure

After #558 merged, the core CI workflow passed on main, but downstream run 31696750390 failed in publish_modal_benchmark.

The trusted publisher tried to open:

.../public-benchmark-results/.../public-results.json

and failed with ENOENT.

Root cause

The automatic Modal smoke producer intentionally permits recoverable operational outcomes to soft-fail without making the producer workflow red. In producer run 31696124161, the exact pair ended as resume-required; collection uploaded bounded diagnostics and no scoring/publication bundle, as designed.

Publication qualification currently checks only that launch and collect jobs succeeded and that artifacts with the expected names exist. The diagnostics-only results artifact satisfies that name check, so the publisher treats an incomplete smoke generation as publishable and later crashes on the correctly absent public-results.json.

Expected fix

Use trusted final control/outcome metadata to classify publication readiness before requesting a publisher token:

• a fully successful, collected pair must have its exact public-results.json and proceeds to strict atomic validation/publication;
• an explicitly incomplete automatic smoke generation that the producer soft-failed becomes a successful publication no-op;
• missing/malformed outcomes, inconsistent success claims, ambiguous artifacts, or successful outcomes without bundles still fail closed.

Acceptance criteria

• The downstream publication check is green for diagnostics-only soft-failed automatic smoke runs.
• Complete runs still publish only after strict validation.
• Integrity failures cannot be reclassified as a no-op.
• CI is green on the resulting main commit.