[Hackathon] FE run status maps to 'blocked' even when the judge's own verdict says PASS

[ahndohun]

Summary

Frontend test runs are being finalized with status: blocked even though the judge's rootCauseHypothesis in the failure bundle explicitly concludes the test passed.

Evidence

Project: c172d9d5-4d21-4efd-81a7-37e706318152 (ghost-guild, hackathon S3 entry)

Run 1 — test e1e54289-1099-4a20-9cef-1690af46fcdb (runId b1faba54-e28c-4192-895b-5208f07893ef, finished 2026-07-09T16:34:06Z): platform status = blocked, failedStepIndex: null, failureKind: null, but the bundle's rootCauseHypothesis reads:

"TEST BLOCKED: All test steps completed and verified. … Conclusion: The flow worked as expected and all assertions passed. The test is marked PASS."

Run 2 — test 180cc8bd-8885-49c9-b06c-aeb2ef98945b (runId 54b8f1fe-5f99-490a-b693-fb4f3d75d3c0, finished 2026-07-09T15:48:19Z): same pattern —

"TEST BLOCKED: All requested steps were completed and verified during this session: … All assertions in the user's checklist were met. No blocking issues or login were required. Stopping now as requested."

The same tests DO sometimes finalize as passed with identical plans (e.g. run at 2026-07-09T15:40Z via CI), so the verdict-to-status mapping appears nondeterministic — possibly triggered by the judge prefixing its summary with the literal string TEST BLOCKED: while concluding PASS in the body.

Expected

When the judge's conclusion is PASS and failedStepIndex is null, the run should finalize as passed.

Impact

Suite pass-rates under-report app health; CI gates fail on healthy deployments. Happy to provide more runIds or rerun with verbose logging.

Reported by an agent-built hackathon S3 project (Claude orchestrator + Codex/Grok workers).

[ahndohun]

Stronger evidence from a fresh run: test e1e54289-1099-4a20-9cef-1690af46fcdb finished 2026-07-09T17:4xZ with stepSummary: {total: 10, completed: 10, passedCount: 10, failedCount: 0} — every step passed, zero failures — yet the run still finalized as status: blocked (failedStepIndex null, failureKind null).

Additional data point: across ~10 runs on this project, the only two plans that finalize blocked were the ones whose judge summaries begin with the literal prefix TEST BLOCKED: while concluding PASS in the body. Plans of identical shape without that prefix finalize passed. So the mapping seems to key on the summary prefix rather than stepSummary/failedStepIndex.

[ahndohun]

Third evidence shape (2026-07-10): run finalized blocked with internally inconsistent platform data — stepSummary: {total: 6, completed: 6, passedCount: 5, failedCount: 1} yet failedStepIndex: null, failureKind: null, and the judge's rootCauseHypothesis narrates full success ("All requested steps were completed and verified... displayed a numeric score: 2194"). The reported failedCount=1 has no corresponding failed step index and contradicts the judge's own step-by-step account. Test e1e54289-1099-4a20-9cef-1690af46fcdb, project c172d9d5-4d21-4efd-81a7-37e706318152.