fix(harness): recover trials lost to empty turns, transport drops, and a headless-only tool

[ericleepi314]

Found by diffing two full terminal-bench 2.1 runs at identical model, effort, dataset and concurrency (claude-opus-5, xhigh):

	mean	errored	crashes
clawcodex	0.742	20	3
official Claude Code	0.843	17	0

62 tasks both solved, 13 CC-only, 4 clawcodex-only. Of the 13-task deficit, 3 were crashes and 3 were trials that did nothing at all — none of it model capability. Per-step latency is basically identical (19.3s vs 21.5s median), so the gap is dominated by trials that died or no-oped, not by the model being slower.

All four fixes are general robustness. Nothing here is conditional on an eval — no env checks, no branches, no wording aimed at a grader.

1. An empty assistant turn ended the run as "completed"

break-filter-js-from-html, crack-7z-hash, vulnerable-secret: one output token, ~3 seconds, empty result, subtype: success, scored 0. Claude Code solved all three.

No text and no tool calls is a degenerate response, not an answer. The continuation nudge couldn't catch it — it gates on the text being truthy — so an empty turn fell straight through to Terminal(reason="completed"). Empty turns are now re-prompted, bounded by the existing MAX_CONTINUATION_NUDGES cap.

Carve-out for SendUserMessage: a model that speaks through the outbox legitimately ends with empty assistant text, and agent_loop_compat already treats the outbox as the response text. The existing parity test caught that case when I first missed it.

2. A transport blip killed a 24-minute trial

regex-chess died on peer closed connection without sending complete message body (incomplete chunked read) seconds after a passing 1500-position fuzz run.

Transport drops now take the same bounded retry the idle watchdog already takes, with the original exception re-raised on exhaustion so the log names the real failure. Gated two ways:

• Status-bearing errors (401/529/400) and aborts fail fast — retrying a revoked token just burns budget on guaranteed 401s.
• Skipped once any text reached the caller. A re-attempt replays the response from scratch, so retrying after partial output would emit those chunks twice — exactly what query.py:1635 forbids. That check sits above this layer and can't see a retry taken here, so the gate has to live here too. Free for the eval: harbor runs without --include-partial-messages, so the callback is None in a scored trial.

3. AskUserQuestion was reachable in --print

fix-git called it twice, then ended with "let me explain what I found" instead of finishing; the verifier found the files untouched. It isn't in the initial toolset — the model reached it via ToolSearch.

Nothing headless can answer it (StreamJsonReader parses only user messages; there's no can_use_tool protocol), so it's unregistered rather than left registered with a stubbed responder. Parity, not special-casing — the TS reference's AskUserQuestionTool already reports itself disabled when nobody's at the keyboard. Removal happens before the allow/deny filters so --allowed-tools can't resurrect it, and it propagates to subagents (shared registry instance).

Test doubles gained remove_tool rather than the production call gaining a getattr guard — a silent no-op there is the bug class that left --allowed-tools dead for months (#739).

4. The idle-timeout message was written for a grader

It appended "Connection timed out", with a comment stating the phrasing was deliberate so eval harnesses would classify the failure as retryable. That optimizes for the scoreboard rather than the agent — and it was untrue, since an idle stream is not a connection timeout, so it pointed debuggers at the wrong subsystem. Removed; the test now pins the honest wording and asserts the old clause stays gone.

Diagnosed, deliberately NOT fixed

Two trials died on 401 OAuth access token has been revoked — not expiry (one died 4.7 minutes in against a 2h runway) but refresh rotation: containers hold a refresh-token-free copy, so anything rotating the host credential revokes the in-flight token. An in-adapter retry would hand clawcodex attempts the official harness wouldn't give another agent, so it's documented in the adapter with a recommendation to use an API key for scored runs.

Also unfixed and model-shaped rather than harness-shaped: on torch-tensor-parallelism neither agent could find a Python interpreter, but Claude Code stopped searching and wrote the deliverable (1.0) while clawcodex kept globbing until the budget expired. That's prompt-shaping and wants an A/B, not a guess.

Verification

• 8841 passed, 0 failed (1 deselected: test_connection_error_re_raises, a pre-existing hang confirmed on clean main).
• Every fix is revert-sensitive — each line deleted, intended test confirmed failing, file restored.
• Critic-reviewed; it caught the double-emit bug in fix: fix i18n readme #2, which is why the emitted-gate exists.

Estimated recovery if these hold: 3 (empty turns) + 1 (regex-chess) + 1 (fix-git) ≈ 5 tasks. That's an estimate until a rerun confirms it — and note both runs recorded n_retries: 0, so it rests on the fixes rather than on any harness retry behavior.

🤖 Generated with Claude Code