Skip to content

Add a two-tier stalled-output fast-fail timeout to the miner's CLI-subprocess coding-agent driver #5167

Description

@JSONbored

Context

createCliSubprocessCodingAgentDriver in packages/gittensory-engine/src/miner/cli-subprocess-driver.ts currently wires only a single flat timeoutMs with no first-output liveness signal. src/selfhost/ai.ts already has a proven firstOutputTimer/resolveClaudeFirstOutputTimeoutMs pattern for the identical claude/codex binaries, built specifically because a naive single-timeout design caused a real production outage (#4994/#5053). The CLI-subprocess driver has no equivalent protection today.

Dependencies

None — independently shippable. This is a self-bounded, fail-closed addition to the driver's own subprocess-handling code and does not touch attempt/governor state, so it can be implemented and tested without any other issue in this batch landing first.

Requirements

  1. Add an optional firstOutputTimeoutMs field to CliSubprocessDriverOptions in cli-subprocess-driver.ts.
  2. Port the liveness-timer pattern from src/selfhost/ai.ts's firstOutputTimer/resolveClaudeFirstOutputTimeoutMs: start a timer when the subprocess spawns, and clear/reset it on the first stdout chunk received.
  3. If no stdout is produced before firstOutputTimeoutMs elapses, kill the subprocess early and report a distinct, clearly-labeled "stalled" error (not the same error string as a genuine full timeout).
  4. If the subprocess produces any stdout before firstOutputTimeoutMs, the first-output timer must not fire again for that invocation — normal full-timeout behavior (governed by the existing flat timeoutMs) continues to apply.
  5. When firstOutputTimeoutMs is not provided, behavior must be unchanged from today (opt-in, backward compatible).
  6. This change must scope the kill action strictly to the CLI subprocess itself — it must not read, write, or mutate any attempt/governor state, must not make any retry/requeue decision, and must not touch any autonomous-loop control-flow. The driver only reports the outcome (killed-for-stall vs. genuine timeout) back to its caller; any decision about what to do next remains entirely outside this issue's scope.

Deliverables / Acceptance Criteria

  • CliSubprocessDriverOptions gains an optional firstOutputTimeoutMs field
  • Subprocess is killed early and reported as "stalled" when zero stdout arrives before firstOutputTimeoutMs
  • A subprocess producing output before firstOutputTimeoutMs is not affected by the first-output timer
  • Existing flat timeoutMs full-timeout behavior is preserved unchanged
  • No attempt/governor state is read or mutated by this change; no retry/requeue logic added
  • Behavior is unchanged when firstOutputTimeoutMs is omitted

Test Coverage Requirements

This PR must ship with full test coverage for every changed line and branch — the repo's Codecov patch gate requires 99%+ coverage and the house standard is to aim for 100%, including both sides of every conditional/nullish-coalescing branch introduced. Add: (1) unit tests covering a subprocess that produces zero stdout past firstOutputTimeoutMs (killed early, distinct "stalled" error), a subprocess that is slow-but-produces-output before firstOutputTimeoutMs (not killed early), and a subprocess that produces output but never exits before the full timeoutMs (genuine timeout, existing behavior preserved), (2) an invariant test asserting that no code path introduced by this change ever reads or writes attempt/governor state — the driver's return value only reports which timeout fired, nothing else, and (3) a regression-style test mirroring the #4994/#5053 outage scenario (using the existing injected-spawn fake pattern) to prove this specific failure mode is now caught, even though this is a new capability rather than a fix to already-shipped behavior.

Expected Outcome

An operator running AMS's CLI-subprocess coding-agent driver gets a fast, clearly-labeled failure when the underlying claude/codex CLI hangs silently at startup, instead of waiting out the full flat timeout with no distinguishing signal — mirroring the protection ORB already has in production.

Links & Resources

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions