Skip to content

ai(subprocess): unhandled stdin EPIPE kills the worker; temp dirs leak; force-push storms undebounced; budget under-books 6x #9479

Description

@JSONbored

Summary

Four independent low-effort defects in the AI subprocess path. Two are one-line fixes; one can take down the whole worker; one leaks disk indefinitely. Grouped because they are all in src/selfhost/ai.ts's spawn/lifecycle area and are cheap to fix together.

1. Unhandled EPIPE on CLI stdin crashes the entire worker (highest severity here)

src/selfhost/ai.ts:905-908:

if (o.input != null) { child.stdin?.write(o.input); child.stdin?.end(); }

There is no child.stdin.on("error", …). child.on("error") catches spawn failures only — it does not receive stdio stream errors.

If claude/codex exits before draining stdin (an unknown flag after a CLI upgrade, an immediate auth abort, an OOM kill), the ~250 KB write fails with EPIPE on an emitter with no error listener ⇒ uncaught exceptioninstallSelfHostCrashHandlers (src/selfhost/process-lifecycle.ts:81-95) logs and calls exit(1), taking down every in-flight queue job in the container.

  • Add child.stdin?.on("error", () => {}). The real failure is already surfaced by the exit-code and empty-output guards.
  • Test: a child that exits immediately while a large stdin write is in flight does not crash the process.

2. Per-call CLI temp directories are never deleted

src/selfhost/ai.ts:594-599 (isolatedCliCwd) creates mkdtemp(/tmp/loopover-ai-*) per call (:1022, :1139) and writes the composed system prompt into it (:1046).

Exhaustive search found no removal anywhere: no finally cleanup, no sweep in src/queue/retention.ts, no tmpfiles.d, no tmpfs mount for the loopover service in docker-compose.yml, no VOLUME/TMPDIR in the Dockerfile. These land on the container's writable overlay layer and survive until the container is recreated — the exact growth mode the compose file already documents and fixes for the runner service. It also leaves private repo review instructions on disk indefinitely.

  • rm(cwd, { recursive: true, force: true }) in a finally.
  • Test: the directory does not exist after a call returns, on both the success and throw paths.

3. Force-push storms are not debounced

Every dedup layer is keyed on head SHA — the queue coalesce key embeds head.sha (src/github/webhook-coalesce.ts:60-68) and the AI-review lock is …@${headSha}:${mode} (src/queue/ai-review-orchestration.ts:106-108). So N pushes in 60 s produce N full prologues (file list, up to 96k chars of grounding fetch, RAG and impact-map embeddings, enrichment POST) and N LLM calls.

The only brake is skipStaleReviewOutput (src/queue/processors.ts:11013-11016), gated on shouldPostPlaceholder. review burst (src/review/ops-wire.ts:122) alerts but does not throttle.

  • Add a short trailing quiet-window on synchronize, or use a PR-scoped (not SHA-scoped) coalesce key with head-SHA revalidation at dequeue.
  • Test: 5 pushes within the window ⇒ one review of the final head.

4. The neuron budget under-books actual spend by up to 6×

freeAiCalls counts 1–2 calls (src/services/ai-review.ts:2764-2765) while runWorkersOpinion can make 3 attempts × 2 models per slot (:1469, :1480). The tie-break judge is pre-budgeted at worst case (:2768-2771), and src/services/ai-slop.ts:87 does it correctly — the main review path is the outlier, so the runaway-loop backstop is 6× looser than it reads.

  • Budget the worst-case attempt count on the main review path, matching the judge and slop paths.
  • Test: budget accounting equals actual maximum call count.

Also worth folding in (same file, same spirit)

  • extractCliUsage parses the whole stdout twice per call (src/selfhost/ai.ts:1073, then again via recordCliUsageMetrics in the finally at :1090).
  • stdout/stderr accumulate into unbounded strings (:885-894) with no size cap.

Metadata

Metadata

Assignees

Labels

gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions