Skip to content

oom(18/29): bound agent hooks & automations state - #10311

Closed
nwparker wants to merge 1 commit into
oom/17-b-native-chat-speechfrom
oom/18-b-agent-hooks-automations
Closed

nwparker wants to merge 1 commit into
oom/17-b-native-chat-speechfrom
oom/18-b-agent-hooks-automations

Conversation

@nwparker

@nwparker nwparker commented Jul 24, 2026 •

Copy link
Copy Markdown
Contributor

OOM hardening slice 18/29 — bound agent hooks & automations state

Part of a 29-PR stack re-landing #10179 ("bound OOM-prone accumulators across Orca"), which was reverted in #10255 for being too large (~1,600 files). This is slice 18/29 (base: oom/17-b-native-chat-speech). Merge in numeric order. The full stack's end-state is byte-for-byte the commit that passed #10179's complete CI matrix (36k+ tests, multi-OS).

Summary

Slice #18 re-introduces OOM ceilings across agent-hooks, automations, and skills: bounded file reads (config/script/plugin/bundle/status), streaming directory/lock cleanup, retained-metadata byte caps on the agent-hook status server, bounded skill-discovery traversal, and concurrency caps on remote fan-outs. Almost all changes are overload-only fail-closed guards; a few introduce user-visible ceilings (skill discovery now throws on overload; Hermes run pages clamp to 100; WSL skill-scan buffer shrinks 128MiB→8MiB).

ELI5

Imagine the app used to read whole files and folders no matter how huge, and kept lists that could grow forever until it ran out of memory and crashed. This change adds size limits everywhere: "don't read a file bigger than X", "don't keep more than N items", "don't do more than 4 of these at once". If something is absurdly big it's skipped or an error is shown instead of crashing. For normal users nothing changes; only people with truly gigantic files or thousands of items hit the new walls.

Proof of OOM fix

  • agent-hook-file-limits.ts: config 64MiB / 1M structural tokens / depth 128, plugin 1MiB, managed script 1MiB — tests: agent-hook-file-bounds.test.ts, hooks-file-bounds.test.ts
  • server.ts: last-status file 16MiB + 1M structural tokens + depth 128; connection watermarks cap 1024; retained id ≤1024B, path ≤16KiB; serializeStatusFile drops oldest entries to stay under 16MiB — tests: server.test.ts, server-status-cache-bound.test.ts, server-retained-metadata-bounds.test.ts
  • managed-hook-lock-records.ts: opendir bufferSize 32, cleanup concurrency 8, lock record ≤2MiB (streams dir instead of readdir array) — test: managed-hook-lock-cleanup-bounds.test.ts
  • managed-hook-owner-identity.ts: host-token file ≤1024B — test: managed-hook-owner-identity.test.ts
  • migration-unsupported-pty-state.ts: MAX_ENTRIES 500, per-entry ≤16KiB (FIFO evict) — test: migration-unsupported-pty-state.test.ts
  • branch-rename-failure-output.ts: key ≤4KiB, output ≤520KiB, MAX_ENTRIES 32 — test: branch-rename-failure-output.test.ts
  • first-work-branch-rename.ts: in-flight cap 32, settled cache 500 (keys sha256-hashed) — test: first-work-branch-rename.test.ts
  • dispatch-tokens.ts: MAX_ENTRIES 1024 with oldest-unused eviction + TTL prune — test: dispatch-tokens.test.ts
  • hermes-cron-output.ts: page ≤100 runs, run-refs ≤10000 (totalSaturated), primary output 5MiB, referenced-log bounded, SQLite iterate+substr caps — test: hermes-cron-output.test.ts
  • external-manager.ts: run-count concurrency 4, remote-manager fan-out concurrency 4 — test: external-manager-remote-fanout.test.ts
  • skill-discovery-limits.ts: roots 512, candidates 1024, dirs 20000, entries 100000, result 4MiB, WSL output 8MiB, per-package dirs 512/entries 10000 — tests: skill-discovery-limits.test.ts, discovery.test.ts, skill-discovery-wsl.test.ts
  • skill-bundle-artifacts.ts: manifest 2MiB, registry 16MiB, release-mapping 2MiB + 1M structural tokens/depth 128 — test: skill-bundle-artifacts.test.ts
  • wsl-hook-relay-launch.ts: version file 4KiB, bundle 8MiB — test: wsl-hook-relay-file-bounds.test.ts
  • wsl-hook-relay-sentinel.ts: pending bytes (MAX_MESSAGE_SIZE+HEADER)*2 and MAX_BUFFERED_FRAME_CHUNKS before kill — test: wsl-hook-relay-sentinel.test.ts
  • agent-trust-presets.ts / remote-agent-trust-presets.ts: git-pointer ≤64KiB, agent-state config ≤4MiB (MAX_AGENT_STATE_FILE_BYTES) — tests: agent-trust-presets.test.ts, remote-agent-trust-presets.test.ts
  • agent-hook-sftp-text-reader.ts / installer-utils-remote.ts: SFTP reads stream-bounded to maxBytes with 10s timeout; mkdirp probes via stat instead of readdir — test: installer-utils-remote.test.ts

Regressions

Normal-path (ordinary use, below the new limits): ⚠️ needs sign-off:

  • low: Under 1024 concurrent in-flight automation dispatch tokens, createAutomationDispatchToken returns an untracked token (eviction can't reclaim in-flight entries), so that dispatch's begin/finish reservation no-ops. (trigger: Only at 1024 simultaneously in-flight automation dispatches — practically unreachable.)

Overload-only (extreme input / sustained backpressure) — intended, documented degradations:

  • Skill discovery throws SkillDiscoveryLimitError (rejecting the whole scan) only past 512 roots / 1024 skills / 100k entries / 20k dirs / 4MiB retained result — far above any real install.
  • Agent-hook status server drops retained metadata fields (launchToken/toolUseId/worktreeId/transcriptPath) or rejects an event only when a single id >1KiB or path >16KiB, and drops oldest persisted rows only past a 16MiB last-status file.
  • Hermes run-count badge saturates at 10000 (reports totalSaturated) only for jobs with >10k runs.
  • Managed-hook lock cleanup skips an oversized (>2MiB) lock record and streams the directory in 8-wide batches only when the lock dir is pathologically large.
  • migration-unsupported-pty and dispatch-token maps FIFO-evict only past 500 / 1024 live entries; oversized (>16KiB) PTY entries are rejected.
  • WSL relay sentinel kills the child only if post-sentinel data floods before the transport attaches (chunk count / byte cap exceeded).
  • Codex/Copilot trust presets skip rewriting a config only when it exceeds the 4MiB read ceiling, or fall back to workspace root when a .git pointer exceeds 64KiB.
  • discoverSkills() now REJECTS (throws SkillDiscoveryLimitError) instead of returning a partial list when traversal/result budgets are exceeded, so an extreme skills tree fails the entire picker rather than degrading. Fail-closed and intentional, but a hard failure a human should acknowledge. (only when: >512 roots, >1024 SKILL.md candidates, >100k dir entries, >20k directories, or >4MiB of retained skill metadata; not reachable in ordinary installs.)
  • WSL skill-discovery subprocess maxBuffer dropped from 128MiB to 8MiB; a large WSL skill collection whose base64 output lands between 8–128MiB now fails the scan (ENOBUFS) where it previously succeeded. (only when: Many/large SKILL.md files under WSL producing >8MiB of encoded scan output.)
  • Hermes run pages are clamped to 100 runs per page (safePageSize=min(100,...)); a caller requesting a larger page silently receives at most 100 rows, and the run count badge saturates at 10000. (only when: A UI/API path requesting >100 Hermes runs per page or a job with >10k runs.)

Build / CI

  • Part of the Main services + CLI group; this group's cumulative tree is interlinked, so it becomes typecheck-green at its checkpoint PR Add dismiss button to update notifications #25 (whole-repo interconnection — see stack README). Content is exact production code.

Adversarial review

REFUTED the one claim that looked like a user-visible regression: Hermes run pages clamping to 100 (safePageSize=min(HERMES_RUN_PAGE_MAX_RUNS=100,...) at hermes-cron-output.ts:687). The caller external-manager.ts:330 ALREADY clamps pageSize to Math.min(100, ...) in BASE (pre-existing), so the internal cap is pure defense-in-depth and no UI/API path ever requested >100. Not a regression.

Confirmed all other ceilings are far above ordinary use:

  • Skill discovery (skill-discovery-limits.ts): 512 roots / 1024 skills / 20k dirs / 100k entries / 4MiB retained. discoverSkills() propagates SkillDiscoveryLimitError (does NOT catch/degrade — confirmed in discovery.ts:270-286 mapWithConcurrency), so an extreme skills tree rejects the whole picker. Ordinary installs have <50 skills; unreachable normally.
  • Agent-hook config read cap 64MiB (comment sized for the supported 30,000-hook cleanup); plugin/script caps 1MiB.
  • Status server retained-metadata: id 1KiB / path 16KiB / last-status file 16MiB — normal panes have tiny payloads.
  • Closed-tab/pane-alias dedup maps FIFO-evict at 1024.

serializeStatusFile manual serializer (server.ts:2106) is the highest-risk change: builds JSON by hand, iterates entries in reverse to keep newest within the 16MiB budget, then fragments.reverse() to restore order. Separator/prefix byte accounting is consistent with the final ','-join; keys re-quoted via JSON.stringify; values via stringifyJsonWithinByteLimit. Covered by server-status-cache-bound.test.ts. No ordering/correctness defect found; 16MiB is unreachable in normal use (handful of small entries). Byte estimator uses length*2 (over-counts UTF-8 for ASCII) so the 4MiB skill budget trips conservatively early — still far above normal.

dispatch-tokens identityDigest(sha256) rewrite is functionally equivalent for match/reserve/release/clear. first-work-branch-rename sha256 dedupe with in-flight cap 32 correctly retries dropped attempts (no worktree poisoning). No normal-path regression and no correctness bug confirmed. Safe to open as a draft PR for human review; the two low items are behavior-change acknowledgements, not bugs.

Files

60 files changed (+3225 / −472).


🤖 Decomposed, reviewed & assembled by Claude Code from the reverted #10179. Draft — do not merge out of order.

Made with Orca 🐋

@nwparker
nwparker force-pushed the oom/17-b-native-chat-speech branch from 814e0ac to a84c732 Compare July 24, 2026 07:48
@nwparker
nwparker force-pushed the oom/18-b-agent-hooks-automations branch from c34b6da to fc15b24 Compare July 24, 2026 07:48
@nwparker
nwparker force-pushed the oom/17-b-native-chat-speech branch from a84c732 to 8b66f08 Compare July 25, 2026 02:21
@nwparker
nwparker force-pushed the oom/18-b-agent-hooks-automations branch from fc15b24 to 3bb8160 Compare July 25, 2026 02:21
@nwparker
nwparker force-pushed the oom/17-b-native-chat-speech branch from 8b66f08 to 704235f Compare July 25, 2026 02:40
@nwparker
nwparker force-pushed the oom/18-b-agent-hooks-automations branch from 3bb8160 to e0b2660 Compare July 25, 2026 02:40
@nwparker
nwparker force-pushed the oom/17-b-native-chat-speech branch from 704235f to 5c1dffb Compare July 25, 2026 03:11
@nwparker
nwparker force-pushed the oom/18-b-agent-hooks-automations branch from e0b2660 to 6764f21 Compare July 25, 2026 03:11
Files: 60 applied, 0 deleted (from 6eb70d8)

Co-authored-by: Orca <help@stably.ai>
@nwparker
nwparker force-pushed the oom/17-b-native-chat-speech branch from 5c1dffb to 375ef05 Compare July 25, 2026 03:34
@nwparker
nwparker force-pushed the oom/18-b-agent-hooks-automations branch from 6764f21 to 67854b5 Compare July 25, 2026 03:34
@nwparker

Copy link
Copy Markdown
Contributor Author

Superseded by #10548, which re-scopes this work to the bound-at-ingress doctrine (see #10533 and src/shared/memory-safety/README.md).

An audit found the original #10179 scope was ~50x the size of the fix for the crash it targeted, and that bounds had landed ahead of their callers. Every file in these slices was re-adjudicated individually: ingress bounds (transport frames, PTY flow control, on-disk transcripts, parsing, enumeration, media decode, user-keyed accumulators) were kept; interior ceilings behind an already-bounded input were dropped. 1,882 files → 1,155, full suite green, and no main feature regressed.

No memory protection was lost — only speculation.

@nwparker nwparker closed this Jul 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant