subagent: startup memory reserve prices a new spawn at earlier workers' whole-process-tree peaks (test/build children included)

[Premshay]

Problem

The spawn memory guard prices a new dedicated subagent from memory figures that include everything a worker ran, not just the agent process itself. On a host where subagents run test suites or builds, a new spawn is priced at the size of the biggest workload any worker has run, and it waits for that much free memory.

Three places use the whole-tree figure:

1. Live peaks set the next start's price. _effective_next_start_gb (src/kiro_crew/subagent.py:1379) takes the maximum over every live dedicated worker's peak_rss_gb. That peak comes from _proc_subtree_sample(info._pid) (src/kiro_crew/subagent_manager/monitoring.py:663, src/kiro_crew/subagent.py:915), i.e. the whole process tree.
2. Settled workers keep their peak reserved. In _startup_memory_reserve_gb (src/kiro_crew/subagent.py:1395), a settled worker still owes max(cost_gb, peak_rss_gb) - last_rss_gb. A worker that reached 20 GB during a test run and is now idle at 1 GB reserves 19 GB until it exits.
3. Learned costs persist it (new in fix(subagent): price the startup memory reserve at the learned cost #13294). _record_cost_impl (src/kiro_crew/subagent_manager/monitoring.py:768) stores the whole-tree peak_rss_gb, and _startup_cost_gb (src/kiro_crew/subagent.py:1454) prices every start at that agent's 90th percentile. Workload peaks now set a lasting price even when nothing heavy is running.

Line references are to main at 6b4556e.

Observed

On a 94 GB host with several agents running tests in parallel:

• One live worker reached a 20.8 GB tree peak while running a test suite. The admission floor rose to 24.8 GB (4.0 GB spawn_min_memory_gb plus 20.8 GB). With about 24 GB free, every spawn on the host was deferred; one caller's spawn_run waited about 8 minutes, then slot-queued.
• Learned-cost history (cost_samples.jsonl, last 50 dedicated samples per agent):

Agent	Median	90th percentile	Max
Agent A	1.15 GB	7.39 GB	11.29 GB
Agent B	1.88 GB	6.84 GB	12.97 GB
Agent C	1.83 GB	5.70 GB	12.55 GB

After #13294, a typical ~1.2 GB start for Agent A is priced at ~7.4 GB each time, because the 90th percentile reflects what those workers ran, not what the agent needs to start.

Why it matters

The reserve exists to cover the time between admitting a start and the new runtime reaching its resident size (#13294's burst case). A worker's test or build children say nothing about that. Counting them over-reserves in proportion to the heaviest workload anyone runs. On hosts with heavy local work, parallel spawning collapses without anything visibly failing: the result is long deferred_low_memory waits.

Suggested direction

Open to maintainers' preference: measure two things separately. Price the next start from the agent's own processes (the runtime plus its MCP children, which the subtree walk already recognises), and record that figure for the learned cost. Keep the whole-tree measurement for what it's good for: live host-pressure decisions such as the raw free-memory check and the reaper. A worker's future growth could still be covered by a separate, capped per-worker allowance rather than by its historical peak.

Relation to #13294

Complementary. #13294 correctly stops pricing starts at the 0.5 GB fallback. This issue is about what the learned and live figures measure.

[github-actions]

Automated triage summary — generated from the issue text alone, before any maintainer has read it. Treat every line as a starting point, not a verdict.

The spawn memory guard prices new dedicated subagent starts from whole-process-tree peak RSS, which includes test/build children that workers spawned, both live and via learned p90 cost samples. On a 94 GB host running parallel test suites, one worker's 20.8 GB tree peak pushed the admission floor to 24.8 GB, deferring every spawn (one spawn_run waited ~8 minutes before slot-queuing).

Possibly already reported

• Mochi bg poller retries spawn_run ~400/min with no backoff when deferred for low memory #13129 (Mochi bg poller retries spawn_run ~400/min with no backoff when deferred for low memory)

If one of these is the same problem, say so and we will close this as a duplicate.

_{Posted by issue-summary.yml. The model had no access to the repository — it read only this issue and the titles of recent ones.}

[dwu96]

🤖 Kiro Crew Auto-Pipeline [operator: dwu96#8c08bee0]: Routing to needs-investigation. Both premises check out on main at e4b5f09ede34c92d894534903b8247fcb0649da8, but the direction you leave open to maintainers is the part that decides the fix, and one feasibility claim does not hold.

Confirmed: _effective_next_start_gb (src/kiro_crew/subagent.py:1383) prices the next start at max([0.0, cost_gb, *(info.peak_rss_gb for info in dedicated)]), and _startup_memory_reserve_gb (:1399) charges a settled worker max(0.0, max(cost_gb, info.peak_rss_gb) - info.last_rss_gb). That peak does come from the whole-tree walk. The docstring at :1425-1429 already reasons about a neighbouring hazard — "a learned p90 far above what this particular worker turned out to need must not become a phantom reserve that no later sample can close" — and handles it by subtracting last_rss_gb. Your point stands that subtracting current RSS does not help when the peak itself measured a test suite.

The claim that needs correcting is "the runtime plus its MCP children, which the subtree walk already recognises". The walk recognises them only for counting. _proc_subtree_sample (:915) passes needles=(STUB_MODULE,), and SubtreeSample (src/kiro_crew/platform_compat.py:8608) carries a single rss_kb — the whole-subtree sum — with matched being a process count, not an RSS attribution. There is no needle-scoped RSS reading to switch to. Producing one means either a new field on that NamedTuple, whose docstring states each reading keeps its own sentinel precisely because the failure modes differ, or a second walk; it is shared by 6 call sites across subagent.py, subagent_manager/monitoring.py and mcp_gateway/pool.py, so the contract change is the actual work rather than the arithmetic.

That plus the two open choices you flag yourself — which process set counts as "the agent's own", and whether future growth gets "a separate, capped per-worker allowance" instead of a historical peak — is design judgment, so it is not mechanically derivable from the diagnosis and does not go to the automated queue.

Two neighbours for whoever picks this up, neither a duplicate: #12285 (unreadable cgroup usage in spawn memory deferrals) is about explaining a deferral, and #13129 is a retry-storm against deferred_low_memory — the same symptom you measured, from the caller's side.