Spill large system prompts to a file to avoid execve E2BIG

[groeneai]

Problem

The clickhouse-ci-task-worker cron started failing with:

OSError: [Errno 7] Argument list too long
claude_agent_sdk._errors.CLIConnectionError: Failed to start Claude Code: [Errno 7] Argument list too long

at every invocation, regardless of which task it picked.

Root cause

Linux limits a single argv element to MAX_ARG_STRLEN = PAGE_SIZE * 32 = 131,072 bytes. The Claude Agent SDK passes the system prompt inline as one argument (subprocess_cli.py):

elif isinstance(self._options.system_prompt, str):
    cmd.extend(["--system-prompt", self._options.system_prompt])

build_system_prompt() concatenates SOUL.md + TASK.md + AGENTS.md + TOOLS.md + the first 300 lines of MEMORY.md + session context + skills section + 8 pre-recalled memU summaries.

Measured on a real worker workspace today:

Component	Bytes
SOUL.md	2,584
TASK.md	56,271
AGENTS.md	9,260
TOOLS.md	3,471
MEMORY.md (truncated to 300 lines)	13,280
Session context + skills	~3,000
Subtotal (no memU)	89,718
8 pre-recalled memU summaries (large categories like task_domain, infrastructure, contacts are multi-KB each)	40,000–60,000
Total	~130,000–150,000 ← over 131,072

Result: execve() returns E2BIG and the subprocess never starts.

Fix

The SDK already supports --system-prompt-file via SystemPromptFile = {"type": "file", "path": ...}. Switch to it when the assembled prompt exceeds 100 KB (well under the kernel limit, leaves headroom for env/argv overhead). Small prompts still go inline so prompt-caching prefix hits are unaffected.

The file is written to <workspace>/.nerve/cache/system_prompts/<session_id>.md. Resumes pick up the same path without re-writing. A lazy GC on each write deletes files older than 7 days.

Test plan

• All 376 existing tests pass (.venv/bin/pytest tests/ -v)
• Restarted nerve locally; confirmed cron clickhouse-ci-task-worker now starts cleanly with the file-based path (log shows Session ...: system prompt N bytes (> 100000), passing via file ...)
• Verified small-prompt sessions still use the inline path (no behavior change for them)
• Lazy GC removes files older than 7 days