[Juliusolsson05]

Fixes #1362

Problem

read_agent_transcript showed almost nothing a modern Codex agent did. Since code mode (Codex 0.144 and later), commands and patches run through custom_tool_call(name="exec"). The call's input is a JavaScript program that calls tools.exec_command({cmd: "…"}), tools.apply_patch(…), MCP tools and so on. The reader handled only function_call / function_call_output, so a Codex child's timeline, shell_commands and file_changes held its messages and nothing it ran.

The owner's corpus (2,457 rollouts) has 98,327 exec scripts and 7,538 top-level apply_patch custom calls. The exec_command arguments are JS object literals, and 73,789 of 87,184 are not JSON.

Change

• One grammar, shared. The renderer's lexical exec grammar moves unchanged from providers/codex/renderer/adapters/command.ts to src/shared/codex/execScript.ts:
  • it keeps matchingCallEnd, parseExecCommandArgument and the literal decoders;
  • it adds codexExecScriptCalls(script), which lists every tools.<name>(…) call in source order. Calls are found with an acorn parse; acorn is already a runtime dependency and the renderer's embedded-operation adapter parses the same scripts. So text that merely quotes tools.x( in a comment, string, template or regex is never a call, and regex-versus-division is decided by the grammar (verification b/c). A script that does not parse (98 of 99,256 recorded) falls back to a lexical scan, since it never ran as written;
  • the renderer imports it back, and its presentation rules stay where they were.
• Reader, exec scripts, in source order:
  • every exec_command argument decodes (92% of calls): one shell_command per call, with its cwd;
  • any argument is computed ({cmd, …} over a mapped array, concatenation, a template): one shell_command whose command is the script itself. A partial list would silently drop the unreadable commands, which is this bug again, and a guessed command is worse;
  • apply_patch: one patch item for the script, naming the files from the patch headers. They are decoded from a quoted argument, or read from the script text for templates and variables. An interpolated header (*** Update File: ${path}) resolves only through one const path = "literal" binding; otherwise the path is dropped, never shown as ${path} (review a, steering q83);
  • anything else (MCP, web__run, ALL_TOOLS queries): a tool_read for exec, so the timeline still shows that a script ran.
• Reader, other custom calls and outputs:
  • a top-level apply_patch custom call becomes a patch item;
  • custom_tool_call_output becomes the same raw-output item as function_call_output, still hidden unless include.rawToolOutputs.

Plan: docs/plans/2026-09-27-codex-custom-call-transcript-reader.md. Its evidence section has the corpus counts.

Tests (4 reader tests red on main, plus grammar unit tests)

• codex-0.157-custom-calls.jsonl: a slice of a recorded 0.157.1 rollout with one script of each census shape (a single command, two commands, a computed argument, a patch, a non-command script), three outputs, the prompt and the final answer.
  • Records are verbatim except for a same-length redaction of the account name; session_meta is left out.
  • The two large outputs are left out whole, because one echoes MCP tool descriptions.
  • The test asserts the timeline, shell_commands, raw outputs and inspect stats.
• codex-custom-call-forms.jsonl: six single recorded records. Provenance is in testing/fixtures/agent-transcripts/README.md.
  • a template-literal patch;
  • a top-level apply_patch;
  • a patch whose text quotes tools.exec_command(...); the old regex scan turned it into a command;
  • an MCP-only script, which pins the tool_read target;
  • two ${path} patches around a command; this fails with ${path} if the resolution is removed;
  • four calls with two template commands, which must give one whole-script item.
• The 0.157 test also asserts the joined text of a multi-block output.
• q86: a never-run if (false) command and patch appear in timeline, shell_commands and file_changes only with executed: "unknown". Removing the marker fails the test.
• src/shared/codex/execScript.test.ts covers the grammar edges from verification b and c (a regex after if (…) and after await, division after }) and the recorded tools["name"](…) form. These fail with the lexer alone.

Verification

• At ed86c13c (0 behind main): npx tsc -b is clean. src/main/agentTranscripts, src/providers/codex/renderer, src/shared/codex and the two MCP transcript-tool suites pass 218/218.
• The renderer suites pass unchanged, which shows the grammar move preserved behaviour.
• Corpus checks:
  • Lexer against the first regex scan (98,823 scripts): 65 differ, all phantom calls or malformed scripts.
  • Parse against lexer (99,256 scripts): 5 differ, all recorded tools["name"] calls only the parse sees. The parse costs 4.2 s over the whole corpus, 36 ms at worst.
• src/providers/shared/renderer + src/providers/codex: 358/361 in one batch at load average 147. The three failures were 8–11 s timeouts in systemic-findings.renderer.test.tsx, which passes 6/6 alone.
• Nothing needed the app: the reader runs in main over files, and the tests drive its real entry points.

Review (3 reviewers, round 1)

• b and c (major): a quoted tools.x( was read as a call. Fixed by the lexical scan in 52a82f85, with a recorded test. Then, after verification b and c found regex-versus-division cases no lexer settles, fixed at the root in 8955ff4b by parsing with acorn.
• a (minors) and steering q83: fixed in d3bdffff: the ${path} resolution, one item per script, the fixture README, and removal of an unpinned dead dispatch.
• Verification (a, and b twice): MERGE-READY. c: FIX-BEFORE-MERGE on residuals it named itself; see the disposition.
• Contract (steering q86, the manager's adjudication):
  • Every shell_command and patch entry read from Codex exec script source carries executed: "unknown" on the entry itself. The reader lists the calls a script contains, read without running it.
  • A call in a branch that never ran is listed only with that marker, and a test pins it. Codex's combined output cannot say which calls ran.
  • Top-level apply_patch custom calls are not marked, since they are the call Codex made.
  • The agent_transcript_read_file description names the field. Nested calls are listed in source order.
• Residuals:
  • search returns long items, including whole-script commands, head-truncated; this predates the PR;
  • scripts that do not parse (98 of 99,256; never ran as written) use a lexical best effort that can list a regex-quoted pseudo-call;
  • the fail-closed branches of ${path} resolution have no recording, and a re-assigned let binding resolves to its first value (0 recorded);
  • an exec call with empty input gives an empty tool_read;
  • Move to headers and spaced tools . x ( forms have no recording.

Out of scope

• Pairing outputs to their calls by call_id; the reader never paired function_call_output either.
• write_stdin and MCP calls inside scripts are only named, as the tool_read target.

🤖 Generated with Claude Code

[Juliusolsson05]

Disposition, review round 1 (a: MERGE-READY; b and c: FIX-BEFORE-MERGE) and steering q83.

• b and c (major): tools.x( inside comments, strings or patch text was read as a call and could become a command that never ran.
  • Fixed in 52a82f85: codexExecScriptCalls is now a lexical scan that skips comments, strings, templates (with ${…} nesting) and regex literals.
  • Over all 98,823 recorded scripts it differs from the regex in 65, always with fewer calls: 62 patch-text phantoms, 1 template-string phantom, and 2 malformed scripts.
  • Recorded test: a 2026-07-12 patch that quotes tools.exec_command(…); the old scan produced a command.
  • b's repro and its recorded tools.find/tools.some rows now yield only the real calls.
• c (minor): search returns long whole-script commands head-truncated. Kept as an explicit residual. It predates this PR: search on main matches every item's full text and returns it head-truncated, whatever the kind.
• c (minor): untested contract parts. The joined multi-block output and the tool_read target are now asserted.
• a and q83: an interpolated ${path} header was reported as a file. Fixed in d3bdffff.
  • It resolves only through one const path = "literal" binding that ends the expression; otherwise the path is dropped, so it fails closed.
  • Pinned at the reader entry point with the recorded 2026-09-24 row; removing the resolution fails with ${path}.
• a (survivors):
  • One patch item per script is pinned: the same row holds two calls.
  • The one-whole-script-item guard is pinned by a recorded four-call row. It is behaviour-equivalent at the reader boundary, because acceptDedupedItem folds adjacent identical items, and it is kept with a comment.
  • An unpinned dispatch for unwrapped custom calls, with no recorded use, is removed.
  • The fixture README with provenance is added.
• Main is merged in 168d9459. tsc is clean and the suites pass 194/194. The body is synced.
• Next: a focused recheck by b and c of their blocker, and a verification by a.

[Juliusolsson05]

Disposition, verification (b and c: FIX-BEFORE-MERGE on the lexer's regex-versus-division guesses). The original false-call blocker is confirmed fixed by both.

• The new cases: a regex after if (…) or await, and division after }. They are valid JS that no lexer heuristic reads both ways.
• Fixed at the root in 8955ff4b: calls are found with an acorn parse. acorn is an existing runtime dependency; the renderer already parses these scripts with it. The lexer is kept only for scripts that do not parse (98 of 99,256 recorded; they never ran as written).
• Corpus: the parse and the lexer disagree on 5 scripts, all recorded tools["name"] calls that the parse newly sees. It costs 4.2 s over the corpus, 36 ms worst case.
• Tests: every b and c reproduction is pinned in src/shared/codex/execScript.test.ts (693ad199), and they fail with the lexer alone.
• Gate: tsc clean; 198/198.
• Residual (c), unchanged: search returns long items head-truncated, which predates this PR.
• b and c are running a focused check of 693ad199; a's verification is still running.

[Juliusolsson05]

Disposition, verification 2 (a and b: MERGE-READY; c: FIX-BEFORE-MERGE, on items c labels residuals).

• c (major residual): a static reading lists calls in branches that never ran (if (false) tools.exec_command(…)).
  • This is inherent to reading a transcript: Codex's exec output is one combined text with no record of which calls executed, and the reader never runs scripts.
  • Ruling (23acb877): state the contract where callers read it. The agent_transcript_read_file description now says Codex commands and patches are the calls a script contains, read without running it; that a dead-branch call is still listed; and that a computed script is one command.
  • A unit test pins the dead-branch listing, so it stays a stated behaviour, not an accident.
  • Cost if wrong: over-counting commands written in dead branches. None of the recorded scripts reviewers checked has one.
• c (minors), kept as body residuals:
  • nested calls are listed in source order, not execution order;
  • the fallback for unparseable scripts can list a regex-quoted pseudo-call; those scripts never ran.
• b and c (test gaps): calls inside template interpolations, and nested in another tool's argument, are now pinned. Skipping TemplateLiteral nodes fails the test.
• a (residuals): the ${path} fail-closed branches have no recording, and a re-assigned let resolves to its first value (0 recorded). Both are listed in the body.
• Gate: 0 behind main; tsc clean; 217/217.
• Needs a call (manager): c's latest verdict is FIX-BEFORE-MERGE on what it calls residuals of the static-reading contract. Is the stated contract acceptable, or should Codex exec calls be reported under a separate "written, not proven run" kind?

[Juliusolsson05]

Disposition, steering q86 (the manager's adjudication of c's FIX-BEFORE-MERGE on the static-reading contract). The narrowed contract is accepted on the condition that the uncertainty is carried on the output entries themselves. Implemented in ed86c13c:

• Every shell_command and patch read from Codex exec script source carries executed: "unknown". This is a new optional field on both kinds in agentTranscriptTypes.ts, and the tool description names it.
• Top-level apply_patch custom calls stay unmarked: they are the call Codex made.
• Pinning test: a never-run if (false) command and patch appear in timeline, shell_commands and file_changes only with the marker; removing it fails the test.
• 0 behind main; tsc clean; 218/218.
• Fixes #1362 stays, under the q86 condition that holds.
• c gets a focused final check of the marker.

[Juliusolsson05]

Disposition, final check c (FIX-BEFORE-MERGE: de-duplication ignored executed). Fixed in 9ad90e3d: transcriptItemsEquivalent now compares the marker, so a recorded direct call and a script-source copy with the same timestamp and text stay two entries in either order. The test covers both orders and a truncated long-path patch keeping its marker; each part fails without its fix. Main is merged in 27a8c132; tsc is clean and the suites pass 219/219. c is rechecking.