Skip to content

feat: audit session rollouts - #693

Merged
LoopGlitch26 merged 6 commits into
Cygnux-Labs:mainfrom
kausthubhk:agent/issue-669
Nov 14, 2026
Merged

LoopGlitch26 merged 6 commits into
Cygnux-Labs:mainfrom
kausthubhk:agent/issue-669

Conversation

@kausthubhk

Copy link
Copy Markdown
Contributor

What and why

Closes #669.

tracekit audit reads local session rollout calls alongside Claude Code transcripts. Audit and the live hook share a call mapping for shell, patch, and MCP tools. Calls without a matching hook are skipped; the default transcript root is unchanged.

Tests use a redacted record shape checked against a real local rollout, with synthetic values. The docs explain the sessions root and that rollout arguments can differ from the model's submitted arguments. Audit remains a local, read-only replay against today's policy, not signed evidence.

How I ran it as a user

Ran tracekit audit ~/.codex/sessions --since 7d locally. Aggregate output only:

your agent's week: 2,511 tool calls; Tracekit would have held 0 and blocked 0

Verification

  • Affected Windows selection: 97 passed, 17 skipped, 19 deselected. Pyflakes, package build and twine validation passed.
  • Linux Python suite: 4,924 passed, 422 skipped, 10 xfailed, five failed. The five failures reproduced on the pristine base; those cases and the skipped documentation check passed after correcting the snapshot environment.
  • Linux TypeScript suite: 147 passed, five failed, 10 skipped. Four failures reproduced on the pristine base; the remaining old-signer-lock failure passed isolated feature and base reruns. All three TypeScript CI examples passed.
  • Linux root-marker selection, E8 v1/v2 gates and offline eval gates passed. Offline evals used Python 3.12 rather than CI's 3.11; E20 took about 31 minutes locally, exceeding the CI job's 25-minute timeout.
  • The Windows full Python suite was interrupted and has no completed result. The optional E8 transcript tailer was not configured because setfacl was unavailable.

Checklist

  • make check passes (the full Linux run had baseline environment failures; Windows full run was interrupted)
  • A regression test covers the change
  • User-visible changes are described above
  • Docs updated for the behavior change
  • Every commit is signed off

Security invariants

  • The agent still never holds a signing key or assigns sequence numbers
  • Nothing an agent-controlled process supplies is newly trusted by the signer or verifier
  • Gaps are still signed, never silent; a check that cannot run warns instead of passing
  • No verification code is shipped inside a bundle

Evidence format

  • This change does not touch the frozen v1 or published v2 formats

Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>

@LoopGlitch26 LoopGlitch26 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, and sorry this waited a day. Reading rollouts per line with the live hook's own mapping is the right shape, and #714's reader table can take it as it is. I read the diff (ba95e4a). CI is green apart from release-dry-run and Offline evals, which fail on main's old lock file (fixed by #692). What's left:

  1. Merge main (or rebase), so those two jobs run on the current lock file.
  2. The rewrite is in the mapping, not in the rollout. A rollout's response_item payload is the Responses API item the model returned (_responses, tracekit/integrations/harness_hooks.py L176); _use's mapping builds the PreToolUse payload Codex makes from it, and that payload is Codex's rewrite of the model's arguments (the # coerced: comment, L222). The new _codex_tool docstring ("Codex's rollout holds the arguments rewritten for its hook") and the docs/limits.md row ("Codex rollout arguments are Codex's rewrite of the model's submitted arguments") say the reverse. Please say instead that audit decides each call in the form Codex's PreToolUse hook gets it, and keep the Codex source references from _use's old docstring (core/src/tools/registry.rs:150-160, 943-960; …, L195-197) on _codex_tool, which now holds that mapping.
  3. $CODEX_HOME. Codex keeps its sessions in $CODEX_HOME/sessions when CODEX_HOME is set (init already reads it, tracekit/agent_hooks.py L851). Please name it in docs/audit.md, in the limits row's "What to do", and in the missing-folder message.
  4. tracekit audit's description (tracekit/audit.py L272) lost "a replay against today's policy"; please keep those words, as the report's disclaimer does.

@LoopGlitch26

Copy link
Copy Markdown
Member

#730 has merged and added a table of readers to tracekit/audit.py (READERS: an agent's name, its default folder and a reader yielding (time, session, tool, args) or None for an unreadable line). Please merge main and make Codex's rollouts one entry there, rather than a per-line check in the Claude Code loop, so tracekit audit with no folder reads both.

@LoopGlitch26

Copy link
Copy Markdown
Member

What's left here, in one place:

  1. Merge main and make Codex's rollouts one entry in READERS (tracekit/audit.py), so tracekit audit with no folder reads them beside Claude Code's transcripts.
  2. The _codex_tool docstring and the docs/limits.md row say audit decides each call in the form Codex's PreToolUse hook gets it, and _codex_tool keeps the Codex source references from _use's old docstring.
  3. $CODEX_HOME/sessions is named in docs/audit.md, in the limits row's "What to do" and in the missing-folder message.
  4. tracekit audit's description keeps "a replay against today's policy".

Items 2 to 4 are from the review of 11-11; item 1 is from the comment above.

@kausthubhk

Copy link
Copy Markdown
Contributor Author

on it

Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
@kausthubhk

Copy link
Copy Markdown
Contributor Author

@LoopGlitch26 pls hold off on your review , few things are pending

@LoopGlitch26

Copy link
Copy Markdown
Member

Noted. I'll wait for your next push and review that head.

Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
@LoopGlitch26

Copy link
Copy Markdown
Member

Thanks for the quick turnaround, and for making Codex one entry in READERS. All four points from the last review are fixed: main is merged; the _codex_tool docstring and the limits row now say audit decides each call in the form Codex's PreToolUse hook gets it, and the Codex source references are kept; $CODEX_HOME/sessions is in the docs, the limits row and the missing-folder message; and the description keeps "a replay against today's policy". tracekit audit with no folder now reads Claude Code and Codex sessions together.

I read the diff at 97198f7. In a sealed container I ran the audit, feedback, harness-hook, docs and help tests (they all pass) and fed the reader some odd rollout lines. Three things are left:

  1. A call with no call_id disappears from the report. _codex_tool now returns None for it (tracekit/integrations/harness_hooks.py:229), and _codex skips it as a call that fires no hook (tracekit/audit.py:202). With this rollout line:

    {"timestamp":"2026-11-13T10:00:00Z","type":"response_item","payload":{"type":"function_call","name":"exec_command","arguments":"{\"cmd\":\"rm -rf ~\"}"}}

    tracekit audit DIR --agent codex prints 0 tool calls and skipped malformed lines: 0. A "call_id": "" or "call_id": 7 gives the same result. Codex writes a call_id on every call (call_id: String in its ResponseItem::FunctionCall), so a line without one has been edited. The file is only the agent's own claim, so audit should show the call rather than let a missing field hide it. The Claude Code reader already decides a tool use that has no id, it just doesn't dedup it (audit.py:153-157). Please do the same here: put the id check back in _use, where it was before this PR (the model record needs an id to link the call), remove it from _codex_tool, and in _codex build the dedup key only when call_id is a non-empty string. Then the no-id line in test_codex_reader_duplicate_malformed_since_and_no_hook (tests/test_audit.py:890) counts as a call.

  2. Some guards have no test, and the suite still passes with each one removed. I checked this by removing them one at a time in a sealed container. What each guard prevents comes from reading the code; with the guards in place, both lines below are handled as described.

    • The try/except around _codex_tool (audit.py:197-201). Without it, a line whose arguments nest deeper than json.loads can take ("arguments": "[[[[…", with 100,000 [) raises a RecursionError out of audit(). main doesn't catch it, so the run ends in a traceback instead of a report. With the guard, it counts as one malformed line (I checked).
    • The namespace check (harness_hooks.py:216-217). Without it, "namespace": 5 raises an AttributeError that the except above doesn't catch, so you get a traceback again. With the check, the call is decided under its bare name.
    • The string check on call_id (harness_hooks.py:229). This one no longer matters if you take point 1.

    One rollout line per guard in the malformed test would cover them. Separately, if not isinstance(args, dict) (audit.py:205-207) can never be true, because agent_hooks.tool always returns a dict (its first line is dict(_input(ti))). Please drop it. As far as I can see, only RecursionError can come out of _codex_tool, so the ValueError, TypeError on L199 can go too.

  3. Docs

    • docs/audit.md:34: the ROOT bullet lists the Cursor, Cline and Gemini CLI folders with their --agent, but not Codex.
    • docs/audit.md:98-101 says each call is "mapped to the form Codex sends to its PreToolUse hook", and then that "apply_patch becomes Edit". Codex's hook gets apply_patch as apply_patch; Edit comes from Tracekit's own hook mapping. That's the same distinction as point 2 of the last review. Something like this would fix it: "Each call is rebuilt as the PreToolUse payload Codex sends for it (an exec_command arrives as Bash, with its cmd as command; an MCP tool as mcp__<server>__<tool>), then decided as Tracekit's Codex hook decides that payload (apply_patch as an Edit of the files its patch names)."
    • The limits row (docs/limits.md:924) should say which Codex calls audit doesn't read, as the Gemini CLI row does for its sandbox:
      • Archived sessions. When a session is archived, Codex moves its rollout to $CODEX_HOME/archived_sessions/ (codex-rs/thread-store/src/local/archive_thread.rs), and audit doesn't read that folder by default. tracekit audit ~/.codex/archived_sessions --agent codex reads it.
      • codex exec --ephemeral runs. They write no rollout, so there is nothing to read.
      • Rollouts from before September 2025. Back then each line was the bare item ({"type":"function_call","name":"shell",…}) with no response_item envelope, so these files show 0 calls and 0 malformed lines. Saying so in the row is enough; there's no need to read the old format.
    • The row's "Why" column lost "and each needs a reader of its own". That's the reason the other agents need a calls file, so please keep it.

CI was still running on this head when I looked, so I'll check it again on your next push.

Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
@LoopGlitch26

Copy link
Copy Markdown
Member

Thanks, this round covers everything, and quickly too. From my side it's ready.

What I checked at 5fd52b6:

  1. Calls with no call_id are counted now. The id check is back in _use, _codex_tool maps without it, and _codex keys dedup only on a non-empty string id. In a sealed container, a rollout line running rm -rf ~ with no call_id, with "" and with 7 now gives one denied Bash call each (it used to give 0 calls and 0 malformed lines), and _use still returns None for all three.
  2. The guards are tested and the dead code is gone. The malformed test now has the 100,000-[ arguments line, "namespace": 5 and the three ids, and the isinstance(args, dict) check and the extra ValueError, TypeError are removed. I re-ran the mutation set: the suite now catches removing the try/except, the namespace check, or _use's id check, and catches moving the id check back into _codex_tool. The earlier catches still hold (the rollout- filter, dedup, --since, malformed lines, CODEX_HOME).
  3. Docs. The ROOT bullet names Codex. The "What it reads" paragraph separates Codex's PreToolUse payload from Tracekit's mapping. The limits row covers archived sessions, --ephemeral and pre-September-2025 rollouts, and keeps "each needs a reader of its own".

I also ran it as a user would, with a fake ~/.codex. tracekit audit --since 7d with no folder read the session and counted a repeated call once and the call without an id once. tracekit audit ~/.codex/archived_sessions --agent codex read the archived rollout. --share printed "1 coding-agent session", and a missing folder named $CODEX_HOME/sessions in its message. The audit, feedback, harness-hook, docs and help tests pass (216 passed, 6 skipped).

Two optional one-liners, fine in this PR or not at all:

  • The comment on tracekit/audit.py:202 still says "or missing call id", which _codex_tool no longer checks.
  • The docs say calls with no valid id are counted without deduplication, but no test pins that: the three no-id lines have different ids (none, "", 7), so keying dedup on any id still passes. A second _rollout(no_id) line (8 calls) would pin it.

CI was still running when I looked (23 passed, 10 pending, none failed).

@LoopGlitch26
LoopGlitch26 enabled auto-merge (squash) November 14, 2026 10:14
@LoopGlitch26
LoopGlitch26 merged commit b86390c into Cygnux-Labs:main Nov 14, 2026
34 checks passed
@LoopGlitch26

Copy link
Copy Markdown
Member

Merged, thanks @kausthubhk. tracekit audit now reads Codex sessions next to Claude Code's: tracekit audit --since 7d with no folder checks both, a call with a missing or odd call_id is still counted, and the limits row says which Codex sessions it can't see. It ships in 1.7.0.

If you'd like another, #847 (SDK retries running past a call's timeout) and #843 (Crush: record the model's text from TurnEnd) are open and specified.

A request: if you use a coding agent, would you run Tracekit on it for a week, on your own projects?

pip install -U tracekit-ai
tracekit audit --since 7d   # read-only: what it would have held last week
tracekit init --dev --v2    # then record and guard from now on

After a week, tracekit report --share prints counts only (no commands, paths or content). Post it with the field-report form; that's all we need.

wanjinhao1 pushed a commit to wanjinhao1/Tracekit that referenced this pull request Nov 15, 2026
No conflicts.

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
@kausthubhk
kausthubhk deleted the agent/issue-669 branch November 16, 2026 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tracekit audit: read Codex session rollouts

2 participants