Repository navigation
feat: audit session rollouts - #693
Conversation
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
LoopGlitch26
left a comment
There was a problem hiding this comment.
Thanks, and sorry this waited a day. Reading rollouts per line with the live hook's own mapping is the right shape, and #714's reader table can take it as it is. I read the diff (ba95e4a). CI is green apart from release-dry-run and Offline evals, which fail on main's old lock file (fixed by #692). What's left:
- Merge main (or rebase), so those two jobs run on the current lock file.
- The rewrite is in the mapping, not in the rollout. A rollout's
response_itempayload is the Responses API item the model returned (_responses,tracekit/integrations/harness_hooks.pyL176);_use's mapping builds the PreToolUse payload Codex makes from it, and that payload is Codex's rewrite of the model's arguments (the# coerced:comment, L222). The new_codex_tooldocstring ("Codex's rollout holds the arguments rewritten for its hook") and thedocs/limits.mdrow ("Codex rollout arguments are Codex's rewrite of the model's submitted arguments") say the reverse. Please say instead that audit decides each call in the form Codex's PreToolUse hook gets it, and keep the Codex source references from_use's old docstring (core/src/tools/registry.rs:150-160, 943-960; …, L195-197) on_codex_tool, which now holds that mapping. $CODEX_HOME. Codex keeps its sessions in$CODEX_HOME/sessionswhenCODEX_HOMEis set (init already reads it,tracekit/agent_hooks.pyL851). Please name it indocs/audit.md, in the limits row's "What to do", and in the missing-folder message.tracekit audit's description (tracekit/audit.pyL272) lost "a replay against today's policy"; please keep those words, as the report's disclaimer does.
|
#730 has merged and added a table of readers to |
|
What's left here, in one place:
Items 2 to 4 are from the review of 11-11; item 1 is from the comment above. |
|
on it |
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
|
@LoopGlitch26 pls hold off on your review , few things are pending |
|
Noted. I'll wait for your next push and review that head. |
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
|
Thanks for the quick turnaround, and for making Codex one entry in I read the diff at 97198f7. In a sealed container I ran the audit, feedback, harness-hook, docs and help tests (they all pass) and fed the reader some odd rollout lines. Three things are left:
CI was still running on this head when I looked, so I'll check it again on your next push. |
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
Signed-off-by: Kausthubh <26836237+kausthubhk@users.noreply.github.com>
|
Thanks, this round covers everything, and quickly too. From my side it's ready. What I checked at 5fd52b6:
I also ran it as a user would, with a fake Two optional one-liners, fine in this PR or not at all:
CI was still running when I looked (23 passed, 10 pending, none failed). |
|
Merged, thanks @kausthubhk. If you'd like another, #847 (SDK retries running past a call's timeout) and #843 (Crush: record the model's text from A request: if you use a coding agent, would you run Tracekit on it for a week, on your own projects? After a week, |
No conflicts. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
What and why
Closes #669.
tracekit auditreads local session rollout calls alongside Claude Code transcripts. Audit and the live hook share a call mapping for shell, patch, and MCP tools. Calls without a matching hook are skipped; the default transcript root is unchanged.Tests use a redacted record shape checked against a real local rollout, with synthetic values. The docs explain the sessions root and that rollout arguments can differ from the model's submitted arguments. Audit remains a local, read-only replay against today's policy, not signed evidence.
How I ran it as a user
Ran
tracekit audit ~/.codex/sessions --since 7dlocally. Aggregate output only:Verification
setfaclwas unavailable.Checklist
make checkpasses (the full Linux run had baseline environment failures; Windows full run was interrupted)Security invariants
Evidence format