Skip to content

Add an offline Vercel AI approval/resume example - #514

Merged
LoopGlitch26 merged 4 commits into
Cygnux-Labs:mainfrom
sunxiuguo:agent/issue-392
Nov 2, 2026
Merged

LoopGlitch26 merged 4 commits into
Cygnux-Labs:mainfrom
sunxiuguo:agent/issue-392

Conversation

@sunxiuguo

@sunxiuguo sunxiuguo commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Current status — 2026-11-02 07:39 UTC

Ready for review after the requested revision in a41d3c185a6981af1722e05c2ccd7c1acf1c0ad5. The example now holds the harmless pretend terraform destroy call under TK-A001 and deletes its private state file after successful close. Actual separate-process and failure-path validation is described in the revision report.

On this exact head, CI, DCO and Supply chain all completed successfully (33 successful checks, 1 skipped coverage check). This supplies the remote full-suite evidence requested by the maintainer; it does not change the historical local read-only-HOME failure below.

The original submission transcript and checklist below are preserved as historical evidence for 8ccec7fe…, not as the current source/output or draft status.


What and why

Fixes #392.

Adds an offline MockLanguageModelV4 example that saves a held run and framework messages to a private JSON file, prints the signer's approval command, and resumes the same call in another process using RunHandle, tracekitAI and approvalResponses. It closes the run after handling the decision. The README Exit bullet and limits reattach row link to it.

The example contains assertions for the hold and execution counts. No SDK, dependency, policy or evidence-format changes are included.

Implementation was AI-assisted and received independent automated review. Its status-message finding was fixed; final review found no unresolved findings. This was automated review, not human review.

At initial submission this was a draft because full-suite validation was incomplete, as described below; see the current status above.

How I ran it as a user

Linux x64, Python 3.12.14, Node 24.19.0, npm 11.9.0 and ai 7.0.159. Each process used the same scratch TRACEKIT_RUNTIME_DIR and XDG_DATA_HOME; HOME was unchanged. The first and resumed Node processes had distinct PIDs (15542 and 15572), with approval in a third process (15568). All exited 0.

The transcript below uses redacted aliases RUN_A, APPROVAL_A and [scratch]; these are not literal values to paste. No run tokens or signing material are included.

After npm run build, from sdk/typescript:

$ node examples/v2_vercel_ai_resume.mjs /tmp/[scratch]/resume-final.json
Held: TK-SHELL-PARSE; executed 0 times; run RUN_A
Saved private state to /tmp/[scratch]/resume-final.json. Run left open.
Approve in another terminal: tracekit approvals approve APPROVAL_A
Then run this example again with the same state file and signer.

A separate process applied the normal local approval for this synthetic fixture:

$ python -m tracekit approvals approve APPROVAL_A --reason "Synthetic issue 392 local workflow test"
approved: APPROVAL_A (self-approved: dev mode only)

The resumed invocation:

$ node examples/v2_vercel_ai_resume.mjs /tmp/[scratch]/resume-final.json
(pretend) ran echo "unterminated
Approval handled.
Executed 1 time(s); closed run RUN_A.

Verification:

$ python -m tracekit last -o /tmp/[scratch]/resume-final.tkb
run RUN_A (vercel-resume-example)
Integrity: VERIFIED.

The signed bundle contains exactly one approval request, answer, consumption, tool result, close and finalization, with the same approval ID throughout. Pending retries and completed-state replay executed nothing; verification after replay remained VERIFIED. Rejection separately executed nothing, closed the run and verified.

The state file was created exclusively with mode 0600. No run token appeared in command output. Synthetic private state and signer keys were cleaned up.

Validation on the reviewed tree:

  • npm run build: passed.
  • make test-ts: 148 passed, 10 existing contract skips, 0 failed; unchanged from baseline.
  • tests/test_docs.py: 7 passed, 12 subtests passed.
  • make lint build: passed, including wheel/sdist and Twine checks.
  • Independent verification of the exported bundle: VERIFIED; one reconciled tool call, none unreconciled.

Full-suite limitation: before implementation, PYTHONPATH=$PWD make check with the local venv passed lint but encountered:

tests/test_adapters2.py::VercelAI::test_vercel_ai_sdk_spans
OSError: [Errno 30] Read-only file system: '/home/agent/.tracekit-client'

The existing mapping test reaches legacy client_dir(), which creates its default directory; the session fixture removes TRACEKIT_CLIENT_HOME. A focused reproduction reported 1 failed and 2 passed.

After confirming that failure, the full run was manually interrupted with SIGINT—not stopped by -x or maxfail:

1 failed, 357 passed, 1 skipped, 1198 subtests passed in 171.79s
exported v2 bundles verified: 131
make: *** [Makefile:27: test] Error 2

The remaining suite was not validated, and the build stage of that make check run was not reached. Standalone lint/build/Twine later passed. No HOME reassignment, test modification or denied-path retry was used. The observed Python skip was Agent Framework's MCP 1.x requirement against installed MCP 2.x; root/Linux coverage beyond the interrupted run remains unproven.

Tests ran on the aa41d057…-based reviewed tree, which exactly matches this branch's published tree. Upstream subsequently merged #508 and #504. Their immutable delta was inspected: contribution requirements and the example's SDK/approval/parser paths are unchanged, with nonoverlapping limits edits. The integrated new-base result has not been rerun.

Checklist

  • make check passes (lint, tests, build); make test-ts too if sdk/typescript/ changed — TypeScript passes; full make check remains incomplete as disclosed.
  • a regression test covers the change — the example embeds runnable assertions, and the separate-process hold/resume, pending/replay/rejection checks above were executed; no dedicated CI regression test was added.
  • user-visible changes are described above (release notes are written at release time; don't edit CHANGELOG.md)
  • docs updated if behaviour or a security claim changed (docs/threat-model-laptop.md, docs/threat-model-server.md) — example links added; security claims unchanged.
  • every commit is signed off (git commit -s, see CONTRIBUTING.md)

Security invariants

  • the agent still never holds a signing key or assigns sequence numbers
  • nothing an agent-controlled process supplies (isolation level, fail mode, gap/tamper records, approval identity) is newly trusted by the signer or verifier
  • gaps are still signed, never silent; a check that cannot run warns instead of passing
  • no verification code is shipped inside a bundle

Evidence format

  • this change does not touch the frozen formats: v1 (core.sig_message, ledger.make_record, the v1 schema, how the verifier treats existing bundles) or the published v2 format
  • or, if it does: the issue explicitly asks for it, existing bundles (including docs/sample/ and tests/fixtures/) still verify, and the change is called out above — not applicable.

@LoopGlitch26

Copy link
Copy Markdown
Member

Thanks, the example reads well. I approved the workflow runs, so CI now runs the full suite on Linux; that covers the make check gap you noted. Two small changes, then mark it ready for review once CI is green:

  1. Hold on a policy rule, not on a parse failure. echo "unterminated is held by TK-SHELL-PARSE because the command can't be read, so the example ends up teaching that malformed commands wait for a person. Use a call a rule asks about, e.g. terraform destroy (TK-A001, an ask rule in both the dev and server packs). Then change the rule_ids assertion and the "Held:" line to match. The pretend tool keeps it harmless.
  2. Delete the state file once the run is closed. It holds the run token. After run.close(), await unlink(file) and say so in the output, so the header's "delete it when finished" happens by itself.

The test_vercel_ai_sdk_spans failure you hit is a real bug and not yours: client_config() creates ~/.tracekit-client just to read a file. I'll fix it in a separate PR and credit you there.

@LoopGlitch26

Copy link
Copy Markdown
Member

The read-only-home failure you found is fixed in #516, credited to you. It also turned up the same bug in doctor, report, view and analyze, and the suite now fails any test that writes into the real home folder.

Address the two requested changes in Cygnux-Labs#514: use the harmless TK-A001 pretend call and delete private state only after successful run close.

Signed-off-by: Lubai <1174273035@qq.com>
@sunxiuguo

Copy link
Copy Markdown
Contributor Author

Thanks — both changes are in a41d3c1.

  • The mock call is now terraform destroy, with TK-A001 in the assertion and Held output. I checked ask in both dev/server packs with both regex engines; the pretend tool still only prints text and never invokes Terraform or a shell.
  • After a successful run.close(), the example unlinks the private state file and reports its deletion. Approval and rejection paths deleted it; a pending retry preserved it. An injected close failure exited nonzero, retained the 0600 state and made no deletion claim.

I reran the real separate-process hold/approval/resume flow: the same approval led to exactly one pretend execution, and the signed bundle verified. Build, lint and docs checks passed; TypeScript tests were 148 passed / 10 existing skips / 0 failed. Independent automated review found no unresolved findings (not human review).

The new head's workflows have now started and DCO has passed; I have left this draft pending the remaining checks, as requested. The earlier local full-suite limitation remains documented; I am not treating an older head's CI as validation of this revision. Thank you for addressing the read-only-home bug in #516.

@sunxiuguo
sunxiuguo marked this pull request as ready for review November 2, 2026 07:39
@LoopGlitch26
LoopGlitch26 merged commit 394b6fa into Cygnux-Labs:main Nov 2, 2026
34 checks passed
@LoopGlitch26

Copy link
Copy Markdown
Member

Merged, thanks. Two small follow-ups from review I'll make myself: the usage line will suggest a state file in your own directory rather than /tmp, and a resume after the run has ended will say so and delete the file instead of waiting on an approval that's gone.

LoopGlitch26 added a commit that referenced this pull request Nov 2, 2026
…t does (#516)

* client: reading the config no longer creates the client folder (#515)

client_config() took its path from client_dir(), which runs makedirs, so a
read (otlp.Mapper.plan asking for signer_isolation) created the folder under
the home directory and failed where the home folder is read-only. Found by
@sunxiuguo while validating #514.

* doctor, report, view, analyze: reading the dev store creates nothing (#515)

doctor, report, view --dev and analyze --v2 took the dev data dir from the
signer's dev_data_dir(), which creates it. The plugin's hook blocks the calls
it can't record once that folder exists, so running doctor before setup
turned the plugin from warning to blocking. They now read the path from
autospawn.data_dir(); the signer, tracekit up and init still make it.

- A missing data dir says a signer makes it when it first starts (view and
  report used to say "start the signer once").
- D-DATA-FS reads the file system of the dir's nearest existing parent.
- tests/conftest.py fails a test that makes the real home's client folder or
  dev data dir; it is silent where they already exist.

* tests: conftest loads without the v2 extra (#515)

The release dry run installs core only, so importing the dev signer's
autospawn (which needs rfc8785) broke conftest there. Without the v2 extra no
dev data dir can be made, so the guard watches only the client folder.

* Readers of the dev store make nothing, and say to run tracekit up; the guard covers Windows (#515)

Independent review of #516:
- tracekit last, export --v2, signer vkey/trust/reveal --dev (signer_config's
  dev branch) still made the dev data dir. They, report, view --dev and
  analyze --v2 now read it with dev_data_dir(create=False): a FileNotFoundError
  that names tracekit up while no dev signer has made it. last says there are
  no runs yet, as before.
- An explicit --data-dir that isn't there keeps the plain message.
- On Windows the dev data dir follows LOCALAPPDATA: the tests that isolate
  XDG_DATA_HOME now isolate LOCALAPPDATA too (NoHooksChosen made the real one
  on CI).
- The guard names the variable each folder follows, and a folder made outside
  any test (setUpClass) fails the session at its end.
- Regression tests for view --dev, analyze --v2 and last.

* tests: the session runs on a runtime dir of its own (#515)

#516's Windows CI: HoldLine wrote the hook's session state to the real runtime
dir, which on Windows is inside the dev data dir, and the guard caught it.
Eight test classes used the per-user runtime dir without setting one: on a
developer's machine, their own dev signer's, so a test could record into its
store. conftest now sets TRACEKIT_RUNTIME_DIR to a private temp dir for the
session (a shell's own value is replaced too), stops a dev signer left
running there at the end, and removes it. Tests that set their own keep it.

* tests: a dev signer left hung in the session's runtime dir fails the session, not the last test (#515)

Re-review of #516: a Hung from the session-end down() surfaced as an error at
teardown of whichever test ran last, and left the dir behind. It now fails the
session at its end, naming the dir; the dir is removed either way, and so are
its side dirs (autospawn._side) and any signer in them.
@LoopGlitch26

Copy link
Copy Markdown
Member

Thanks again for contributing to Tracekit. A request: if you use a coding agent (Claude Code, Codex, Cursor, Gemini CLI or OpenCode), would you run Tracekit on it for a week and tell us how it went?

pip install -U tracekit-ai
tracekit audit --since 7d      # read-only: what it would have held in your last week (Claude Code history)
tracekit init --dev --v2       # then record and guard from now on

After a week, tracekit report --share prints counts only (no commands, paths or content). Post it in #385 with anything that got in your way: a false alarm, a confusing message, something slow. If you find a problem, you're welcome to fix it too: say so on #385 and it's yours.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Example (TypeScript): approve a held Vercel AI SDK call and resume it in another process

2 participants