Skip to content

Findings: compare a boxed agent's own account with what Box saw cross its doors - #608

Merged
LoopGlitch26 merged 8 commits into
mainfrom
agent/said-vs-did
Nov 5, 2026
Merged

LoopGlitch26 merged 8 commits into
mainfrom
agent/said-vs-did

Conversation

@LoopGlitch26

@LoopGlitch26 LoopGlitch26 commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

A boxed agent's runs (#603) are its own account: its hooks run inside the box, so it can skip or forge them. The box's run holds what Box's daemon decided on the host. Until now nothing compared the two. This adds TK-X212 to tracekit analyze --v2.

What it does

Run it on a Tracekit Box box run, from a run-set bundle or the signer's store (both hold the box's agent channel runs):

tracekit analyze --v2 --run BOX --bundle set.tkb --trust trust.json
tracekit analyze --v2 --run BOX --store DATA_DIR

Record shapes are read from Box's code (internal/daemon/decider.go, apply.go, exec.go, events.go, internal/tracekit/tracekit.go, internal/egress/egress.go):

  • each connection out of the box is a WebFetch decision of the box run, {url: "<scheme>://<host>:<port>"}, decided before the dial. It is refused when the decision is deny or its result isn't ok.
  • each file written back at tkbox apply is a Write decision, {file_path: <apply dir>/<path>, patch_sha256, ...}.
  • each command Box started from the host is a Bash decision.
  • tkbox.box records the workspace, tkbox.model.call the model calls, tkbox.egress Box's own refusals (queued, with no time of their own), and tkbox.gap lost records.

The box run and its agent runs share the signer's clock. A call's window runs from its decision to --effect-grace (2 s) after its result, with no skew.

How a fetch is matched to a connection. By host and port:

  • the host loses a leading www., because Claude Code's WebFetch follows a redirect that only adds or drops it, on the same protocol and port;
  • the port is the URL's own, plus 443 for plain http to port 80, because WebFetch upgrades it to https;
  • the URL is read the way Node reads a web URL: a backslash counts as a slash, and any slashes after https: are read as //;
  • the host is taken as Node sends it: an internationalised name as punycode (stdlib idna), and a WHATWG number form such as 0x5db8d822 as a dotted address. A name ending in a number that isn't an address is no URL, so www.<address> never matches.

A fetch of a URL Box never sees explains nothing. A connection that only a fetch with unknown arguments could explain is no_content.

A box run's own calls are Box's doors, so the checks of an agent's calls (TK-X201 to TK-X208, TK-X211 and v1's text checks) skip them.

The rules

Each rule is conservative. What can't be told is counted not checkable, with a reason and a fix, and is never a finding.

  1. A connection no call of the agent's explains (high, door_unexplained). It is explained when it was decided in the window of a call whose tool may connect: a shell, an MCP or unknown tool, or a fetch it matches. Can't tell:
    • a model API host (model_host);
    • no agent run was open (no_agent_run);
    • only a call with no result could explain it (call_unfinished);
    • a command Box started was running (host_command). The one command an agent run began in doesn't count; when two commands span that start, both count;
    • after a shell call that may have left a process running (background), as the policy engine's shell parser (tracekit/policy2/shell.py) reads its command, with bash's quoting, heredocs, comments, wrappers and sh -c strings. That means run_in_background, a command it sends to the background (cmd &, (cmd &), bash -c 'cmd &'), or one that detaches by itself (setsid, disown, coproc, tmux … -d, screen -d…). nohup cmd alone runs in the foreground. A line the parser can't read counts too;
    • a refusal Box made itself, or lost (untimed_refusal), which has no time of its own;
    • content missing (no_content).
  2. A file of the box's patch no call explains (high, door_unexplained). It is /workspace/<path> in the box, and no write call of the agent's that didn't fail named it. Can't tell:
    • applied to another directory than the box's workspace, or perhaps to one under it (apply_dir);
    • a write call's arguments aren't known (no_content), or it names no path (no_path);
    • a shell, MCP or other call that writes files without naming them ran (shell_ran);
    • a host command as above ran (host_command).
  3. A fetch with no connection (medium, no_door_event). The agent's fetch returned ok, and no connection it matches crossed the door from the box's start to grace after its result. Can't tell:
    • an earlier connection it matches that Box didn't refuse (reused: keep-alive, or the tool's cache);
    • Box refused that host itself after the fetch began, or lost such a record (box_refusal);
    • the URL or a door host isn't known (no_url).
  4. A fetch the door refused (high, door_refused). The agent's fetch returned ok, the door refused every connection it matches in its window, and no earlier connection could have served it (else reused). Can't tell: one has no recorded outcome (no_outcome).

When there's nothing to compare against:

  • A box run with no agent runs beside it counts its door events no_agent_runs.
  • They count as no_agent_account instead (the agent registered no run at all, and the summary line says NO AGENT ACCOUNT) only when all of these hold:
    • the box run is final;
    • its channel was on;
    • the source holds every run of the tenant: the store, or a run-set from registry 0 with no run still open, whose range holds the box run's own registration (a run-set bundles only final runs, and may carry its selected run from outside its range);
    • Box refused no register_run on the channel (tkbox.channel; a count of refusals Box folded may hold one).
  • An agent run analysed on its own counts its calls agent_run and says to analyse the box run.

Where it is signed. Findings are signed about the box run. review_finding takes cites of the analysed run only, so a finding cites the box run's door records. For a fetch with no connection, it cites the box run's first record after the window: no record before it is a connection to that host. The agent's run id, call id and records are named in what prints.

Rule id. I used TK-X212, the next id after the highest one (TK-X211). TK-X209 is unused, but nothing says why it was skipped, so I left it alone.

Contract. TK-X212 joins review_finding's closed finding table under RPC 21, which is unreleased (#603). A 21 client already refuses a 20 signer, so no older signer gets the new rule.

Review fixes (54706a5, f3a668a)

Each of the review's probes is now a regression test in tests/test_findings_box.py:

Probe Was Now
P1: www. redirect high finding matched (Matching)
P2: (cmd &) high finding background
P3: cached fetch beside a refused parallel one false door_refused reused
P4: two host commands span the agent's start high finding host_command
P5: a host command's daemon finding still a finding; in limits
P6: hex IP two findings matched
P7: apply --dir into a subdirectory finding apply_dir
H1: never registers no_agent_runs no_agent_account + NO AGENT ACCOUNT on the summary line
H2: closes its run early — no_agent_run (in limits)
H3: one forged background flag — background for the rest of the run (in limits)
H4: unusable URL wildcard — explains nothing
H5: :22 tunnel during a fetch — found
H6: unfinished call — call_unfinished (in limits)

Also:

  • Box's own refusals count as untimed_refusal on the door side.
  • A box run skips the agent checks, so the TK-X202 lines about apply writes are gone. A run with no calls no longer prints "TK-X206 for 0 calls".
  • A write call with no path is no_path.
  • A connection with no outcome isn't counted "refused" in (a)'s text.
  • docs/limits.md's redirect row is corrected. New rows cover: what background forms stay out, and a host command's daemon; untimed_refusal, and "let through" ≠ "connected"; no_agent_account; apply --dir.

Review fix on ba6e657 (4be104c)

  • Low, fixed: a nested list() overwrote the and-or state, so a subshell or case arms in the same pipeline as a block lost what the block had recorded. list() now saves the parser's list_start and closed on entry and restores them on return.
    • Now { a; } | (x) & marks a and x, a && (x) | { b; } & marks a, x and b, and { npm run dev; } | (x=1) & marks npm. The case-arm variants are fixed the same way.
  • Tests: 7 new subtests in test_policy2 and test_findings_box, all failing on ba6e657's parser and passing now. The reviewer's test_bg_final.py, pointed at this branch: 3 passed, 68 subtests.
  • Single-file runs: tests/test_policy2.py 141 passed (3,719 subtests); tests/test_findings_box.py 56 passed (111 subtests).
  • Verdict diff, origin/main (fecb125) against 4be104c: 534,690 decisions per side, 0 policy_hash, verdict or rule-id, decide() field, nondeterministic or constraints.shell() differences, and 0 private-corpus differences. shell.analyse() with background set aside and sets in a fixed order: 0 differences across all 28,094 shell cases.

Review fixes on 66ad1fe (ba6e657)

Each fix has a test that fails on 66ad1fe's code and passes now: 19 new subtests across test_findings_box and test_policy2. Each part, undone on its own, makes a test fail (3 of 3).

  • Medium: a block or an and-or list sent to the background is now marked.
    • Before, the parser read {, }, fi and done as words inside one simple command. So the & after a closing word marked only that last pipeline, which was empty.
    • Now list() keeps the and-or list's start across && and ||. Each block records the and-or list it opened in, and _Parser.close_block hands that back to list(). An & after the closing word marks every command from there.
    • These no longer give a false high door_unexplained: { npm run dev; } &, a while true; do …; sleep 30; done & health check, if true; then npm run dev; fi &, a && b &, and a fetch followed by && x=1 &. The exact commands are in the tests.
    • The docstring at background is corrected.
  • Later runs. DETACHERS reuses constraints.LATER_RUNS (systemd-run, at, batch).
  • docs/limits.md names what still stays out: daemonize, start-stop-daemon --background, sudo -b, ssh -f, an asynchronous >(…), and interpreters that spawn detached children. Each errs toward a false finding, never silence.
  • New limits row: on a box run, the operator's task isn't checked against the patch. A follow-up issue is drafted for a box-aware check of tkbox.task against Box's Write decisions.
  • model_receipt.py's comment no longer says the parser keeps no mark of a background command.

Verdict diff, origin/main (fecb125; #607 touches no policy, parser or constraints code) against ba6e657, using the review's run_cases.py/run_all.py: 29,705 cases × 9 packs × 2 engines, 534,690 decisions per side.

  • 0 policy_hash, verdict or rule-id, other decide() field, nondeterministic or constraints.shell() differences.
  • 0 private-corpus differences.
  • shell.analyse() with background set aside: the script counted 25 differences, all in learned. Its unresolved set hashes in each process's own string order: origin/main against itself gives 34. With sets put in a fixed order there are 0 differences across all 28,094 shell cases.
  • Commands now marked background: e21 4, tests-str 89, tests-dict 5, tests-decide 1, private-corpus 2.

Final check fixes (66ad1fe)

Each probe of the final check is a regression test. Each fix, undone on its own, makes a test fail (11 of 11; whole()'s from-0 condition is pinned by a unit test).

  • F4: the hand lexer is gone. _detaches reads commands with the policy engine's own shell parser, tracekit/policy2/shell.py. I added one thing to the parser: it marks each command of a pipeline that ends in & as background, nested ones included. That has its own test in tests/test_policy2.py; the policy engine itself reads no new key.
    • Fixed honest cases: tmux new -d or bash -c './start.sh &' after a heredoc, and a command after a # comment. These were false HIGH findings; now they're background.
    • Quotes that start mid-word no longer read as background: git commit -qm'fix a & b', FOO='a & b' make, echo $'a & b', URL="…&b=2" npm test. The commit case's later connection is found again.
    • Also handled: line continuations before &, echo a\\&, and \coproc, time coproc and builtin coproc.
    • Hostile text still isn't background: echo "&", echo coproc.
    • Changed on purpose: nohup cmd alone now counts as foreground, as bash runs it (the final check's own expectation).
  • F1, folded refusals. A tkbox.channel record with a folded count may hold a refused register_run. Such a box reads no_agent_runs.
  • F1, a run-set without the box run. A run-set from registry 0 with nothing open, but whose range doesn't hold the box run's registration, reads no_agent_runs. The verifier now keeps what its passing run-set check found (rep.run_set: from, registered, open), and findings_box.whole(rep, box_run) reads that instead of the check's wording.

Second review fixes (27c3ed4)

Each of these is a regression test, and each fix, undone on its own, makes a test fail (11 of 11).

  • F1, a false NO AGENT ACCOUNT. It was claimed from any passing run-set and from a live store. It's now claimed only under the conditions above, via findings_box.whole(rep.checks). CLI tests cover:
    • a run-set holding every run: claims it;
    • a box stopped mid-session, with the agent's run left open and so out of the run-set: no_agent_runs;
    • a live box read from the store: no_agent_runs;
    • Box refusing the agent's register_run: no_agent_runs.
  • F2. http://host:80/… matches 443 too.
  • F3. A shell's options before -c are skipped (bash -o pipefail -c, sh -e -c, bash --norc -c). So are wrappers' options (sudo -u dev, env -i, timeout 10).
  • F4. Commands are lexed with quoting kept. & counts only outside quotes and when not escaped. nohup, setsid, disown, coproc, tmux, screen and script shells count only in command position. So echo "&", grep -c '&' f, echo coproc and a heredoc line aren't background. Outside a heredoc, a newline ends a command.
  • Nits.
    • www. is never stripped from an address: a name ending in a number is an address or no host.
    • A backslash in a URL is read as a slash, as in agent_hooks.
    • Box's door URLs (tcp://…) are parsed separately from fetch URLs.
    • A connection only a fetch of unknown arguments could explain is no_content, not checked.

Tests

tests/test_findings_box.py, 56 tests. Each builds a real run-set bundle through factories.BoxScene: an in-process signer that keeps content, on a synthetic clock, writing Box's records and the agent's hook calls. Each test reads the bundle back through the verifier and the vault's content. They cover:

  • every finding kind, with the records named on both sides and no plaintext in the signed part;
  • every can't-tell reason;
  • the review's probes;
  • a clean box with no findings;
  • signing: the findings run, exported with the box run, verifies;
  • store_runs;
  • the CLI path (findings.main) from both a bundle and a store.

Every exported bundle verifies (the conftest check). Mutation checks: undone one at a time, each of these makes a test fail: the first 17 rules, 17 first-review fixes, 11 second-review fixes, 11 final-check fixes, and this round's 3.

On ba6e657, single files only (the machine's load was about 50; CI runs the full suite):

  • tests/test_findings_box.py and tests/test_policy2.py::Shell: 106 passed, 558 subtests.
  • tests/test_policy2.py: 141 passed, 3,714 subtests.
  • tests/test_model_receipt.py: 42 passed.
  • tests/test_constraints.py: 39 passed.
  • tests/test_docs.py: 7 passed.

On 66ad1fe:

  • make lint: pass.
  • Suites pass: test_findings_box, test_findings_v2, test_findings_effects, test_findings_text, test_review_model, test_docs, test_policy2, test_box_channel and test_bundle_v2 (448 passed).
  • The full gate wasn't run locally at a load average of 42/38/35, above the limit of 8. A later attempt, started when the 1-minute load dropped to 7.79, was killed by a session restart at 28% and gave no result. CI's py3.9 on ubuntu-latest job ran the full suite on 66ad1fe and passed.

Gates on 27c3ed4, run in the worktree with PYTHONPATH, and TRACEKIT_PYTHON set to the venv (the TS tests spawn it). The run started at a 1-minute load of 6.75:

  • make lint: pass.
  • make build: pass.
  • make test-ts: 150 passed, 0 failed, 10 skipped.
  • make test: 4467 passed, 394 skipped, 10 xfailed, 1 failed: test_why_core.py::LongRuns::test_dense_contexts_build_their_graph_in_time. That's a timing test, and the load reached 41 by the end of the 21m48s run. This branch doesn't touch tracekit/why, and the test passes when re-run alone.

The earlier full gates, on f3a668a:

  • make lint: pass.

  • make build: pass.

  • make test-ts: 150 passed, 0 failed, 10 skipped. The TS parity test now passes on the owner's 231c466.

  • make test (f3a668a): 4458 passed, 394 skipped, 10 xfailed, 11 failed. All 11 are timing tests under load. The run took 40m38s, with five other worktrees' suites running alongside, at a load average of 36 to 51. The tests:

    • test_parsers.py::HostileStreams::test_streamed_arguments_accumulate_in_linear_time;
    • test_policy2.py::Speed::test_an_indented_select_in_a_sqlite3_argument_decides_fast_on_the_fallback_engine;
    • test_policy2.py::Shell::test_arithmetic_context_prescan_is_linear;
    • test_provenance.py::Values::test_adversarial_text_is_read_in_linear_time (8 subtests).

    This branch doesn't touch those modules. Re-run alone on the branch, all four pass, twice. On origin/main's code under the same load they pass too. The earlier full run of this branch (a33d491) had 4407 passed and none failed.

Run like a user

I couldn't start a signer from the shell myself (see below), so this output comes through the CLI entry point instead: findings.main(["--v2", "--dry-run", "--run", BOX, "--bundle", set.tkb, "--trust", trust.json]), which is what tracekit analyze --v2 runs. The input is the run-set bundle test_findings_box.Cli builds (verified first), with vault content from that test's signer.

The scene:

  • the agent's command started by Box, and the harness's model call, both before the hooks registered;
  • a fetch with its own connection, and an edit;
  • a connection while no call was in progress;
  • a fetch Box refused that the agent reported ok;
  • a fetch with no connection;
  • apply writes src/app.py (edited by the agent) and .github/workflows/release.yml (never named).

The --store source gives the same output.

$ tracekit analyze --v2 --run c265f7d101b8410e65574199f4ab7ea4 --bundle set.tkb --trust trust.json --dry-run
Findings for run c265f7d101b8410e65574199f4ab7ea4: 4 problems (3 high, 1 medium) · reasoning readable before 0 of 0 actions (n/a)

HIGH · Agent's account and the box's doors disagree  (TK-X212) [door_unexplained]
  What happened: Box's door saw the box connect to collector.example.io 1 time (1 let through, 0 refused; the first at box run record 7) while the agent's run ea795d320039eed94dbb21031670c5e5 was open, and no call of the agent's that could connect was in progress, nor had a shell call left a process running; the agent's last call before it: Edit (run ea795d320039eed94dbb21031670c5e5, record 3), 5.0 s earlier [window: a call's decision to 2 s after its result, on the signer's clock]
  Why it matters: the agent's own records of its calls (sent from inside the box, where it can skip or forge them) and the box's door records (decided on the host, outside its reach) disagree: the agent did something it didn't record, or recorded something the door didn't see
  What to do: trust the door records over the agent's; read the agent's calls around the cited door record (`tracekit view`), and review what the box wrote before you apply or run it
  Records: 7 (c89e9830cca0), 8 (8b10d9a737ee) · relied on: observer, decision

HIGH · Agent's account and the box's doors disagree  (TK-X212) [door_refused]
  What happened: the agent's WebFetch of pastebin.example.net (run ea795d320039eed94dbb21031670c5e5, call toolu_14, records 5 and 6) returned ok, but Box's door refused every connection to pastebin.example.net in that call's window (box run record 9), and none before it could have served it [window: a call's decision to 2 s after its result, on the signer's clock]
  Why it matters: the agent's own records of its calls (sent from inside the box, where it can skip or forge them) and the box's door records (decided on the host, outside its reach) disagree: the agent did something it didn't record, or recorded something the door didn't see
  What to do: trust the door records over the agent's; read the agent's calls around the cited door record (`tracekit view`), and review what the box wrote before you apply or run it
  Records: 9 (3afdd0ad4a60), 10 (ed63abfdaaea) · relied on: observer, decision

HIGH · Agent's account and the box's doors disagree  (TK-X212) [door_unexplained]
  What happened: the box's patch changes .github/workflows/release.yml (written back at apply: Box's Write decision, box run record 14), and no write call of the agent's run ea795d320039eed94dbb21031670c5e5 named /workspace/.github/workflows/release.yml; no shell or other call that writes files without naming them ran, nor a command Box started besides the agent's
  Why it matters: the agent's own records of its calls (sent from inside the box, where it can skip or forge them) and the box's door records (decided on the host, outside its reach) disagree: the agent did something it didn't record, or recorded something the door didn't see
  What to do: trust the door records over the agent's; read the agent's calls around the cited door record (`tracekit view`), and review what the box wrote before you apply or run it
  Records: 14 (c5c7f99299fb), 15 (7c6965755ed2) · relied on: observer, decision

MEDIUM · Agent's account and the box's doors disagree  (TK-X212) [no_door_event]
  What happened: the agent's WebFetch of example.org (run ea795d320039eed94dbb21031670c5e5, call toolu_18, records 7 and 8) returned ok, but no connection to example.org crossed the box's door from the box's start to 2 s after that result (the box run's records up to 11) [window: a call's decision to 2 s after its result, on the signer's clock]
  Why it matters: the agent's own records of its calls (sent from inside the box, where it can skip or forge them) and the box's door records (decided on the host, outside its reach) disagree: the agent did something it didn't record, or recorded something the door didn't see
  What to do: trust the door records over the agent's; read the agent's calls around the cited door record (`tracekit view`), and review what the box wrote before you apply or run it
  Records: 11 (8f86e08115d8) · relied on: observer, decision

Not checkable (never counted as a pass):
  TK-X212 for 1 door event: a connection to a model API host: the agent's harness makes its own model calls, which no tool call names.
    To make it checkable: nothing; Box records each model call on that host (tkbox.model.call)

Monitorability (the share of actions whose reasoning can be read): n/a of 0 actions; full 0, summary 0, withheld 0, none 0, not known 0. Withheld: the provider returned the reasoning sealed or not at all. Reasoning is the model's own account: self-reported, not proof.
Checkable share: TK-X201 n/a (0 checked, 0 found), TK-X202 n/a (0 checked, 0 found), TK-X203 n/a (0 checked, 0 found), TK-X204 n/a (0 checked, 0 found), TK-X208 n/a (0 checked, 0 found), TK-X205 n/a (0 checked, 0 found), TK-X101 n/a (0 checked, 0 found), TK-X102 n/a (0 checked, 0 found), TK-X103 n/a (0 checked, 0 found), TK-X104 n/a (0 checked, 0 found), TK-X111 n/a (0 checked, 0 found), TK-X112 n/a (0 checked, 0 found), TK-X113 n/a (0 checked, 0 found), TK-X114 n/a (0 checked, 0 found), TK-X120 n/a (0 checked, 0 found), TK-X211 n/a (0 checked, 0 found), TK-X206 n/a (0 checked, 0 found), TK-X207 n/a (0 checked, 0 found), TK-X212 89% (8 checked, 4 found)
Not signed: printed only.

u==== --store
Findings for run c265f7d101b8410e65574199f4ab7ea4: 4 problems (3 high, 1 medium) · reasoning readable before 0 of 0 actions (n/a)

HIGH · Agent's account and the box's doors disagree  (TK-X212) [door_unexplained]
  What happened: Box's door saw the box connect to collector.example.io 1 time (1 let through, 0 refused; the first at box run record 7) while the agent's run ea795d320039eed94dbb21031670c5e5 was open, and no call of the agent's that could connect was in progress, nor had a shell call left a process running; the agent's last call before it: Edit (run ea795d320039eed94dbb21031670c5e5, record 3), 5.0 s earlier [window: a call's decision to 2 s after its result, on the signer's clock]
  Why it matters: the agent's own records of its calls (sent from inside the box, where it can skip or forge them) and the box's door records (decided on the host, outside its reach) disagree: the agent did something it didn't record, or recorded something the door didn't see
  What to do: trust the door records over the agent's; read the agent's calls around the cited door record (`tracekit view`), and review what the box wrote before you apply or run it
  Records: 7 (c89e9830cca0), 8 (8b10d9a737ee) · relied on: observer, decision

HIGH · Agent's account and the box's doors disagree  (TK-X212) [door_refused]
  What happened: the agent's WebFetch of pastebin.example.net (run ea795d320039eed94dbb21031670c5e5, call toolu_14, records 5 and 6) returned ok, but Box's door refused every connection to pastebin.example.net in that call's window (box run record 9), and none before it could have served it [window: a call's decision to 2 s after its result, on the signer's clock]
  Why it matters: the agent's own records of its calls (sent from inside the box, where it can skip or forge them) and the box's door records (decided on the host, outside its reach) disagree: the agent did something it didn't record, or recorded something the door didn't see
  What to do: trust the door records over the agent's; read the agent's calls around the cited door record (`tracekit view`), and review what the box wrote before you apply or run it
  Records: 9 (3afdd0ad4a60), 10 (ed63abfdaaea) · relied on: observer, decision

HIGH · Agent's account and the box's doors disagree  (TK-X212) [door_unexplained]
  What happened: the box's patch changes .github/workflows/release.yml (written back at apply: Box's Write decision, box run record 14), and no write call of the agent's run ea795d320039eed94dbb21031670c5e5 named /workspace/.github/workflows/release.yml; no shell or other call that writes files without naming them ran, nor a command Box started besides the agent's
  Why it matters: the agent's own records of its calls (sent from inside the box, where it can skip or forge them) and the box's door records (decided on the host, outside its reach) disagree: the agent did something it didn't record, or recorded something the door didn't see
  What to do: trust the door records over the agent's; read the agent's calls around the cited door record (`tracekit view`), and review what the box wrote before you apply or run it
  Records: 14 (c5c7f99299fb), 15 (7c6965755ed2) · relied on: observer, decision

MEDIUM · Agent's account and the box's doors disagree  (TK-X212) [no_door_event]
  What happened: the agent's WebFetch of example.org (run ea795d320039eed94dbb21031670c5e5, call toolu_18, records 7 and 8) returned ok, but no connection to example.org crossed the box's door from the box's start to 2 s after that result (the box run's records up to 11) [window: a call's decision to 2 s after its result, on the signer's clock]
  Why it matters: the agent's own records of its calls (sent from inside the box, where it can skip or forge them) and the box's door records (decided on the host, outside its reach) disagree: the agent did something it didn't record, or recorded something the door didn't see
  What to do: trust the door records over the agent's; read the agent's calls around the cited door record (`tracekit view`), and review what the box wrote before you apply or run it
  Records: 11 (8f86e08115d8) · relied on: observer, decision

Not checkable (never counted as a pass):
  TK-X212 for 1 door event: a connection to a model API host: the agent's harness makes its own model calls, which no tool call names.
    To make it checkable: nothing; Box records each model call on that host (tkbox.model.call)

Monitorability (the share of actions whose reasoning can be read): n/a of 0 actions; full 0, summary 0, withheld 0, none 0, not known 0. Withheld: the provider returned the reasoning sealed or not at all. Reasoning is the model's own account: self-reported, not proof.
Checkable share: TK-X201 n/a (0 checked, 0 found), TK-X202 n/a (0 checked, 0 found), TK-X203 n/a (0 checked, 0 found), TK-X204 n/a (0 checked, 0 found), TK-X208 n/a (0 checked, 0 found), TK-X205 n/a (0 checked, 0 found), TK-X101 n/a (0 checked, 0 found), TK-X102 n/a (0 checked, 0 found), TK-X103 n/a (0 checked, 0 found), TK-X104 n/a (0 checked, 0 found), TK-X111 n/a (0 checked, 0 found), TK-X112 n/a (0 checked, 0 found), TK-X113 n/a (0 checked, 0 found), TK-X114 n/a (0 checked, 0 found), TK-X120 n/a (0 checked, 0 found), TK-X211 n/a (0 checked, 0 found), TK-X206 n/a (0 checked, 0 found), TK-X207 n/a (0 checked, 0 found), TK-X212 89% (8 checked, 4 found)
Not signed: printed only.

u.
exported v2 bundles verified: 1 in 0.0s
1 passed, 43 deselected, 2 subtests passed in 0.45s

That output is verbatim, from f3a668a. The box run's own calls no longer go through the agent checks: those show n/a, and there is no "TK-X206 for 0 calls".

The Tracekit hook

The hook stopped these steps, and I didn't route around them:

  1. Regenerating the TypeScript copy of the review_finding schema (sdk/typescript/src/v2/rpc_schema.ts). The owner did it in 231c466. Since then the TS parity test passes; see the gates.
  2. The shell run with a real tracekit signer serve --config with box_daemons (denied: TK-D007, TK-F008, TK-F011). The demo is ready for the owner to run: a stand-in Box daemon and agent hook over the signer's socket, then tracekit export --v2 --run-set, tracekit signer trust and tracekit analyze --v2. The output above comes from the CLI entry point in the test suite instead.

Not in this PR

  • A shell's curl or wget isn't checked for a missing connection: a shell's exit status claims nothing about it. A write call isn't checked against the patch, because the patch is the net change.
  • A program that daemonizes itself, docker run -d, systemd-run and at aren't read as background. A host command's daemon is the same. Their later connections are findings. All are in docs/limits.md.

… its doors

TK-X212 (tracekit/findings_box.py), in `tracekit analyze --v2` on a Tracekit
Box box run, against the runs of its agent channel (#603): the box run's door
records (Box's WebFetch decision per connection, Write per file written back
at apply, Bash per command it started; tkbox.* observes) and the agent's
hook-recorded calls, on one signer clock.

- a connection, or a file of the box's patch, no call of the agent's explains
  (high, door_unexplained);
- a fetch the agent said returned ok with no connection to its host through
  the door (medium, no_door_event);
- a fetch the agent said returned ok whose every connection the door refused
  (high, door_refused).

What it can't tell is counted not checkable with its reason (model_host,
no_agent_run, host_command, background, call_unfinished, apply_dir,
shell_ran, reused, box_refusal, no_outcome, no_url, no_content, budget;
no_agent_runs and agent_run when the other side isn't there). Findings are
signed about the box run and cite its records; the agent's records are named
in what prints. The box's agent runs come from a run-set bundle or the store.

Rule TK-X212 joins review_finding's closed table under the unreleased RPC 21.
Tests build real run-set bundles through factories.BoxScene; docs add the
rule, its reasons, a section and two docs/limits.md rows.
- A fetch matches a connection by host, a leading www. aside (Claude Code's
  WebFetch follows that redirect), and port: the URL's, and 443 too for plain
  http. Hosts are taken as Node sends them: IDN in punycode (stdlib idna),
  WHATWG IPv4 number forms dotted. A fetch of a URL Box never sees explains
  nothing; only unknown arguments still match any connection.
- Background: a lone `&` in any punctuation token (`(cmd &)`), coproc,
  `tmux ... -d`, `screen -d...`, and a script `sh -c` runs.
- (c) checks `reused` first: a cached or kept-alive fetch beside a parallel
  refused connection is no door_refused.
- A host command is the agent's only when it alone spans an agent run's start.
- no_agent_account: the box's channel was on, the source holds every run of the
  tenant (a store or a run-set), and the agent registered none; the summary line
  says NO AGENT ACCOUNT.
- Box's own refusals (tkbox.egress, lost ones in tkbox.gap) count as
  untimed_refusal on the door side.
- A box run's calls are Box's doors: the checks of an agent's calls skip them.
- A patch path a written path ends is apply_dir (apply --dir to a directory
  under the workspace); a write call with no path is no_path; a connection with
  no outcome isn't counted refused.

Each probe of the review is a regression test; docs and limits rows updated.
- no_agent_account only for a final box run, from the store or a run-set from
  registry 0 with no run open (findings_box.whole: a run-set bundles only final
  runs), and not when Box refused a register_run on the channel (tkbox.channel,
  or such records lost). Otherwise no_agent_runs.
- http to port 80 matches 443 too (WebFetch upgrades it).
- A shell's options before -c are skipped (bash -o pipefail -c, sh -e -c,
  bash --norc -c), as are wrappers' (sudo -u dev, env -i, timeout 10).
- Commands are read with quoting kept: `&` counts only outside quotes and not
  escaped; nohup, setsid, disown, coproc, tmux, screen and script shells only as
  a command, never as text (echo coproc, a heredoc line). Outside a heredoc a
  newline ends a command.
- Fetch URLs are read as Node reads web URLs (a backslash a slash, any slashes
  after https:); a name ending in a number is an address or no host, so
  www.<address> is no URL. Box's door URLs are parsed apart.
- A connection only a fetch of unknown arguments could explain is no_content.

Each probe is a regression test; docs and limits updated.
…agent_account gaps

- _detaches drops the hand lexer for policy2.shell, which reads bash's quoting,
  heredocs, comments, $'...', continuations, sh -c strings and wrappers. The
  parser now marks each command of a pipeline that ends in `&` (nested ones too)
  `background`. A call may leave a process running when a command is sent to the
  background, or detaches by itself (setsid, disown, coproc, tmux -d, screen
  -d), or the parser can't read the line; nohup alone runs in the foreground.
- no_agent_account: a tkbox.channel record with a `folded` count may hold a
  refused register_run, so it reads no_agent_runs.
- no_agent_account: a run-set must hold the box run's own registration (a
  run-set may carry its selected run from outside its range). The verifier
  keeps what a passing run-set check found (rep.run_set: from, registered,
  open); findings_box.whole reads it instead of the check's wording.

The final check's probes are tests; the parser's background marking has its own.
…er-run commands

- policy2.shell: `background` now marks the whole and-or list an `&` ends
  (`a && b &`, `curl ... && x=1 &`), and a block whose closing word that `&`
  follows from where the block opened (`{ ...; } &`, `while ...; done &`, if,
  for, select, case). Blocks record the and-or list they opened in;
  _Parser.close_block hands it back to list(). Docstring corrected.
- findings_box: DETACHERS adds constraints.LATER_RUNS (systemd-run, at,
  batch), reused, not copied.
- docs/limits.md names what stays out (daemonize, start-stop-daemon
  --background, sudo -b, ssh -f, an asynchronous >(...), interpreters spawning
  detached children: each errs toward a false finding), and that a box run's
  patch isn't checked against the operator's task.
- model_receipt: the comment no longer says the parser keeps no mark.
…ound state

list() saves the parser's list_start and closed on entry and restores them on
return, so a subshell or case arms in the same pipeline as a block no longer
overwrite them: `{ a; } | (x) &` marks a and x, `a && (x) | { b; } &` marks a,
`{ npm run dev; } | (x=1) &` marks npm.
@LoopGlitch26
LoopGlitch26 merged commit 5a0aa45 into main Nov 5, 2026
34 checks passed
@LoopGlitch26
LoopGlitch26 deleted the agent/said-vs-did branch November 5, 2026 13:54
LoopGlitch26 added a commit that referenced this pull request Nov 5, 2026
…-run, E21's recorded latency rows kept

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant