Repository navigation
Findings: compare a boxed agent's own account with what Box saw cross its doors - #608
Merged
Merged
Conversation
… its doors TK-X212 (tracekit/findings_box.py), in `tracekit analyze --v2` on a Tracekit Box box run, against the runs of its agent channel (#603): the box run's door records (Box's WebFetch decision per connection, Write per file written back at apply, Bash per command it started; tkbox.* observes) and the agent's hook-recorded calls, on one signer clock. - a connection, or a file of the box's patch, no call of the agent's explains (high, door_unexplained); - a fetch the agent said returned ok with no connection to its host through the door (medium, no_door_event); - a fetch the agent said returned ok whose every connection the door refused (high, door_refused). What it can't tell is counted not checkable with its reason (model_host, no_agent_run, host_command, background, call_unfinished, apply_dir, shell_ran, reused, box_refusal, no_outcome, no_url, no_content, budget; no_agent_runs and agent_run when the other side isn't there). Findings are signed about the box run and cite its records; the agent's records are named in what prints. The box's agent runs come from a run-set bundle or the store. Rule TK-X212 joins review_finding's closed table under the unreleased RPC 21. Tests build real run-set bundles through factories.BoxScene; docs add the rule, its reasons, a section and two docs/limits.md rows.
- A fetch matches a connection by host, a leading www. aside (Claude Code's WebFetch follows that redirect), and port: the URL's, and 443 too for plain http. Hosts are taken as Node sends them: IDN in punycode (stdlib idna), WHATWG IPv4 number forms dotted. A fetch of a URL Box never sees explains nothing; only unknown arguments still match any connection. - Background: a lone `&` in any punctuation token (`(cmd &)`), coproc, `tmux ... -d`, `screen -d...`, and a script `sh -c` runs. - (c) checks `reused` first: a cached or kept-alive fetch beside a parallel refused connection is no door_refused. - A host command is the agent's only when it alone spans an agent run's start. - no_agent_account: the box's channel was on, the source holds every run of the tenant (a store or a run-set), and the agent registered none; the summary line says NO AGENT ACCOUNT. - Box's own refusals (tkbox.egress, lost ones in tkbox.gap) count as untimed_refusal on the door side. - A box run's calls are Box's doors: the checks of an agent's calls skip them. - A patch path a written path ends is apply_dir (apply --dir to a directory under the workspace); a write call with no path is no_path; a connection with no outcome isn't counted refused. Each probe of the review is a regression test; docs and limits rows updated.
…ts no count of nothing
- no_agent_account only for a final box run, from the store or a run-set from registry 0 with no run open (findings_box.whole: a run-set bundles only final runs), and not when Box refused a register_run on the channel (tkbox.channel, or such records lost). Otherwise no_agent_runs. - http to port 80 matches 443 too (WebFetch upgrades it). - A shell's options before -c are skipped (bash -o pipefail -c, sh -e -c, bash --norc -c), as are wrappers' (sudo -u dev, env -i, timeout 10). - Commands are read with quoting kept: `&` counts only outside quotes and not escaped; nohup, setsid, disown, coproc, tmux, screen and script shells only as a command, never as text (echo coproc, a heredoc line). Outside a heredoc a newline ends a command. - Fetch URLs are read as Node reads web URLs (a backslash a slash, any slashes after https:); a name ending in a number is an address or no host, so www.<address> is no URL. Box's door URLs are parsed apart. - A connection only a fetch of unknown arguments could explain is no_content. Each probe is a regression test; docs and limits updated.
…agent_account gaps - _detaches drops the hand lexer for policy2.shell, which reads bash's quoting, heredocs, comments, $'...', continuations, sh -c strings and wrappers. The parser now marks each command of a pipeline that ends in `&` (nested ones too) `background`. A call may leave a process running when a command is sent to the background, or detaches by itself (setsid, disown, coproc, tmux -d, screen -d), or the parser can't read the line; nohup alone runs in the foreground. - no_agent_account: a tkbox.channel record with a `folded` count may hold a refused register_run, so it reads no_agent_runs. - no_agent_account: a run-set must hold the box run's own registration (a run-set may carry its selected run from outside its range). The verifier keeps what a passing run-set check found (rep.run_set: from, registered, open); findings_box.whole reads it instead of the check's wording. The final check's probes are tests; the parser's background marking has its own.
…er-run commands
- policy2.shell: `background` now marks the whole and-or list an `&` ends
(`a && b &`, `curl ... && x=1 &`), and a block whose closing word that `&`
follows from where the block opened (`{ ...; } &`, `while ...; done &`, if,
for, select, case). Blocks record the and-or list they opened in;
_Parser.close_block hands it back to list(). Docstring corrected.
- findings_box: DETACHERS adds constraints.LATER_RUNS (systemd-run, at,
batch), reused, not copied.
- docs/limits.md names what stays out (daemonize, start-stop-daemon
--background, sudo -b, ssh -f, an asynchronous >(...), interpreters spawning
detached children: each errs toward a false finding), and that a box run's
patch isn't checked against the operator's task.
- model_receipt: the comment no longer says the parser keeps no mark.
…ound state
list() saves the parser's list_start and closed on entry and restores them on
return, so a subshell or case arms in the same pipeline as a block no longer
overwrite them: `{ a; } | (x) &` marks a and x, `a && (x) | { b; } &` marks a,
`{ npm run dev; } | (x=1) &` marks npm.
LoopGlitch26
added a commit
that referenced
this pull request
Nov 5, 2026
…-run, E21's recorded latency rows kept Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A boxed agent's runs (#603) are its own account: its hooks run inside the box, so it can skip or forge them. The box's run holds what Box's daemon decided on the host. Until now nothing compared the two. This adds TK-X212 to
tracekit analyze --v2.What it does
Run it on a Tracekit Box box run, from a run-set bundle or the signer's store (both hold the box's agent channel runs):
Record shapes are read from Box's code (
internal/daemon/decider.go,apply.go,exec.go,events.go,internal/tracekit/tracekit.go,internal/egress/egress.go):WebFetchdecision of the box run,{url: "<scheme>://<host>:<port>"}, decided before the dial. It is refused when the decision is deny or its result isn't ok.tkbox applyis aWritedecision,{file_path: <apply dir>/<path>, patch_sha256, ...}.Bashdecision.tkbox.boxrecords the workspace,tkbox.model.callthe model calls,tkbox.egressBox's own refusals (queued, with no time of their own), andtkbox.gaplost records.The box run and its agent runs share the signer's clock. A call's window runs from its decision to
--effect-grace(2 s) after its result, with no skew.How a fetch is matched to a connection. By host and port:
www., because Claude Code's WebFetch follows a redirect that only adds or drops it, on the same protocol and port;httpto port 80, because WebFetch upgrades it to https;https:are read as//;idna), and a WHATWG number form such as0x5db8d822as a dotted address. A name ending in a number that isn't an address is no URL, sowww.<address>never matches.A fetch of a URL Box never sees explains nothing. A connection that only a fetch with unknown arguments could explain is
no_content.A box run's own calls are Box's doors, so the checks of an agent's calls (TK-X201 to TK-X208, TK-X211 and v1's text checks) skip them.
The rules
Each rule is conservative. What can't be told is counted not checkable, with a reason and a fix, and is never a finding.
door_unexplained). It is explained when it was decided in the window of a call whose tool may connect: a shell, an MCP or unknown tool, or a fetch it matches. Can't tell:model_host);no_agent_run);call_unfinished);host_command). The one command an agent run began in doesn't count; when two commands span that start, both count;background), as the policy engine's shell parser (tracekit/policy2/shell.py) reads its command, with bash's quoting, heredocs, comments, wrappers andsh -cstrings. That meansrun_in_background, a command it sends to the background (cmd &,(cmd &),bash -c 'cmd &'), or one that detaches by itself (setsid, disown, coproc,tmux … -d,screen -d…).nohup cmdalone runs in the foreground. A line the parser can't read counts too;untimed_refusal), which has no time of its own;no_content).door_unexplained). It is/workspace/<path>in the box, and no write call of the agent's that didn't fail named it. Can't tell:apply_dir);no_content), or it names no path (no_path);shell_ran);host_command).no_door_event). The agent's fetch returned ok, and no connection it matches crossed the door from the box's start to grace after its result. Can't tell:reused: keep-alive, or the tool's cache);box_refusal);no_url).door_refused). The agent's fetch returned ok, the door refused every connection it matches in its window, and no earlier connection could have served it (elsereused). Can't tell: one has no recorded outcome (no_outcome).When there's nothing to compare against:
no_agent_runs.no_agent_accountinstead (the agent registered no run at all, and the summary line saysNO AGENT ACCOUNT) only when all of these hold:register_runon the channel (tkbox.channel; a count of refusals Box folded may hold one).agent_runand says to analyse the box run.Where it is signed. Findings are signed about the box run.
review_findingtakes cites of the analysed run only, so a finding cites the box run's door records. For a fetch with no connection, it cites the box run's first record after the window: no record before it is a connection to that host. The agent's run id, call id and records are named in what prints.Rule id. I used TK-X212, the next id after the highest one (TK-X211). TK-X209 is unused, but nothing says why it was skipped, so I left it alone.
Contract. TK-X212 joins
review_finding's closed finding table under RPC 21, which is unreleased (#603). A 21 client already refuses a 20 signer, so no older signer gets the new rule.Review fixes (54706a5, f3a668a)
Each of the review's probes is now a regression test in
tests/test_findings_box.py:www.redirectMatching)(cmd &)backgrounddoor_refusedreusedhost_commandapply --dirinto a subdirectoryapply_dirno_agent_runsno_agent_account+NO AGENT ACCOUNTon the summary lineno_agent_run(in limits)backgroundfor the rest of the run (in limits):22tunnel during a fetchcall_unfinished(in limits)Also:
untimed_refusalon the door side.no_path.docs/limits.md's redirect row is corrected. New rows cover: what background forms stay out, and a host command's daemon;untimed_refusal, and "let through" ≠ "connected";no_agent_account;apply --dir.Review fix on ba6e657 (4be104c)
list()overwrote the and-or state, so a subshell or case arms in the same pipeline as a block lost what the block had recorded.list()now saves the parser'slist_startandclosedon entry and restores them on return.{ a; } | (x) &marksaandx,a && (x) | { b; } &marksa,xandb, and{ npm run dev; } | (x=1) &marksnpm. The case-arm variants are fixed the same way.test_policy2andtest_findings_box, all failing on ba6e657's parser and passing now. The reviewer'stest_bg_final.py, pointed at this branch: 3 passed, 68 subtests.tests/test_policy2.py141 passed (3,719 subtests);tests/test_findings_box.py56 passed (111 subtests).policy_hash, verdict or rule-id,decide()field, nondeterministic orconstraints.shell()differences, and 0 private-corpus differences.shell.analyse()withbackgroundset aside and sets in a fixed order: 0 differences across all 28,094 shell cases.Review fixes on 66ad1fe (ba6e657)
Each fix has a test that fails on 66ad1fe's code and passes now: 19 new subtests across
test_findings_boxandtest_policy2. Each part, undone on its own, makes a test fail (3 of 3).{,},fianddoneas words inside one simple command. So the&after a closing word marked only that last pipeline, which was empty.list()keeps the and-or list's start across&&and||. Each block records the and-or list it opened in, and_Parser.close_blockhands that back tolist(). An&after the closing word marks every command from there.door_unexplained:{ npm run dev; } &, awhile true; do …; sleep 30; done &health check,if true; then npm run dev; fi &,a && b &, and a fetch followed by&& x=1 &. The exact commands are in the tests.backgroundis corrected.DETACHERSreusesconstraints.LATER_RUNS(systemd-run, at, batch).docs/limits.mdnames what still stays out:daemonize,start-stop-daemon --background,sudo -b,ssh -f, an asynchronous>(…), and interpreters that spawn detached children. Each errs toward a false finding, never silence.tkbox.taskagainst Box's Write decisions.model_receipt.py's comment no longer says the parser keeps no mark of a background command.Verdict diff, origin/main (fecb125; #607 touches no policy, parser or constraints code) against ba6e657, using the review's
run_cases.py/run_all.py: 29,705 cases × 9 packs × 2 engines, 534,690 decisions per side.policy_hash, verdict or rule-id, otherdecide()field, nondeterministic orconstraints.shell()differences.shell.analyse()withbackgroundset aside: the script counted 25 differences, all inlearned. Itsunresolvedset hashes in each process's own string order: origin/main against itself gives 34. With sets put in a fixed order there are 0 differences across all 28,094 shell cases.background: e21 4, tests-str 89, tests-dict 5, tests-decide 1, private-corpus 2.Final check fixes (66ad1fe)
Each probe of the final check is a regression test. Each fix, undone on its own, makes a test fail (11 of 11;
whole()'s from-0 condition is pinned by a unit test)._detachesreads commands with the policy engine's own shell parser,tracekit/policy2/shell.py. I added one thing to the parser: it marks each command of a pipeline that ends in&asbackground, nested ones included. That has its own test intests/test_policy2.py; the policy engine itself reads no new key.tmux new -dorbash -c './start.sh &'after a heredoc, and a command after a# comment. These were false HIGH findings; now they'rebackground.git commit -qm'fix a & b',FOO='a & b' make,echo $'a & b',URL="…&b=2" npm test. The commit case's later connection is found again.&,echo a\\&, and\coproc,time coprocandbuiltin coproc.echo "&",echo coproc.nohup cmdalone now counts as foreground, as bash runs it (the final check's own expectation).tkbox.channelrecord with afoldedcount may hold a refusedregister_run. Such a box readsno_agent_runs.no_agent_runs. The verifier now keeps what its passing run-set check found (rep.run_set: from, registered, open), andfindings_box.whole(rep, box_run)reads that instead of the check's wording.Second review fixes (27c3ed4)
Each of these is a regression test, and each fix, undone on its own, makes a test fail (11 of 11).
NO AGENT ACCOUNT. It was claimed from any passing run-set and from a live store. It's now claimed only under the conditions above, viafindings_box.whole(rep.checks). CLI tests cover:no_agent_runs;no_agent_runs;register_run:no_agent_runs.http://host:80/…matches 443 too.-care skipped (bash -o pipefail -c,sh -e -c,bash --norc -c). So are wrappers' options (sudo -u dev,env -i,timeout 10).&counts only outside quotes and when not escaped. nohup, setsid, disown, coproc, tmux, screen and script shells count only in command position. Soecho "&",grep -c '&' f,echo coprocand a heredoc line aren't background. Outside a heredoc, a newline ends a command.www.is never stripped from an address: a name ending in a number is an address or no host.agent_hooks.tcp://…) are parsed separately from fetch URLs.no_content, not checked.Tests
tests/test_findings_box.py, 56 tests. Each builds a real run-set bundle throughfactories.BoxScene: an in-process signer that keeps content, on a synthetic clock, writing Box's records and the agent's hook calls. Each test reads the bundle back through the verifier and the vault's content. They cover:store_runs;findings.main) from both a bundle and a store.Every exported bundle verifies (the conftest check). Mutation checks: undone one at a time, each of these makes a test fail: the first 17 rules, 17 first-review fixes, 11 second-review fixes, 11 final-check fixes, and this round's 3.
On ba6e657, single files only (the machine's load was about 50; CI runs the full suite):
tests/test_findings_box.pyandtests/test_policy2.py::Shell: 106 passed, 558 subtests.tests/test_policy2.py: 141 passed, 3,714 subtests.tests/test_model_receipt.py: 42 passed.tests/test_constraints.py: 39 passed.tests/test_docs.py: 7 passed.On 66ad1fe:
make lint: pass.test_findings_box,test_findings_v2,test_findings_effects,test_findings_text,test_review_model,test_docs,test_policy2,test_box_channelandtest_bundle_v2(448 passed).py3.9 on ubuntu-latestjob ran the full suite on 66ad1fe and passed.Gates on 27c3ed4, run in the worktree with
PYTHONPATH, andTRACEKIT_PYTHONset to the venv (the TS tests spawn it). The run started at a 1-minute load of 6.75:make lint: pass.make build: pass.make test-ts: 150 passed, 0 failed, 10 skipped.make test: 4467 passed, 394 skipped, 10 xfailed, 1 failed:test_why_core.py::LongRuns::test_dense_contexts_build_their_graph_in_time. That's a timing test, and the load reached 41 by the end of the 21m48s run. This branch doesn't touchtracekit/why, and the test passes when re-run alone.The earlier full gates, on f3a668a:
make lint: pass.make build: pass.make test-ts: 150 passed, 0 failed, 10 skipped. The TS parity test now passes on the owner's 231c466.make test(f3a668a): 4458 passed, 394 skipped, 10 xfailed, 11 failed. All 11 are timing tests under load. The run took 40m38s, with five other worktrees' suites running alongside, at a load average of 36 to 51. The tests:test_parsers.py::HostileStreams::test_streamed_arguments_accumulate_in_linear_time;test_policy2.py::Speed::test_an_indented_select_in_a_sqlite3_argument_decides_fast_on_the_fallback_engine;test_policy2.py::Shell::test_arithmetic_context_prescan_is_linear;test_provenance.py::Values::test_adversarial_text_is_read_in_linear_time(8 subtests).This branch doesn't touch those modules. Re-run alone on the branch, all four pass, twice. On origin/main's code under the same load they pass too. The earlier full run of this branch (a33d491) had 4407 passed and none failed.
Run like a user
I couldn't start a signer from the shell myself (see below), so this output comes through the CLI entry point instead:
findings.main(["--v2", "--dry-run", "--run", BOX, "--bundle", set.tkb, "--trust", trust.json]), which is whattracekit analyze --v2runs. The input is the run-set bundletest_findings_box.Clibuilds (verified first), with vault content from that test's signer.The scene:
src/app.py(edited by the agent) and.github/workflows/release.yml(never named).The
--storesource gives the same output.That output is verbatim, from f3a668a. The box run's own calls no longer go through the agent checks: those show
n/a, and there is no "TK-X206 for 0 calls".The Tracekit hook
The hook stopped these steps, and I didn't route around them:
review_findingschema (sdk/typescript/src/v2/rpc_schema.ts). The owner did it in 231c466. Since then the TS parity test passes; see the gates.tracekit signer serve --configwithbox_daemons(denied: TK-D007, TK-F008, TK-F011). The demo is ready for the owner to run: a stand-in Box daemon and agent hook over the signer's socket, thentracekit export --v2 --run-set,tracekit signer trustandtracekit analyze --v2. The output above comes from the CLI entry point in the test suite instead.Not in this PR
curlorwgetisn't checked for a missing connection: a shell's exit status claims nothing about it. A write call isn't checked against the patch, because the patch is the net change.docker run -d,systemd-runandataren't read as background. A host command's daemon is the same. Their later connections are findings. All are indocs/limits.md.