From 438b12d13f25db3284c3e735fadc82434981d969 Mon Sep 17 00:00:00 2001 From: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com> Date: Fri, 13 Nov 2026 17:42:04 +0530 Subject: [PATCH 1/2] Claude Code tailer: start a stopped tailer again, park an idle one In dev mode nothing watched the transcript tailer: once it was killed, or failed before it could report, no hook started another, and every later call of the run read as reconcile.fabricated. It also gave up after two hours with no new line, or with no transcript yet, while the run was open. - The hook hands each run's tailer a lock it holds for its life. Each hook event of the run checks it; when it is free, the hook records a signed tailer_lost and starts a new tailer where the stopped one got (at most 10 times a run). - The tailer appends [transcript, offset, digest] to a progress log after each line it reports, after the hook's first line of start offsets. A new tailer reads on from there, so no line is read twice or skipped, and it checks the bytes already read against the digests: an edit made while no tailer ran is still a tailer_lost. - A dev-mode tailer quiet for IDLE_TAILER_S parks with no gap, and the run's next event starts it again. System mode keeps its idle limit. Refs #815 Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com> --- docs/coding-agents.md | 4 +- docs/limits.md | 2 +- tests/test_tailer.py | 262 ++++++++++++++++++++++++++- tracekit/integrations/claude_code.py | 139 +++++++++++--- tracekit/tailer.py | 129 +++++++++---- 5 files changed, 472 insertions(+), 64 deletions(-) diff --git a/docs/coding-agents.md b/docs/coding-agents.md index 656c7678b..9e308836c 100644 --- a/docs/coding-agents.md +++ b/docs/coding-agents.md @@ -558,7 +558,9 @@ Without anything else installed, the signer turns what it saw and then never hea with `parent` from a bound harness process is a `run.child` record in its parent; - in system mode, a transcript tailer that stops calling the signer while the run is open, killed, crashed or never started, from the moment the hook took its token (`tailer_lost`), and a second tailer token for the same run - (`tailer_redelegated`); + (`tailer_redelegated`); in dev mode, a tailer the run's next hook event finds stopped without a word (`tailer_lost`): + the hook then starts a new one where the stopped one got, so the lines written meanwhile are recorded late, not lost + (at most 10 times a run, [limits](limits.md)); - writes after the run ended (`late_activity`, in a run-set), apart from each client's first one, from which it learns that the run ended (an idle close) and moves to a new run. A hook moves to a new run too when the signer was reset (its data dir or run token key made anew) and refuses the session's run token: the earlier run, if the store still diff --git a/docs/limits.md b/docs/limits.md index 99c7d5774..ef587f724 100644 --- a/docs/limits.md +++ b/docs/limits.md @@ -90,7 +90,7 @@ What Tracekit guarantees and doesn't, per deployment, on one page: [guarantees]( | **(1.2)** A coding-agent hook completes each call that got no post-tool event (declined in the harness's own permission prompt, interrupted, or a post-tool payload it couldn't read) as an error when the session ends. That result is the hook's word, not the harness's; a session killed before its end event leaves such calls as `decided_not_completed` gaps instead. | No hook event says that a call was declined or interrupted, and a background sub-agent's call can still run when a turn ends; the session's end is the first point where none can. | Read a `tool.result` error "no PostToolUse came" as "not seen to run". | [coding agents](coding-agents.md#what-the-signer-records-as-a-gap) | | **(1.3)** A coding-agent hook blocks only tool calls. A prompt, Stop, tool result or session event whose hook fails (the signer unreachable, `tracekit` missing or broken) runs unrecorded, with a hook error in the transcript. Hook commands `tracekit init` wrote before 1.3, and Cursor hooks it wrote up to 1.4.2 (`failClosed` on every event), still turn such a failure into a block. | On a prompt, a block erases it; on Stop, it sends the agent back to work in a loop. Every tool call is still decided before it runs. | Read hook errors in the transcript; run `tracekit init` again after upgrading to get the current commands (`tracekit doctor` flags old ones in dev and system mode as `D-HOOKS-EVENT`, and old system-mode ones also as `D-HOOKS-VENV`). | [plugin](../plugin/README.md#when-a-hook-cant-run) | | **(1.2)** `concurrent_run` covers only runs bound to a harness process (harness binding, Linux system mode): a second run started from inside the same harness process while another session's run is open there, once both have recorded an event since it registered. SDK and server runs, which run several at once by design, and harnesses without binding get no such check. A run started with `parent` (a sub-agent's) gets none either: the agent can read its session's run token, so a second session can pose as a child, and shows only as a `run.child` record in the parent. Which runs overlap is kept in memory and snapshots: a signer restart with no snapshot since forgets a pending one. | One harness process drives one session at a time; without a binding the signer can't tell a forged second run from a legitimate parallel one, and an older session's run left open and idle is no overlap. | Turn on harness binding for coding agents; pass `parent` for a sub-agent's run; read each `run.child` against the harness's transcripts. | [format v2](format-v2.md#3-events) | -| **(1.2)** A killed transcript tailer is a `tailer_lost` gap only in system mode, once it has made no call (heartbeat or record) for 90 s while the run is open, counted from when the hook took its token, and only while the signer runs: the watch is kept in memory. In dev mode the tailer runs as the agent's user, which may not send heartbeats, so a killed tailer leaves no gap. Heartbeats show that a process holding the run's tailer token is alive, not that it reads the session's real transcript: the agent's user starts the tailer (through sudo) and names its transcript, so it can stop the real one and start one of its own on a file it writes. A second tailer token for the run is a `tailer_redelegated` gap, but a decoy handed the first token (the agent's user can read the hook's memory) isn't. A tailer that keeps its heartbeats but stops reading is caught only by its idle limit. | The signer can't watch the tailer's process or see the transcript; the heartbeat is its only signal, and only an identity the agent doesn't control may send it. | Use [system mode](deploy.md#laptop-system-mode) (init as root with `--v2`) where a killed tailer must show; compare the run's transcript records with the transcript itself where it matters; re-check runs that span a signer restart. | [coding agents](coding-agents.md#what-the-signer-records-as-a-gap) | +| **(1.2)** A killed transcript tailer is a `tailer_lost` gap only in system mode, once it has made no call (heartbeat or record) for 90 s while the run is open, counted from when the hook took its token, and only while the signer runs: the watch is kept in memory. In dev mode the tailer runs as the agent's user, which may not send heartbeats: it holds a lock file for its life instead, and the run's next hook event that finds the lock free records a `tailer_lost` gap and starts a new tailer where the stopped one got (at most 10 times a run). The agent's user can take that lock itself, so a tailer it stops and replaces that way leaves no gap, nor does one that stops after the run's last hook event; one quiet for two hours stops with no gap, and the run's next event starts it again. Heartbeats show that a process holding the run's tailer token is alive, not that it reads the session's real transcript: the agent's user starts the tailer (through sudo) and names its transcript, so it can stop the real one and start one of its own on a file it writes. A second tailer token for the run is a `tailer_redelegated` gap, but a decoy handed the first token (the agent's user can read the hook's memory) isn't. A tailer that keeps its heartbeats but stops reading is caught only by its idle limit. | The signer can't watch the tailer's process or see the transcript; the heartbeat is its only signal, and only an identity the agent doesn't control may send it. | Use [system mode](deploy.md#laptop-system-mode) (init as root with `--v2`) where a killed tailer must show; compare the run's transcript records with the transcript itself where it matters; re-check runs that span a signer restart. | [coding agents](coding-agents.md#what-the-signer-records-as-a-gap) | | **(1.5.3)** The transcript tailer checks the transcript lines it already read on every wake while that check is quick (under 10 ms, a transcript of about 25 MB); a larger one only once a hundred checks' time has passed, at most every 30 s, while the transcript keeps changing (a 400 MB one: about every 16 s; one of 800 MB or more: every 30 s). An edit to an already read line in that window is a `tailer_lost` gap only once the check runs, after the lines written meanwhile; one undone before then, or made just before the run ends while the transcript keeps changing, is none. | Finding an edit anywhere in what was read means reading all of it again: 0.3 s for 816 MB, which on every wake kept the tailer busy for much of a CPU core. | Keep coding-agent sessions short where in-place edits must be caught at once (`/clear` or a new session starts a new transcript); compare the run's transcript records with the transcript itself where it matters. | [coding agents](coding-agents.md#what-the-signer-records-as-a-gap) | | A session's later run (the signer closed the earlier one after an hour idle, or the session was resumed) records its transcript from where the hook last saw it while the earlier run was open: the lines Claude Code wrote after that run's last hook call (most often the closing text of a turn) can be in both runs. A resumed session is read from where its transcript (Codex's rollout, Gemini CLI's chat recording) ended at SessionStart, so a harness that writes the earlier history only after that hook repeats it in the new run. **(1.5.1)** A Claude Code session that goes on after a SessionEnd with no SessionStart (a message to the session, a usage limit's reset) is read from where its transcripts ended at the new run's first hook: what Claude Code wrote between the SessionEnd and that hook (the new prompt; a first tool call's tool use, then a `reconcile.fabricated` gap) is read by no run, since the earlier run's tailer stopped at the SessionEnd. A call in that window whose hook didn't record it (a killed or timed-out hook) leaves no `reconcile.hook_missing` gap. | The hook can't see how far the earlier run's tailer got; it measures the transcript at its own calls, and a SessionEnd removes the session's state. | Read a session's runs in order (`tracekit view --dev`): each holds what was written while it was open. | [coding agents](coding-agents.md#what-was-asked-and-said) | | **(1.5.1)** A Claude Code run's decides are compared with the transcript's tool uses without what Claude Code drops or fills in before its hooks: a key the tool doesn't take that the model sent (`run_in_background` on Agent, `persistent` and `timeout` on Monitor, `description` on Write; the first two are the tool's own in other Claude Code builds), Monitor's default `timeout_ms` (300000), a SendMessage `summary` that is its message's first line cut to 200 characters, and a Bash call's free-text `description` (the hook decides a retry that only rewords it with the approved call's, [approve, then retry](approvals.md#approve-then-retry)). A decide that differs from the model's tool use only there is no `reconcile.args_mismatch`, even when the hook, not Claude Code, made the difference; any other key or value is one, including a key a model makes up that isn't listed yet. Which runs are Claude Code's is the agent name the run registered, which the client gives: a run that says it is Claude Code gets the same leeway, for those keys only (no shipped rule reads them). An LLM gateway's record of a Claude Code call holds the arguments as the model sent them, before any of Claude Code's changes, so those changes (a Bash command's leading `cd`, Edit's `replace_all`, a key the tool doesn't take) read as `args_mismatch` there. | Claude Code hands its PreToolUse hook the tool use after its own checks and defaults, and its transcript keeps the model's keys; the list is the changes seen in Claude Code 2.1 transcripts against its hooks' input. | Read an `args_mismatch` on a Claude Code call against the call in its transcript: a key the model added that the tool doesn't take is Claude Code's own change. | [coding agents](coding-agents.md#what-was-asked-and-said) | diff --git a/tests/test_tailer.py b/tests/test_tailer.py index 39d67928b..7839f91e1 100644 --- a/tests/test_tailer.py +++ b/tests/test_tailer.py @@ -6,6 +6,7 @@ import json import os import shutil +import subprocess import sys import tempfile import time @@ -68,6 +69,22 @@ def call(self, method, **fields): return self.signer.call(self.identity, method, req) +class InProcess: + """The hook's client, answered in process by `signer` as ME.""" + + def __init__(self, signer): + self.signer = signer + + def register_run(self, req): + return self.call("register_run", req) + + def call(self, method, req=None, **fields): + req = {**(req or {}), **fields} + if "request_id" in REQUESTS[method]["properties"]: + req.setdefault("request_id", uuid.uuid4().hex) + return self.signer.call(ME, method, req) + + class _Ending: """A run handle with no signer: once `closing`, every call but tailer_lost is refused run_closed (a closing run's grace window takes late gap records); `lost`: the tailer_lost reasons.""" @@ -637,9 +654,17 @@ def setUp(self): self.client.register_run.return_value = {"run_id": "run-1", "run_token": "tok"} self.client.call.return_value = {"run_token": "delegated"} + def popen(self): + """A mock Popen whose tailers hold the run's tailer lock the hook hands them, as a running one does.""" + def run(argv, **kw): + for fd in kw.get("pass_fds", ()): + self.addCleanup(os.close, os.dup(fd)) + return mock.DEFAULT + return mock.Mock(side_effect=run) + def start(self, system=None, sid="s1", path="/home/agent/t.jsonl", source="startup"): p = {"hook_event_name": "SessionStart", "session_id": sid, "transcript_path": path, "source": source} - popen = mock.Mock() + popen = self.popen() with mock.patch.object(self.cc.subprocess, "Popen", popen), \ mock.patch.object(self.cc, "system_config", return_value=system), \ contextlib.redirect_stderr(io.StringIO()): @@ -652,10 +677,57 @@ def test_dev_mode_starts_one_same_user_tailer_per_run(self): self.assertEqual(argv[1:], ["-I", "-m", "tracekit.tailer"]) self.assertTrue(popen.call_args.kwargs["start_new_session"]) handle = json.loads(popen.return_value.stdin.write.call_args.args[0]) + lock, log = self.cc._tailer_files(self.cc._state("s1"), "run-1") self.assertEqual(handle, {"run_id": "run-1", "run_token": "tok", "path": "/home/agent/t.jsonl", - "offsets": {"/home/agent/t.jsonl": 0}, "until": self.cc._state("s1")}) + "offsets": {"/home/agent/t.jsonl": 0}, "until": self.cc._state("s1"), + "lock": lock, "progress": log}) self.start().assert_not_called() # SessionStart again (resume, clear): the run has its tailer + def test_a_tailer_stopped_more_than_restarts_times_is_not_started_again(self): + path = "/home/agent/t.jsonl" + files.write_json(self.cc._state("s1"), {"run_id": "run-1", "run_token": "tok", "stream": "x", "seq": 0, + "transcript": path, "offsets": {path: 0}, "tailer": "run-1", + "restarts": self.cc.RESTARTS}) + lock, _ = self.cc._tailer_files(self.cc._state("s1"), "run-1") + open(lock, "w").close() # its tailer held it, and is gone + self.start().assert_not_called() + method, req = self.client.call.call_args.args + self.assertEqual(method, "tailer_lost") + self.assertIn(f"it stopped {self.cc.RESTARTS + 1} times in this run and is not started again", req["reason"]) + self.assertFalse(os.path.exists(lock)) + self.client.call.reset_mock() + self.start().assert_not_called() # told once + self.client.call.assert_not_called() + + def test_a_tailer_process_holds_the_lock_it_was_handed_until_it_is_killed(self): + lock, log = self.cc._tailer_files(self.cc._state("s1"), "run-1") + procs, real = [], subprocess.Popen + + def popen(*a, **kw): + procs.append(real(*a, **kw)) + return procs[-1] + with mock.patch.object(self.cc.subprocess, "Popen", popen): # closes the hook's own copy of the lock + self.cc._start(self.client, {}, [sys.executable, "-c", "import time; time.sleep(60)"], + {"lock": lock, "progress": log}, self.cc._free(lock, create=True)) + self.client.call.assert_not_called() # it started + self.assertIsNone(self.cc._free(lock)) # held by the process + procs[0].kill() # SIGKILL: nothing of its own runs + procs[0].wait() + held = self.cc._free(lock) + self.assertIsNotNone(held) + held.close() + + def test_a_runs_tailer_files_are_kept_with_its_session(self): + files.write_json(self.cc._state("s1"), {"run_id": "run-1"}) + old = time.time() - svc.IDLE_S - 1 + kept, gone = self.cc._tailer_files(self.cc._state("s1"), "run-1"), self.cc._tailer_files( + self.cc._state("s9"), "run-9") # s9: a session whose state is gone + for f in (*kept, *gone): + open(f, "w").close() + os.utime(f, (old, old)) + self.cc._prune() + self.assertEqual([os.path.exists(f) for f in (*kept, *gone)], [True, True, False, False]) + def test_system_mode_runs_it_as_the_tailer_user_with_a_delegated_token(self): t = {"user": "tracekit-tailer", "uid": 999998, "python": "/opt/tracekit/bin/python"} popen = self.start({"signer": "/run/s.sock", "tailer": t}) @@ -682,7 +754,7 @@ def call(method, req): raise RPCError("run_closed", req["run_id"]) return {"decision": "allow"} self.client.call.side_effect = call - popen = mock.Mock() + popen = self.popen() with mock.patch.object(self.cc.subprocess, "Popen", popen), \ mock.patch.object(self.cc, "system_config", return_value=None): self.cc._run(self.client, "s1", send="decide", tool_call_id="t", tool="Bash", args={}, args_source="parsed") @@ -703,7 +775,7 @@ def call(method, req): raise RPCError("run_closed", req["run_id"]) return {"decision": "allow"} self.client.call.side_effect = call - popen = mock.Mock() + popen = self.popen() with mock.patch.object(self.cc.subprocess, "Popen", popen), \ mock.patch.object(self.cc, "system_config", return_value=None): self.cc._run(self.client, "s1", send="decide", tool_call_id="t", tool="Bash", args={}, args_source="parsed") @@ -731,7 +803,7 @@ def test_a_session_that_goes_on_with_no_state_reads_on_from_where_its_transcript from the transcript's first line: every earlier run's call was a hook_missing gap again (2,840 in one run).""" path = self.transcript(PROMPT, TOOL_USE, RESULT) p = {"hook_event_name": "UserPromptSubmit", "session_id": "s1", "transcript_path": path, "prompt": "go on"} - popen = mock.Mock() + popen = self.popen() with mock.patch.object(self.cc.subprocess, "Popen", popen), \ mock.patch.object(self.cc, "system_config", return_value=None): self.cc._handle(self.client, p, "UserPromptSubmit", "s1", None) @@ -749,7 +821,7 @@ def test_a_run_a_prompts_later_call_registers_starts_its_tailer_at_once(self): # the prompt's first call can't reach the signer; one of its observations then registers the run path = self.transcript(PROMPT, TOOL_USE, RESULT) self.client.register_run.side_effect = [SignerUnavailable("busy"), {"run_id": "run-1", "run_token": "tok"}] - popen = mock.Mock() + popen = self.popen() with self.assertRaises(SignerUnavailable): self.prompt(path, popen) self.assertEqual(json.loads(popen.return_value.stdin.write.call_args.args[0])["offsets"], @@ -757,7 +829,7 @@ def test_a_run_a_prompts_later_call_registers_starts_its_tailer_at_once(self): def test_a_tailer_started_again_after_a_failed_start_reads_from_where_its_run_began(self): path = self.transcript(PROMPT, TOOL_USE, RESULT) - begun, popen = os.path.getsize(path), mock.Mock() + begun, popen = os.path.getsize(path), self.popen() system = {"signer": "/run/s.sock", "tailer": {"user": "tracekit-tailer", "uid": 999998, "python": "/opt/tracekit/bin/python"}} self.client.call.side_effect = RPCError("forbidden", "delegate_run refused") @@ -782,5 +854,181 @@ def test_a_resumed_sessions_run_reads_its_transcript_from_where_it_ends(self): self.assertEqual(handle["offsets"], {path: os.path.getsize(path) - len(b'{"partial')}) +class Killed(BaseException): + """A tailer killed: it stops where it is, says nothing and cleans nothing up.""" + + +@unittest.skipIf(os.name == "nt", "the tailer is POSIX-only") +class HookRestartsTheTailer(unittest.TestCase): + """A dev-mode run's tailer that stops is started again by the run's next hook event (claude_code._restart), against + an in-process signer. Each tailer runs in process on the handle the hook gave it and holds the lock descriptor the + hook handed it, as the tailer process does.""" + + def setUp(self): + from tracekit.integrations import claude_code + self.cc, self.d = claude_code, tempfile.mkdtemp(dir="/tmp") + self.addCleanup(shutil.rmtree, self.d, True) + env = mock.patch.dict(os.environ, {"TRACEKIT_RUNTIME_DIR": os.path.join(self.d, "run")}) + env.start() + self.addCleanup(env.stop) + self.signer = svc.SignerService(os.path.join(self.d, "data"), grace_s=600, fsck_every_s=0) + self.addCleanup(self.signer.close) + self.client, self.path, self.started = InProcess(self.signer), os.path.join(self.d, "s1.jsonl"), [] + + def write(self, *entries, path=None): + path = path or self.path + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "ab") as f: + f.write(b"".join(line(e) for e in entries)) + + def popen(self, argv, **kw): + p = mock.Mock() + held = [os.dup(fd) for fd in kw.get("pass_fds", ())] + p.stdin.write.side_effect = lambda b: self.started.append((json.loads(b), held)) + return p + + def hook(self, **fields): + """A hook event of session s1: SessionStart (startup) without `send`, else that call.""" + with mock.patch.object(self.cc.subprocess, "Popen", self.popen), \ + mock.patch.object(self.cc, "system_config", return_value=None): + self.cc._run(self.client, "s1", transcript=self.path, **fields) + + def decide(self, tcid, tool, args): + self.hook(send="decide", tool_call_id=tcid, tool=tool, args=args, args_source="parsed") + + def tailer(self, *steps, then=None): + """Run the tailer the hook started last, one step per wake, then `then` (Killed: it is killed; None: the + session ends).""" + h, held = self.started[-1] + steps = list(steps) + + def wait(): + if steps: + return steps.pop(0)() + if then: + raise then + os.remove(h["until"]) + try: + tailer.tail(Run(self.signer, h), h["path"], os.getuid(), h.get("until"), wait, offsets=h.get("offsets"), + **{k: h[k] for k in ("digests", "progress", "lock") if k in h}) + except Killed: + pass + finally: + for fd in held: # its process is gone, and its lock with it + os.close(fd) + + def finish(self): + """The run's (reconcile records, tailer_lost reasons, model exchange ids) once it is final.""" + run = Run(self.signer, self.started[0][0]) + run.call("close_run") + self.signer.sweep(time.monotonic() + 3600) + with open(os.path.join(self.d, "data", "store", "records.jsonl"), "rb") as f: + events = [json.loads(x)["event"] for x in f.read().splitlines()] + return (sorted((e["type"], e.get("tool_call_id")) for e in events if e["type"].startswith("reconcile.")), + [e["data"]["reason"] for e in events if e["type"] == "capture.gap" + and e["data"]["kind"] == "tailer_lost"], + [e["data"].get("exchange_id") for e in events if e["type"] == "model.exchange"]) + + def test_a_killed_tailer_is_started_again_from_where_it_stopped_after_a_signed_gap(self): + """In dev mode nothing watched the tailer: once it was killed (or failed before it could say why), the run's + hook never started another, and every call after it was a reconcile.fabricated gap (about 34,000 of 69,335 in + one dev store). The run's next hook event now finds its lock free, records the stop as a signed tailer_lost and + starts a new tailer where the first one's progress log says it got, in each transcript: nothing is read twice + or skipped.""" + sub = os.path.join(self.d, "s1", "subagents", "agent-a1.jsonl") + wf = os.path.join(self.d, "s1", "subagents", "workflows", "wf_1-a2b", "agent-a2.jsonl") + self.write(PROMPT, TOOL_USE) + self.hook(start="startup") + self.decide("toolu_1", "Bash", ARGS) + self.tailer(lambda: self.write(RESULT), lambda: self.write(PROMPT, path=sub), + lambda: self.write(PROMPT, path=wf), then=Killed) + sub_use = use_line("toolu_s", "Read", {"file_path": "s.txt"}) + wf_use = use_line("toolu_w", "Read", {"file_path": "w.txt"}) + self.write(NEXT) # while no tailer runs: the session's model, a subagent and a workflow's agent ask for calls + self.write(sub_use, path=sub) + self.write(wf_use, path=wf) + self.decide("toolu_2", "Read", {"file_path": "a.txt"}) # its PreToolUse finds the tailer stopped + self.assertEqual(len(self.started), 2) + self.decide("toolu_s", "Read", {"file_path": "s.txt"}) + self.decide("toolu_w", "Read", {"file_path": "w.txt"}) + self.assertEqual(len(self.started), 2) # the new tailer holds the lock + self.tailer() + found, lost, exchanges = self.finish() + self.assertEqual(found, []) + self.assertEqual(len(lost), 1) + self.assertIn("the transcript tailer stopped without a word (killed, or it failed before it could say why); a " + "new one reads on from where it stopped (restart 1 of at most 10)", lost[0]) + self.assertEqual(sorted(exchanges), sorted(["u-3", "msg_1", "u-3", "u-3", "msg_2", "msg_toolu_s", + "msg_toolu_w"])) + h = self.started[-1][0] + self.assertFalse(os.path.exists(h["lock"]) or os.path.exists(h["progress"])) # it ended for good + + def test_a_transcript_the_stopped_tailer_never_reported_from_is_read_from_its_start(self): + """The session state's offsets are where the hook last saw the transcripts (a later run's start): a sub-agent's + first call, decided just before its tailer was killed, is read by the new tailer all the same.""" + sub = os.path.join(self.d, "s1", "subagents", "agent-a1.jsonl") + self.write(PROMPT, TOOL_USE) + self.hook(start="startup") + self.decide("toolu_1", "Bash", ARGS) + + def first_call_then_killed(): + self.write(use_line("toolu_s", "Read", {"file_path": "s.txt"}), path=sub) + self.decide("toolu_s", "Read", {"file_path": "s.txt"}) + raise Killed + self.tailer(first_call_then_killed) + self.write(NEXT) + self.decide("toolu_2", "Read", {"file_path": "a.txt"}) + self.tailer() + found, lost, exchanges = self.finish() + self.assertEqual((found, len(lost)), ([], 1)) + self.assertEqual(sorted(exchanges), ["msg_1", "msg_2", "msg_toolu_s", "u-3"]) + + def test_a_line_already_read_changed_while_no_tailer_ran_is_lost(self): + self.write(PROMPT, TOOL_USE) + self.hook(start="startup") + self.decide("toolu_1", "Bash", ARGS) + self.tailer(then=Killed) + with open(self.path, "r+b") as f: + data = f.read() + f.seek(0) + f.write(data.replace(b"ls -la", b"ls -lh")) + self.write(NEXT) + self.decide("toolu_2", "Read", {"file_path": "a.txt"}) + self.tailer(lambda: self.fail("still tailing a transcript changed in place")) + found, lost, _ = self.finish() + self.assertEqual(len(lost), 2) + self.assertIn("a line already read changed in place while no tailer ran", lost[1]) + self.assertEqual(found, [("reconcile.fabricated", "toolu_2")]) # its line came after the change: not read + + def test_an_idle_tailer_parks_and_the_runs_next_event_starts_it_again_with_no_gap(self): + """A run stays open while an approval waits, and a person may answer hours later. The tailer gave up after + IDLE_TAILER_S with no new line ("no new transcript line for 7200s"), and every call after was a + reconcile.fabricated gap. It now parks, and the run's next event starts it again from where it got.""" + self.write(PROMPT, TOOL_USE) + self.hook(start="startup") + self.decide("toolu_1", "Bash", ARGS) + with mock.patch.object(tailer, "IDLE_TAILER_S", -1): + self.tailer(lambda: None, then=AssertionError("still tailing past the idle limit")) + self.write(RESULT, NEXT) + self.decide("toolu_2", "Read", {"file_path": "a.txt"}) + self.assertEqual(len(self.started), 2) + self.tailer() + found, lost, exchanges = self.finish() + self.assertEqual((found, lost), ([], [])) + self.assertEqual(exchanges, ["u-3", "msg_1", "msg_2"]) + + def test_a_transcript_that_appears_after_the_idle_limit_is_read_once_the_tailer_is_started_again(self): + """The tailer gave up on a transcript that had not appeared after IDLE_TAILER_S ("the transcript never + appeared"), so a session whose first prompt came later read none of it.""" + self.hook(start="startup") + with mock.patch.object(tailer, "IDLE_TAILER_S", -1): + self.tailer(lambda: None, then=AssertionError("still tailing past the idle limit")) + self.write(PROMPT, TOOL_USE) + self.decide("toolu_1", "Bash", ARGS) + self.tailer() + found, lost, exchanges = self.finish() + self.assertEqual((found, lost, exchanges), ([], [], ["u-3", "msg_1"])) + + if __name__ == "__main__": unittest.main() diff --git a/tracekit/integrations/claude_code.py b/tracekit/integrations/claude_code.py index bbef2dcd3..223639b12 100644 --- a/tracekit/integrations/claude_code.py +++ b/tracekit/integrations/claude_code.py @@ -32,7 +32,10 @@ Each run of a session gets one transcript tailer (tracekit.tailer) on the transcript the hook payloads name, from the run's first event: it records the model's tool uses, thinking and text, and the user's prompts (as commitments, with -their content when the signer keeps content) from the transcript, which no hook payload carries. A new session's run +their content when the signer keeps content) from the transcript, which no hook payload carries. In dev mode each event +of the run checks that its tailer still holds the run's tailer lock, and starts it again where its progress log says it +got when it doesn't (_restart): after a signed tailer_lost gap when it stopped without a word, at most RESTARTS times a +run; with none when it parked itself (idle). A new session's run reads it from its first line; a later run of the session (after an idle close) has its tailer start where the hook last saw the transcripts while the earlier run was open (_seen), and a resumed session's first run, or one a session with no state left starts (Claude Code went on after a SessionEnd with no SessionStart), where its transcript ended then: the @@ -114,6 +117,9 @@ # never closed by the hook: it is left for the signer's idle close, its tail unproven GONE = ("run_closed", "unknown_run", "run_token_invalid") SKEW = r"Tracekit signer .* serves client " # tracekit.sdk.client's warning, which _skew tells the person instead +# lean: a run's dev-mode tailer that stops without a word is started again this many times (a crash at the same line +# would otherwise cost a gap and a process at every event); raise it if long runs see more kills +RESTARTS = 10 VERSION = re.compile(r"[\w.+!-]{1,64}", re.ASCII) # a version shown to a person: ASCII, no control character @@ -434,14 +440,101 @@ def _seen(transcript): *sorted(glob.glob(os.path.join(subs, "workflows", "*", "agent-*.jsonl")))]} +def _tailer_files(path, run_id): + """The lock a dev-mode tailer of run `run_id` holds for its life, and its progress log (tracekit.tailer), beside the + session's state `path`; kept with it (_prune).""" + base = f"{path[:-len('.json')]}.tailer-{hashlib.sha256(run_id.encode()).hexdigest()[:16]}" + return base + ".lock", base + ".log" + + +def _free(lock, create=False): + """The tailer lock `lock`, held now (a file to hand the tailer), when no tailer holds it; None while one does, once + its tailer ended for good (it removed the file), or when there is none (system mode, or a tailer an earlier hook + started), unless `create`.""" + try: + f = os.fdopen(os.open(lock, os.O_RDONLY | os.O_NOFOLLOW | (os.O_CREAT if create else 0), 0o600), "rb") + except FileNotFoundError: + return None + try: + lock_file(f, blocking=False) + if os.path.samestat(os.fstat(f.fileno()), os.stat(lock)): + return f + except OSError: # held: its tailer runs; or gone: it ended for good + pass + f.close() + return None + + +def _start(client, st, argv, handle, held=None): + """Start the tailer `argv` with `handle` on its stdin, detached; `held`: the run's tailer lock, which the tailer then + holds for its life. One that could not start is a signed gap, once, and the call goes on.""" + try: + p = subprocess.Popen(argv, stdin=subprocess.PIPE, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, + start_new_session=True, pass_fds=(held.fileno(),) if held else ()) + p.stdin.write(json.dumps(handle).encode()) + p.stdin.close() + except OSError as e: + for f in (handle["lock"], handle["progress"]) if held else (): # no hook is to take it for a stopped one + with contextlib.suppress(FileNotFoundError): + os.remove(f) + RunHandle(client, st).call("tailer_lost", reason=f"the transcript tailer did not start: {e}"[:1024], offset=0) + finally: + if held: + held.close() + + +def _restart(client, path, st): + """Start the run's dev-mode tailer again once it no longer holds its lock, where its progress log says it got (the + bytes it read checked by their digests): after a signed tailer_lost gap when it stopped without a word, at most + RESTARTS times a run, or with none when it parked itself (tracekit.tailer: idle). A run already final gets + neither.""" + lock, log = _tailer_files(path, st["run_id"]) + held = _free(lock) + if held is None: + return + from tracekit.tailer import resume # only here: it loads the signer's module + offsets, digests, parked = resume(log) + handle = {"run_id": st["run_id"], "run_token": st["run_token"], "path": st["transcript"], + "offsets": offsets or st.get("offsets") or {}, "digests": digests, "until": path, "lock": lock, + "progress": log} + n = st.get("restarts", 0) + again = parked or n < RESTARTS + try: + if not parked: + RunHandle(client, st).call("tailer_lost", offset=handle["offsets"].get(st["transcript"], 0), reason=( + "the transcript tailer stopped without a word (killed, or it failed before it could say why); " + ( + f"a new one reads on from where it stopped (restart {n + 1} of at most {RESTARTS}): the lines " + "written meanwhile are recorded late" if again else + f"it stopped {n + 1} times in this run and is not started again: what the transcript says after " + "that is not recorded: read the transcript itself for it"))) + except RPCError as e: + if e.code not in GONE: + held.close() + raise + again = False # the run ended: nothing more is read for it + if not again: + for f in (lock, log): + with contextlib.suppress(FileNotFoundError): + os.remove(f) + held.close() + return + if not parked: + st["restarts"] = n + 1 + files.write_json(path, st) + _start(client, st, [sys.executable, "-I", "-m", "tracekit.tailer"], handle, held) + + def _tail(client, path, st): - """Start tracekit.tailer for the run, detached, unless it has one. System mode: through sudo as the tailer's own - user (the root-owned system config names it), with a token the signer issued to that user's uid for this run; with - no tailer installed, the signer records a tailer_lost gap for the run instead. A run whose state names no offsets - (registered before its transcript was known, by an earlier hook) starts where the transcripts end the first time - this runs, kept in the state so a retry after a failed start reads from the same place.""" - if (AGENT != "claude-code" or not (st and st.get("transcript")) or st.get("tailer") == st["run_id"] - or os.name == "nt"): # Claude Code transcripts only; POSIX only + """Start tracekit.tailer for the run, detached, unless it has one (dev mode: start it again once it stopped, + _restart). System mode: through sudo as the tailer's own user (the root-owned system config names it), with a + token the signer issued to that user's uid for this run; with no tailer installed, the signer records a tailer_lost + gap for the run instead. A run whose state names no offsets (registered before its transcript was known, by an + earlier hook) starts where the transcripts end the first time this runs, kept in the state so a retry after a + failed start reads from the same place.""" + if AGENT != "claude-code" or not (st and st.get("transcript")) or os.name == "nt": # Claude Code only; POSIX only + return + if st.get("tailer") == st["run_id"]: + _restart(client, path, st) return # lean: a run a session goes on with after a SessionEnd (no SessionStart: a message to the session, a usage limit's # reset) starts where the transcripts end at its first event; what Claude Code wrote between the SessionEnd and @@ -453,26 +546,21 @@ def _tail(client, path, st): handle = {"run_id": st["run_id"], "run_token": st["run_token"], "path": st["transcript"], "offsets": st["offsets"]} sc = system_config() or {} t = sc.get("tailer") - argv = None # a tailer that dies or is killed stops its heartbeats: the signer writes the tailer_lost gap if t: - # this tailer cannot see SessionEnd (the session state is the agent's): it exits once a call finds the run - # final, or with a tailer_lost after tailer.IDLE_TAILER_S without a new line + # a tailer that dies or is killed stops its heartbeats: the signer writes the tailer_lost gap. This one cannot + # see SessionEnd (the session state is the agent's): it exits once a call finds the run final handle["run_token"] = RunHandle(client, st).call("delegate_run", identity=f"uid:{t['uid']}")["run_token"] - argv = ["sudo", "-n", "-u", t["user"], t["python"], "-I", "-m", "tracekit.tailer"] + _start(client, st, ["sudo", "-n", "-u", t["user"], t["python"], "-I", "-m", "tracekit.tailer"], handle) elif sc.get("signer"): # system mode, but init could not give the tailer access RunHandle(client, st).call("tailer_lost", reason="no transcript tailer is installed", offset=0) - else: # exits once SessionEnd removes the session state - handle["until"] = path - argv = [sys.executable, "-I", "-m", "tracekit.tailer"] - if argv: - try: - p = subprocess.Popen(argv, stdin=subprocess.PIPE, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, - start_new_session=True) - p.stdin.write(json.dumps(handle).encode()) - p.stdin.close() - except OSError as e: # no tailer could start: a signed gap says so, once, and the call goes on - RunHandle(client, st).call("tailer_lost", reason=f"the transcript tailer did not start: {e}"[:1024], - offset=0) + else: # exits once SessionEnd removes the session state; holds the run's tailer lock, so a hook sees it stop + lock, log = _tailer_files(path, st["run_id"]) + held = _free(lock, create=True) + if held: # else a tailer of this run holds it already + with os.fdopen(os.open(log, os.O_WRONLY | os.O_CREAT | os.O_TRUNC | os.O_NOFOLLOW, 0o600), "w") as f: + f.write(json.dumps({"offsets": st["offsets"]}) + "\n") # where a restarted one reads what it never did + _start(client, st, [sys.executable, "-I", "-m", "tracekit.tailer"], + dict(handle, until=path, lock=lock, progress=log), held) st["tailer"] = st["run_id"] files.write_json(path, st) @@ -992,6 +1080,9 @@ def _prune(): # could lock a fresh file beside one still held, so two runs get registered, rare enough for dev mode if f.endswith(".lock") and os.path.exists(f[:-len(".lock")] + ".json"): continue + if ".tailer-" in f and os.path.exists(os.path.join(os.path.dirname(f), os.path.basename(f).split(".")[0] + + ".json")): # a run's tailer's: kept with its session + continue session = f.endswith(".json") and os.path.basename(f).count(".") == 1 # not claude-code-..json try: if os.path.getmtime(f) < now - (SESSION_KEEP_S if session else IDLE_S): diff --git a/tracekit/tailer.py b/tracekit/tailer.py index 73429bcf3..8ede2d6a6 100644 --- a/tracekit/tailer.py +++ b/tracekit/tailer.py @@ -1,14 +1,23 @@ """L1 transcript tailer (04-design ยง6): follows one Claude Code session transcript and reports it to the signer. python -I -m tracekit.tailer (started by the v2 hook at the run's first event; the run handle comes on stdin as - JSON: {"run_id", "run_token", "path", "until"?, "offsets"?}) + JSON: {"run_id", "run_token", "path", "until"?, "offsets"?, "digests"?, + "progress"?, "lock"?}) `offsets` ({path: offset}): where to start reading the session's transcripts: {path: 0} for a new session; for a later run (an idle close, a resumed session, a session that goes on after SessionEnd), where the hook measured them, as the lines before are an earlier run's. Dev mode: a detached process of the agent's own user, which exits once the hook's session state (`until`) is gone -(SessionEnd). System mode (`tracekit init --v2`): run through sudo as the tracekit-tailer user, which reads the agent's +(SessionEnd). It holds the run's tailer lock (`lock`, a file the hook locked and handed it as an open descriptor) for +its life, and appends [transcript, offset, digest] to its progress log (`progress`, which the hook begins with +{"offsets": the run's first tailer's}) after each line it reports: the offset just past that line, and the sha256 of +the transcript's bytes before it. A hook that finds the lock free while +the run is open starts a new tailer from there (`offsets`, with `digests` to check each transcript's bytes already read +against): after a signed tailer_lost gap, as the tailer stopped without a word (killed, or it failed before it could +report), or with none when it parked itself (`"parked"`, the log's last line) after IDLE_TAILER_S without a new line. +A tailer that ends for good (the run ended, SessionEnd, or a loss it reported) removes its lock and log first. +System mode (`tracekit init --v2`): run through sudo as the tracekit-tailer user, which reads the agent's ~/.claude/projects through an ACL and nothing of the signer's. It calls the signer as its own uid with a token the hook delegated (`delegate_run`), and signer.yaml lets that uid call only model_event, state_write, tailer_lost, tailer_heartbeat and status. @@ -30,10 +39,10 @@ that as its `body` when the signer keeps content (the content vault, docs/privacy.md); without the vault the text stays in the transcript. A rate limit (quota_exceeded with retry_after_ms) sends the line again once it has passed. It exits once the signer says the run is over, or (dev mode) once `until` is gone. A transcript that never -appears or cannot be reached, and a tailer that stops after IDLE_TAILER_S without a new line, are a tailer_lost too. -Every HEARTBEAT_S it sends `tailer_heartbeat` (another refusal, e.g. a quota, is tried again at the next); a tailer that -is killed stops them, and the signer writes the tailer_lost gap itself (a signer that does not know the call, or a token -not granted it, just gets none). +appears (by SessionEnd) or cannot be reached, and a tailer with no progress log (system mode) that stops after +IDLE_TAILER_S without a new line, are a tailer_lost too. Every HEARTBEAT_S it sends `tailer_heartbeat` (another +refusal, e.g. a quota, is tried again at the next); a tailer that is killed stops them, and the signer writes the +tailer_lost gap itself (a signer that does not know the call, or a token not granted it, just gets none). The session's subagents' transcripts (/subagents/agent-*.jsonl, next to the session's, and a workflow's agents' in /subagents/workflows//) are followed the same way, with the same checks, from the wake they appear on: a call one of them makes is decided under the session's run. @@ -60,8 +69,8 @@ POLL_S = 1.0 CHECK_S = 30 # the longest a changing transcript's bytes already read go without being hashed again CHEAP_S = 0.01 # a rehash that took less is done on every wake that sees the file changed -# lean: twice the signer's default run idle timeout, so a quiet run has closed before the tailer gives up on it; ask the -# signer for the run's state instead if signers run with a longer idle_s +# lean: twice the signer's default run idle timeout, so a quiet run has closed before a tailer no hook resumes (system +# mode) gives up on it; ask the signer for the run's state instead if signers run with a longer idle_s IDLE_TAILER_S = 2 * IDLE_S ENDED = ("run_closed", "unknown_run", "run_token_invalid") IN_EVENTS = 0x2 | 0x4 | 0x40 | 0x80 | 0x100 | 0x200 # MODIFY, ATTRIB, MOVED_FROM, MOVED_TO, CREATE, DELETE @@ -161,7 +170,7 @@ def __call__(self, **fields): def _report(send, e, sent, name="main"): """The model_event of one transcript entry, if it has one (`send`, a _Sender); `sent` collects tool result ids - between responses; `name`: the transcript it came from.""" + between responses; `name`: the transcript it came from. -> whether it sent one.""" msg = e.get("message") if isinstance(e.get("message"), dict) else {} content = msg.get("content") blocks = [b for b in content if isinstance(b, dict)] if isinstance(content, list) else [] @@ -178,18 +187,23 @@ def _report(send, e, sent, name="main"): for b in uses], "tool_results_sent": list(sent)} if uses else {})) if uses: sent.clear() - elif e.get("type") == "user": - if isinstance(content, str) or any(b.get("type") == "text" for b in blocks): - send(provider="anthropic", model="unknown", phase="request", exchange_id=e.get("uuid"), - content_digest=event_hash(read), **send.body(read)) - sent += [b.get("tool_use_id") for b in blocks if b.get("type") == "tool_result"] + return True + if e.get("type") != "user": + return False + said = isinstance(content, str) or any(b.get("type") == "text" for b in blocks) + if said: + send(provider="anthropic", model="unknown", phase="request", exchange_id=e.get("uuid"), + content_digest=event_hash(read), **send.body(read)) + sent += [b.get("tool_use_id") for b in blocks if b.get("type") == "tool_result"] + return said class _Transcript: - """One transcript being followed: its kept file, how far it was read, and the digest of the bytes before that.""" + """One transcript being followed: its kept file, how far it was read, and the digest of the bytes before that + (`digest`: the hex sha256 those bytes must have, from a stopped tailer's progress log).""" - def __init__(self, path, name="main", off=0): - self.path, self.name, self.f, self.off, self.sent = path, name, None, off, [] + def __init__(self, path, name="main", off=0, digest=None): + self.path, self.name, self.f, self.off, self.sent, self.digest = path, name, None, off, [], digest # the bytes before `off`; the ctime they were last checked at, when (monotonic), and how long that took self.read, self.checked, self.at, self.cost = hashlib.sha256(), None, None, 0.0 @@ -229,6 +243,8 @@ def _verify(t, uid, last=False): began = time.thread_time() # CPU time: a busy machine (or agent) doesn't make a rehash look costly _sha256(t.f, t.off, t.read) # an earlier run's lines, from here on checked as the ones it reads t.at, t.cost = time.monotonic(), time.thread_time() - began + if t.digest is not None and t.read.hexdigest() != t.digest: + raise Lost("a line already read changed in place while no tailer ran") _check(t.f, t.path, uid, t.off) ctime = os.fstat(t.f.fileno()).st_ctime_ns # any write moves it, and no user can set it back # a ctime under a second old may not move for a change in the same clock tick: checked again once it is older @@ -245,8 +261,9 @@ def _verify(t, uid, last=False): return True -def _follow(send, t, uid): - """Report transcript t's new complete lines (`send`, a _Sender), once _verify has checked it. -> whether it grew.""" +def _follow(send, t, uid, note=lambda t: None): + """Report transcript t's new complete lines (`send`, a _Sender), once _verify has checked it; `note(t)` after each + line that sent a model_event. -> whether it grew.""" if not _verify(t, uid): return False t.f.seek(t.off) @@ -256,7 +273,7 @@ def _follow(send, t, uid): if e is None: raise Lost(f"the line at offset {t.off} is not a JSON object") try: - _report(send, e, t.sent, t.name) + sent = _report(send, e, t.sent, t.name) except SignerUnavailable: raise except RPCError as x: @@ -265,10 +282,12 @@ def _follow(send, t, uid): raise Lost(f"the line at offset {t.off} was refused: {x.message[:256]}") from None except Exception as x: # e.g. a number canonical JSON cannot hold raise Lost(f"the line at offset {t.off} is unreadable ({type(x).__name__})") from None + t.read.update(data[t.off - start:t.off - start + n]) t.off += n + if sent: + note(t) except SignerUnavailable: # read again from `off` on the next wake pass - t.read.update(data[:t.off - start]) return t.off > start @@ -294,41 +313,81 @@ def _tell(run, main, t, e): run.call("tailer_lost", reason=name + why, offset=t.off) -def tail(run, path, uid, until=None, wait=None, status=None, offsets=None): +def resume(progress): + """Where a stopped dev-mode tailer got, from its progress log (module docstring): ({transcript: the offset just + past the last line it reported, else where the run's first tailer started it}, {transcript: the hex sha256 of its + bytes before that offset}, whether it parked).""" + offsets, digests, last = {}, {}, None + try: + with open(progress, "rb") as f: + for raw in f: + try: + last = json.loads(raw) + except ValueError: # a line cut short when it was killed + last = None + continue + if isinstance(last, dict) and isinstance(last.get("offsets"), dict): + for p, o in last["offsets"].items(): + if type(o) is int and o >= 0: + offsets[p] = o + digests.pop(p, None) + elif isinstance(last, list) and len(last) == 3 and isinstance(last[0], str) and \ + type(last[1]) is int and last[1] >= 0 and isinstance(last[2], str): + offsets[last[0]], digests[last[0]] = last[1], last[2] + except FileNotFoundError: + pass + return offsets, digests, last == "parked" + + +def tail(run, path, uid, until=None, wait=None, status=None, offsets=None, digests=None, progress=None, lock=None): """Follow `path`, and its subagents' transcripts, for `run` until the run ends, `until` is gone, or a transcript is lost (reported). `status()`: the signer's status answer, which says whether it keeps content (VaultBody); - `offsets` {path: offset}: where to start each (module docstring).""" + `offsets` {path: offset}: where to start each, and `digests` {path: hex sha256} what the bytes before must hash + to; `progress`, `lock`: dev mode's progress log and the run's tailer lock (module docstring).""" wait, grew, beat = wait or _waiter(path), time.monotonic(), -HEARTBEAT_S send = _Sender(run, status) # lean: keeps one file open per subagent transcript; close quiet ones and reopen them by (dev, inode) if sessions # run more subagents than the tailer's file limit start = {p: o for p, o in (offsets.items() if isinstance(offsets, dict) else ()) if type(o) is int and o >= 0} - main = cur = _Transcript(path, off=start.get(path, 0)) + sums = {p: d for p, d in (digests.items() if isinstance(digests, dict) else ()) if isinstance(d, str)} + main = cur = _Transcript(path, off=start.get(path, 0), digest=sums.get(path)) ts = {path: main} + log = os.open(progress, os.O_WRONLY | os.O_APPEND | os.O_CREAT | os.O_NOFOLLOW, 0o600) if progress else None + + def note(t): + if log is not None: + os.write(log, json.dumps([t.path, t.off, t.read.hexdigest()]).encode() + b"\n") + done = False # it ends for good: no hook is to start it again try: while True: - gone, idle = until is not None and not os.path.exists(until), time.monotonic() - grew > IDLE_TAILER_S + gone = until is not None and not os.path.exists(until) cur = main for p in _subagents(path) if main.f is not None else (): - ts.setdefault(p, _Transcript(p, os.path.basename(p), start.get(p, 0))) + ts.setdefault(p, _Transcript(p, os.path.basename(p), start.get(p, 0), sums.get(p))) for cur in ts.values() if gone else (): # the last wake checks every transcript before a line can be _verify(cur, uid, True) # refused for the run's end (SessionEnd closes it first) for cur in ts.values(): - if _follow(send, cur, uid): + if _follow(send, cur, uid, note): grew = time.monotonic() cur = main - if main.f is None and (gone or idle): + idle = time.monotonic() - grew > IDLE_TAILER_S + if main.f is None and (gone or idle and log is None): raise Lost("the transcript never appeared") - if idle: - raise Lost(f"no new transcript line for {IDLE_TAILER_S:.0f}s: the tailer stopped") if gone: + done = True return + if idle and log is not None: # parked: a hook starts it again from here at the session's next event + os.write(log, b'"parked"\n') + return + if idle: + raise Lost(f"no new transcript line for {IDLE_TAILER_S:.0f}s: the tailer stopped") if beat is not None and time.monotonic() - beat >= HEARTBEAT_S: beat = _heartbeat(run) wait() except RPCError as e: if e.code not in ENDED: raise + done = True try: # the run ended: a line already read that changed is still told, while the signer takes late records for cur in ts.values(): if cur.f is not None: @@ -342,10 +401,18 @@ def tail(run, path, uid, until=None, wait=None, status=None, offsets=None): raise except Exception as e: # Lost, or a failure of the tailer's own (a bug, a read error): never a silent stop _tell(run, main, cur, e) + done = True finally: for t in ts.values(): if t.f is not None: t.f.close() + if log is not None: + os.close(log) + for f in (lock, progress) if done and progress else (): + try: + os.remove(f) + except FileNotFoundError: # SessionEnd removed it + pass def main(): @@ -356,7 +423,7 @@ def main(): client = Client() try: tail(RunHandle(client, h), h["path"], uid, h.get("until"), status=lambda: client.status({}), - offsets=h.get("offsets")) + offsets=h.get("offsets"), digests=h.get("digests"), progress=h.get("progress"), lock=h.get("lock")) except RPCError as e: # the run ended while the loss was being reported if e.code not in ENDED: raise From f355ee174d4546becfe51334f0bcac0647334c6e Mon Sep 17 00:00:00 2001 From: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com> Date: Fri, 13 Nov 2026 18:47:41 +0530 Subject: [PATCH 2/2] Tailer restart: read every transcript at SessionEnd, dev mode only Review fixes for the stopped-tailer restart: - A tailer started at SessionEnd (the first event after it parked or was killed) had its first wake as its last, and listed the sub-agents' transcripts only once the main one was open: it read the main one alone. It now opens the main one before listing them. - The hook starts no tailer for a lock file in system mode, where the runtime dir is the agent's, and none for a run it found ended. - SessionEnd's cleanup takes a file the tailer removed meanwhile. - Tests: restarts counted through hook events up to the cap, and a lock removed between the hook's open and its lock. - docs/limits.md: the agent's user can also write the progress log, and a missing log reads on from where the hook last saw the transcripts. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com> --- docs/limits.md | 2 +- tests/test_tailer.py | 120 +++++++++++++++++++++++++-- tracekit/integrations/claude_code.py | 21 +++-- tracekit/tailer.py | 2 + 4 files changed, 129 insertions(+), 16 deletions(-) diff --git a/docs/limits.md b/docs/limits.md index ef587f724..35037b6cb 100644 --- a/docs/limits.md +++ b/docs/limits.md @@ -90,7 +90,7 @@ What Tracekit guarantees and doesn't, per deployment, on one page: [guarantees]( | **(1.2)** A coding-agent hook completes each call that got no post-tool event (declined in the harness's own permission prompt, interrupted, or a post-tool payload it couldn't read) as an error when the session ends. That result is the hook's word, not the harness's; a session killed before its end event leaves such calls as `decided_not_completed` gaps instead. | No hook event says that a call was declined or interrupted, and a background sub-agent's call can still run when a turn ends; the session's end is the first point where none can. | Read a `tool.result` error "no PostToolUse came" as "not seen to run". | [coding agents](coding-agents.md#what-the-signer-records-as-a-gap) | | **(1.3)** A coding-agent hook blocks only tool calls. A prompt, Stop, tool result or session event whose hook fails (the signer unreachable, `tracekit` missing or broken) runs unrecorded, with a hook error in the transcript. Hook commands `tracekit init` wrote before 1.3, and Cursor hooks it wrote up to 1.4.2 (`failClosed` on every event), still turn such a failure into a block. | On a prompt, a block erases it; on Stop, it sends the agent back to work in a loop. Every tool call is still decided before it runs. | Read hook errors in the transcript; run `tracekit init` again after upgrading to get the current commands (`tracekit doctor` flags old ones in dev and system mode as `D-HOOKS-EVENT`, and old system-mode ones also as `D-HOOKS-VENV`). | [plugin](../plugin/README.md#when-a-hook-cant-run) | | **(1.2)** `concurrent_run` covers only runs bound to a harness process (harness binding, Linux system mode): a second run started from inside the same harness process while another session's run is open there, once both have recorded an event since it registered. SDK and server runs, which run several at once by design, and harnesses without binding get no such check. A run started with `parent` (a sub-agent's) gets none either: the agent can read its session's run token, so a second session can pose as a child, and shows only as a `run.child` record in the parent. Which runs overlap is kept in memory and snapshots: a signer restart with no snapshot since forgets a pending one. | One harness process drives one session at a time; without a binding the signer can't tell a forged second run from a legitimate parallel one, and an older session's run left open and idle is no overlap. | Turn on harness binding for coding agents; pass `parent` for a sub-agent's run; read each `run.child` against the harness's transcripts. | [format v2](format-v2.md#3-events) | -| **(1.2)** A killed transcript tailer is a `tailer_lost` gap only in system mode, once it has made no call (heartbeat or record) for 90 s while the run is open, counted from when the hook took its token, and only while the signer runs: the watch is kept in memory. In dev mode the tailer runs as the agent's user, which may not send heartbeats: it holds a lock file for its life instead, and the run's next hook event that finds the lock free records a `tailer_lost` gap and starts a new tailer where the stopped one got (at most 10 times a run). The agent's user can take that lock itself, so a tailer it stops and replaces that way leaves no gap, nor does one that stops after the run's last hook event; one quiet for two hours stops with no gap, and the run's next event starts it again. Heartbeats show that a process holding the run's tailer token is alive, not that it reads the session's real transcript: the agent's user starts the tailer (through sudo) and names its transcript, so it can stop the real one and start one of its own on a file it writes. A second tailer token for the run is a `tailer_redelegated` gap, but a decoy handed the first token (the agent's user can read the hook's memory) isn't. A tailer that keeps its heartbeats but stops reading is caught only by its idle limit. | The signer can't watch the tailer's process or see the transcript; the heartbeat is its only signal, and only an identity the agent doesn't control may send it. | Use [system mode](deploy.md#laptop-system-mode) (init as root with `--v2`) where a killed tailer must show; compare the run's transcript records with the transcript itself where it matters; re-check runs that span a signer restart. | [coding agents](coding-agents.md#what-the-signer-records-as-a-gap) | +| **(1.2)** A killed transcript tailer is a `tailer_lost` gap only in system mode, once it has made no call (heartbeat or record) for 90 s while the run is open, counted from when the hook took its token, and only while the signer runs: the watch is kept in memory. In dev mode the tailer runs as the agent's user, which may not send heartbeats: it holds a lock file for its life instead, and the run's next hook event that finds the lock free records a `tailer_lost` gap and starts a new tailer where the stopped one got, by the run's progress log (at most 10 times a run). One quiet for two hours stops with no gap ("parked" in that log), and the run's next event starts it again. Both files are the agent's user's: it can take the lock itself, so a tailer it stops and replaces that way leaves no gap, and it can write the log (an offset past lines not yet read, then "parked") before it kills the tailer, so the new one skips those lines with no gap, as it could stop a dev-mode tailer unrecorded before. A tailer that stops after the run's last hook event leaves no gap either, and with its log removed a new one reads on from where the hook last saw the transcripts, skipping what the stopped one hadn't read by then. Heartbeats show that a process holding the run's tailer token is alive, not that it reads the session's real transcript: the agent's user starts the tailer (through sudo) and names its transcript, so it can stop the real one and start one of its own on a file it writes. A second tailer token for the run is a `tailer_redelegated` gap, but a decoy handed the first token (the agent's user can read the hook's memory) isn't. A tailer that keeps its heartbeats but stops reading is caught only by its idle limit. | The signer can't watch the tailer's process or see the transcript; the heartbeat is its only signal, and only an identity the agent doesn't control may send it. | Use [system mode](deploy.md#laptop-system-mode) (init as root with `--v2`) where a killed tailer must show; compare the run's transcript records with the transcript itself where it matters; re-check runs that span a signer restart. | [coding agents](coding-agents.md#what-the-signer-records-as-a-gap) | | **(1.5.3)** The transcript tailer checks the transcript lines it already read on every wake while that check is quick (under 10 ms, a transcript of about 25 MB); a larger one only once a hundred checks' time has passed, at most every 30 s, while the transcript keeps changing (a 400 MB one: about every 16 s; one of 800 MB or more: every 30 s). An edit to an already read line in that window is a `tailer_lost` gap only once the check runs, after the lines written meanwhile; one undone before then, or made just before the run ends while the transcript keeps changing, is none. | Finding an edit anywhere in what was read means reading all of it again: 0.3 s for 816 MB, which on every wake kept the tailer busy for much of a CPU core. | Keep coding-agent sessions short where in-place edits must be caught at once (`/clear` or a new session starts a new transcript); compare the run's transcript records with the transcript itself where it matters. | [coding agents](coding-agents.md#what-the-signer-records-as-a-gap) | | A session's later run (the signer closed the earlier one after an hour idle, or the session was resumed) records its transcript from where the hook last saw it while the earlier run was open: the lines Claude Code wrote after that run's last hook call (most often the closing text of a turn) can be in both runs. A resumed session is read from where its transcript (Codex's rollout, Gemini CLI's chat recording) ended at SessionStart, so a harness that writes the earlier history only after that hook repeats it in the new run. **(1.5.1)** A Claude Code session that goes on after a SessionEnd with no SessionStart (a message to the session, a usage limit's reset) is read from where its transcripts ended at the new run's first hook: what Claude Code wrote between the SessionEnd and that hook (the new prompt; a first tool call's tool use, then a `reconcile.fabricated` gap) is read by no run, since the earlier run's tailer stopped at the SessionEnd. A call in that window whose hook didn't record it (a killed or timed-out hook) leaves no `reconcile.hook_missing` gap. | The hook can't see how far the earlier run's tailer got; it measures the transcript at its own calls, and a SessionEnd removes the session's state. | Read a session's runs in order (`tracekit view --dev`): each holds what was written while it was open. | [coding agents](coding-agents.md#what-was-asked-and-said) | | **(1.5.1)** A Claude Code run's decides are compared with the transcript's tool uses without what Claude Code drops or fills in before its hooks: a key the tool doesn't take that the model sent (`run_in_background` on Agent, `persistent` and `timeout` on Monitor, `description` on Write; the first two are the tool's own in other Claude Code builds), Monitor's default `timeout_ms` (300000), a SendMessage `summary` that is its message's first line cut to 200 characters, and a Bash call's free-text `description` (the hook decides a retry that only rewords it with the approved call's, [approve, then retry](approvals.md#approve-then-retry)). A decide that differs from the model's tool use only there is no `reconcile.args_mismatch`, even when the hook, not Claude Code, made the difference; any other key or value is one, including a key a model makes up that isn't listed yet. Which runs are Claude Code's is the agent name the run registered, which the client gives: a run that says it is Claude Code gets the same leeway, for those keys only (no shipped rule reads them). An LLM gateway's record of a Claude Code call holds the arguments as the model sent them, before any of Claude Code's changes, so those changes (a Bash command's leading `cd`, Edit's `replace_all`, a key the tool doesn't take) read as `args_mismatch` there. | Claude Code hands its PreToolUse hook the tool use after its own checks and defaults, and its transcript keeps the model's keys; the list is the changes seen in Claude Code 2.1 transcripts against its hooks' input. | Read an `args_mismatch` on a Claude Code call against the call in its transcript: a key the model added that the tool doesn't take is Claude Code's own change. | [coding agents](coding-agents.md#what-was-asked-and-said) | diff --git a/tests/test_tailer.py b/tests/test_tailer.py index 7839f91e1..d22a993a8 100644 --- a/tests/test_tailer.py +++ b/tests/test_tailer.py @@ -684,16 +684,11 @@ def test_dev_mode_starts_one_same_user_tailer_per_run(self): self.start().assert_not_called() # SessionStart again (resume, clear): the run has its tailer def test_a_tailer_stopped_more_than_restarts_times_is_not_started_again(self): - path = "/home/agent/t.jsonl" - files.write_json(self.cc._state("s1"), {"run_id": "run-1", "run_token": "tok", "stream": "x", "seq": 0, - "transcript": path, "offsets": {path: 0}, "tailer": "run-1", - "restarts": self.cc.RESTARTS}) - lock, _ = self.cc._tailer_files(self.cc._state("s1"), "run-1") - open(lock, "w").close() # its tailer held it, and is gone + lock, _ = self.state(restarts=self.cc.RESTARTS) self.start().assert_not_called() method, req = self.client.call.call_args.args self.assertEqual(method, "tailer_lost") - self.assertIn(f"it stopped {self.cc.RESTARTS + 1} times in this run and is not started again", req["reason"]) + self.assertIn(f"it is not started again (it stopped {self.cc.RESTARTS + 1} times in this run)", req["reason"]) self.assertFalse(os.path.exists(lock)) self.client.call.reset_mock() self.start().assert_not_called() # told once @@ -728,6 +723,58 @@ def test_a_runs_tailer_files_are_kept_with_its_session(self): self.cc._prune() self.assertEqual([os.path.exists(f) for f in (*kept, *gone)], [True, True, False, False]) + def test_a_lock_its_tailer_removed_as_it_ended_is_not_taken_for_a_stopped_tailers(self): + lock, _ = self.cc._tailer_files(self.cc._state("s1"), "run-1") + tailers = self.cc._free(lock, create=True) + real = self.cc.lock_file + + def ended(f, blocking=True): # between the hook's open and its lock, the tailer ends for good + os.remove(lock) + tailers.close() + return real(f, blocking) + with mock.patch.object(self.cc, "lock_file", ended): + self.assertIsNone(self.cc._free(lock)) + + def state(self, **fields): + """Session s1's state: run-1, whose tailer the hook started; its tailer lock, free (that tailer is gone).""" + path = "/home/agent/t.jsonl" + files.write_json(self.cc._state("s1"), {"run_id": "run-1", "run_token": "tok", "stream": "x", "seq": 0, + "transcript": path, "offsets": {path: 0}, "tailer": "run-1", + **fields}) + lock, log = self.cc._tailer_files(self.cc._state("s1"), "run-1") + open(lock, "w").close() + return lock, log + + def test_session_end_takes_a_file_its_tailer_removed_meanwhile(self): + self.start() + real, gone = self.cc.glob.glob, self.cc._state("s1")[:-len(".json")] + ".tailer-0.log" # listed, then removed + p = {"hook_event_name": "SessionEnd", "session_id": "s1", "reason": "other"} + with mock.patch.object(self.cc.glob, "glob", lambda pat: real(pat) + [gone] * pat.endswith("*")), \ + mock.patch.object(self.cc, "system_config", return_value=None): + self.assertEqual(self.cc._handle(self.client, p, "SessionEnd", "s1", None), 0) + self.assertFalse(os.path.exists(self.cc._state("s1"))) + + def test_system_mode_starts_no_tailer_for_a_tailer_lock_in_the_agents_runtime_dir(self): + """The runtime dir is the agent's: a tailer lock file it puts there makes no system-mode hook start a tailer + of the agent's uid, nor sign a gap.""" + self.state() + system = {"signer": "/run/s.sock", "tailer": {"user": "tracekit-tailer", "uid": 999998, + "python": "/opt/tracekit/bin/python"}} + self.start(system).assert_not_called() + self.client.call.assert_not_called() + + def test_a_run_the_hook_found_ended_gets_no_new_tailer(self): + lock, log = self.state(ended=True) + with open(log, "w") as f: + f.write('{"offsets": {}}\n"parked"\n') + popen = self.popen() + with mock.patch.object(self.cc.subprocess, "Popen", popen), \ + mock.patch.object(self.cc, "system_config", return_value=None): + self.cc._run(self.client, "s1", register=False) + popen.assert_not_called() + self.client.call.assert_not_called() # parked: nothing was missed + self.assertFalse(os.path.exists(lock) or os.path.exists(log)) + def test_system_mode_runs_it_as_the_tailer_user_with_a_delegated_token(self): t = {"user": "tracekit-tailer", "uid": 999998, "python": "/opt/tracekit/bin/python"} popen = self.start({"signer": "/run/s.sock", "tailer": t}) @@ -907,7 +954,8 @@ def wait(): return steps.pop(0)() if then: raise then - os.remove(h["until"]) + with contextlib.suppress(FileNotFoundError): # SessionEnd removed it + os.remove(h["until"]) try: tailer.tail(Run(self.signer, h), h["path"], os.getuid(), h.get("until"), wait, offsets=h.get("offsets"), **{k: h[k] for k in ("digests", "progress", "lock") if k in h}) @@ -920,7 +968,8 @@ def wait(): def finish(self): """The run's (reconcile records, tailer_lost reasons, model exchange ids) once it is final.""" run = Run(self.signer, self.started[0][0]) - run.call("close_run") + with contextlib.suppress(RPCError): # SessionEnd closed it + run.call("close_run") self.signer.sweep(time.monotonic() + 3600) with open(os.path.join(self.d, "data", "store", "records.jsonl"), "rb") as f: events = [json.loads(x)["event"] for x in f.read().splitlines()] @@ -929,6 +978,59 @@ def finish(self): and e["data"]["kind"] == "tailer_lost"], [e["data"].get("exchange_id") for e in events if e["type"] == "model.exchange"]) + def park(self): + with mock.patch.object(tailer, "IDLE_TAILER_S", -1): + self.tailer(then=AssertionError("still tailing past the idle limit")) + + def session_end_after(self, stop): + """SessionEnd as the first event after the tailer stopped (`stop`): the hook starts it once more, then closes + the run and removes the session's state, so the new tailer's first wake is its last. Meanwhile a sub-agent said + something and called a tool no hook decided, and a workflow's agent called one too.""" + sub = os.path.join(self.d, "s1", "subagents", "agent-a1.jsonl") + wf = os.path.join(self.d, "s1", "subagents", "workflows", "wf_1-a2b", "agent-a2.jsonl") + self.write(PROMPT, TOOL_USE) + self.hook(start="startup") + self.decide("toolu_1", "Bash", ARGS) + stop() + self.write({**SAID, "uuid": "u-s", "message": {**SAID["message"], "id": "msg_s"}}, + use_line("toolu_x", "Read", {"file_path": "x.txt"}), path=sub) + self.write(use_line("toolu_w", "Read", {"file_path": "w.txt"}), path=wf) + with mock.patch.object(self.cc.subprocess, "Popen", self.popen), \ + mock.patch.object(self.cc, "system_config", return_value=None): + self.assertEqual(self.cc._handle(self.client, {"hook_event_name": "SessionEnd", "session_id": "s1", + "transcript_path": self.path, "reason": "other"}, + "SessionEnd", "s1", None), 0) + self.assertEqual(len(self.started), 2) + self.tailer() + found, lost, exchanges = self.finish() + self.assertEqual(found, [("reconcile.hook_missing", "toolu_w"), ("reconcile.hook_missing", "toolu_x")]) + self.assertEqual(sorted(exchanges), ["msg_1", "msg_s", "msg_toolu_w", "msg_toolu_x", "u-3"]) + return lost + + def test_session_end_right_after_a_park_reads_every_transcript(self): + """The new tailer listed the sub-agents' transcripts only once the main one was open: on a first wake that was + also its last it read the main one alone, and a sub-agent's call no hook decided was no hook_missing.""" + self.assertEqual(self.session_end_after(self.park), []) + + def test_session_end_right_after_a_kill_reads_every_transcript(self): + lost = self.session_end_after(lambda: self.tailer(then=Killed)) + self.assertEqual(len(lost), 1) + self.assertIn("the lines written meanwhile are recorded late", lost[0]) + + def test_a_tailer_killed_more_than_restarts_times_is_not_started_again(self): + self.write(PROMPT, TOOL_USE) + self.hook(start="startup") + with mock.patch.object(self.cc, "RESTARTS", 2): + for i in range(3): + self.tailer(then=Killed) + self.write(use_line(f"toolu_{i}0", "Read", {"file_path": f"{i}.txt"})) + self.decide(f"toolu_{i}0", "Read", {"file_path": f"{i}.txt"}) # finds it stopped + self.assertEqual(len(self.started), 3) # its first, and 2 restarts + _, lost, _ = self.finish() + self.assertEqual(len(lost), 3) + self.assertIn("(restart 2 of at most 2)", lost[1]) + self.assertIn("it is not started again (it stopped 3 times in this run)", lost[2]) + def test_a_killed_tailer_is_started_again_from_where_it_stopped_after_a_signed_gap(self): """In dev mode nothing watched the tailer: once it was killed (or failed before it could say why), the run's hook never started another, and every call after it was a reconcile.fabricated gap (about 34,000 of 69,335 in diff --git a/tracekit/integrations/claude_code.py b/tracekit/integrations/claude_code.py index 223639b12..d8169f84a 100644 --- a/tracekit/integrations/claude_code.py +++ b/tracekit/integrations/claude_code.py @@ -486,27 +486,35 @@ def _start(client, st, argv, handle, held=None): def _restart(client, path, st): """Start the run's dev-mode tailer again once it no longer holds its lock, where its progress log says it got (the bytes it read checked by their digests): after a signed tailer_lost gap when it stopped without a word, at most - RESTARTS times a run, or with none when it parked itself (tracekit.tailer: idle). A run already final gets - neither.""" + RESTARTS times a run, or with none when it parked itself (tracekit.tailer: idle). None is started for a run the + hook found ended, nor once the signer refuses the gap (the run is final); a parked one's run may have closed + unknown to the hook: the new tailer then exits at its first record, which the signer refuses.""" lock, log = _tailer_files(path, st["run_id"]) held = _free(lock) if held is None: return + if (system_config() or {}).get("signer"): # system mode: the signer watches its tailer, not a file of the agent's + held.close() + return from tracekit.tailer import resume # only here: it loads the signer's module offsets, digests, parked = resume(log) + # lean: with no progress log (removed) a new tailer reads on from where the hook last saw the transcripts, so what + # the stopped one had not read by then is skipped; keep the run's start offsets in the state if that happens handle = {"run_id": st["run_id"], "run_token": st["run_token"], "path": st["transcript"], "offsets": offsets or st.get("offsets") or {}, "digests": digests, "until": path, "lock": lock, "progress": log} n = st.get("restarts", 0) - again = parked or n < RESTARTS + why = ("the run ended" if st.get("ended") else None if parked or n < RESTARTS else + f"it stopped {n + 1} times in this run") + again = why is None try: if not parked: RunHandle(client, st).call("tailer_lost", offset=handle["offsets"].get(st["transcript"], 0), reason=( "the transcript tailer stopped without a word (killed, or it failed before it could say why); " + ( f"a new one reads on from where it stopped (restart {n + 1} of at most {RESTARTS}): the lines " "written meanwhile are recorded late" if again else - f"it stopped {n + 1} times in this run and is not started again: what the transcript says after " - "that is not recorded: read the transcript itself for it"))) + f"it is not started again ({why}): what the transcript says after that is not recorded: read " + "the transcript itself for it"))) except RPCError as e: if e.code not in GONE: held.close() @@ -1131,7 +1139,8 @@ def _handle(client, p, event, sid, tid): if e.code not in GONE: # closed already (idle), or a signer reset: only the state is left raise for f in glob.glob(glob.escape(_state(sid)[:-len(".json")]) + "*"): - os.remove(f) + with contextlib.suppress(FileNotFoundError): # a tailer that ended removes its own lock and log + os.remove(f) return 0 transcript, prompt = _transcript(p), p.get("prompt") if event == "UserPromptSubmit" else None if event == "SessionStart": diff --git a/tracekit/tailer.py b/tracekit/tailer.py index 8ede2d6a6..51c8bb992 100644 --- a/tracekit/tailer.py +++ b/tracekit/tailer.py @@ -362,6 +362,8 @@ def note(t): while True: gone = until is not None and not os.path.exists(until) cur = main + if main.f is None: # opened before its subagents are listed: a first wake can be the last (SessionEnd) + _verify(main, uid) for p in _subagents(path) if main.f is not None else (): ts.setdefault(p, _Transcript(p, os.path.basename(p), start.get(p, 0), sums.get(p))) for cur in ts.values() if gone else (): # the last wake checks every transcript before a line can be