Repository navigation
Description
A follow-up from the review of #837. Line numbers are at afa55b1, the head of #837.
What happens now
When a write call loses its connection, the SDK sends it again with the same request_id (an event call also keeps
its stream and client_seq). If that retry fails too, the SDK reports the call as not recorded. But the first try
may have been written, and in some cases it certainly was.
RunHandle.complete warns the result of <id> was not recorded: <error> for every SignerUnavailable and RPCError
(tracekit/sdk/client.py:726-729; TypeScript sdk/typescript/src/v2/client.ts:690-693). observe, state_write,
model_event and the other write calls raise the same errors, and nothing in them says whether a record exists.
Three cases, each reproduced at afa55b1 with a fake signer (a Unix socket, or _Https.post patched as the
Transports.https test helper does):
- The signer dies after writing. It reads
complete, writes it, and exits before it answers. The retry can't
reconnect. The warning isthe result of call-1 was not recorded: no signer answering at /tmp/.../c.sock: [Errno 2] No such file or directory. The record may exist. - The signer restarts after writing. The connection drops after the write, and the signer restarts. Its
request_idcache lives in memory only (tracekit/signer/pipeline.py:757-761), so the retry is refused
client_seq_reused. The warning isthe result of call-1 was not recorded: client_seq_reused: <stream>/0, in Python
and in TypeScript. The record exists: the signer refuses aclient_seqonly when it is at or below the last one it
recorded for that stream (pipeline.py:786-788). - The signer answers
unavailable. That is a storage write whose outcome it couldn't tell
(pipeline.py:848-853). The retry Signer HTTP API: statuses and Request-Id, TRACEKIT_API_KEY, SDK retries of event calls, --signer help #837 adds is then refusedclient_seq_reused, andcallraises the first
unavailable(client.py:382-385,client.ts:477).completewarnswas not recorded: unavailable: storage refused writes. The record exists, for the same reason as in case 2.
Two more paths end as "was not recorded" when the call may have been written: three lost connections
(client.py:394-395, client.ts:487), and a timeout after the request went out (client.py:372-374, and
_Https.post at client.py:235-237).
Someone reading these warnings believes a tool result is missing from the evidence, when it may be there or is there.
What should happen
The SDK says what it knows, in one of three plain forms, and says what to do next.
- Not recorded. Use this when the signer refused the call and no earlier try of it may have been written
(invalid_request,run_closed,unknown_tool_call,args_mismatch,forbidden, a size limit). Use it too when
nothing was sent (no signer at the first connect, a token file that can't be read, the run's previous event still
unanswered).completekeeps today's wording and adds the next step:the result of call-1 was not recorded: <error>. Once the run closes, its bundle shows call-1 as decided and not completed. - May not have been recorded. Use this when a try was sent and got no answer (a lost connection, a timeout), or the
signer answeredunavailable, and no later try settled it.completewarns:the result of call-1 may not have been recorded (<error>). Once the run closes, its bundle has call-1's tool.result if it was, or a decided_not_completed gap if it wasn't.The other write calls raise the same exception as today, and its message adds the same sentence.
It also says not to send the call again as a new one: if the first try landed, the call would be recorded twice. - Recorded. Use this when an event call's retry (
decide,complete,observe,state_write,model_event) is
refusedclient_seq_reusedafter a try that may have been written, and the SDK assigned thatclient_seq(the
caller didn't pass it in). The earlier try was recorded and only its answer was lost.completegives no "not
recorded" warning. It returns None, because there is no answer to return.decidekeeps Signer HTTP API: statuses and Request-Id, TRACEKIT_API_KEY, SDK retries of event calls, --signer help #837's behaviour: it raises
theunavailable, so the caller takes its outage path, because the decision itself is unknown.
Nothing changes about what is raised or when. The fail-mode paths (decide catching SignerUnavailable) stay as they
are. Only the wording changes, plus a field on the error that says which of the three cases it is.
Where to change it
tracekit/sdk/client.py,Client.call(lines 348-395). Note when a try of a write call may have reached the
signer. A write call is one whose request has arequest_id(line 351). A try may have reached the signer on a
_ConnectionLost, on the timeout at line 372, on theSignerUnavailablethat_Https.postraises for a timeout,
or on anunavailableanswer. When the call then raises, set a field on the exception, for examplee.recorded:
False,None(unknown) orTrue. For the unknown case, add the sentence to its message.RunHandle.complete(lines 714-729): word the warning from that field.sdk/typescript/src/v2/client.ts: make the same change inClient.send(lines 415-491) andRunHandle.complete
(lines 684-696). Add the field toSignerUnavailable(line 50) andRPCError(line 64).- The docstrings of
complete(client.py:715-718,client.ts:684-685): say which three things the warning can say.
Tests to add (each fails at afa55b1)
In tests/test_sdk_client.py:
FakeSigners(a signer on a Unix socket): the signer readscomplete, stops listening and shuts the connection
down. Assert the warning says "may not have been recorded" and points to the bundle. Today it says "was not
recorded". Useshutdown(socket.SHUT_RDWR):close()alone leaves the descriptor open while the socket'smakefile
reader holds it, and the client never sees the drop.FakeSigners: the signer drops the connection on the firstcompleteand answers the retry with{"error": {"code": "client_seq_reused", "message": "<stream>/0"}}. Assert there is no "not recorded" warning, and that
Client.callraises withrecordedset to True. Today the warning is "was not recorded: client_seq_reused".Transports.https: answercompletewith[OUTAGE, REUSED]and assert no "not recorded" warning. Answer it with
client._ConnectionLostthree times and assert "may not have been recorded". Answer it with anargs_mismatcherror
and assert "was not recorded" (unchanged).test_a_retry_refused_as_already_recorded_raises_the_first_tries_outagemust still pass:decidestill raises
unavailable.
In sdk/typescript/test/v2.test.mjs: add the same three complete cases, using https() and fake(). Capture the
warning with process.on("warning", ...). Its name is TracekitWarning.
Out of scope
- When the SDK retries, and what
decidereturns during an outage. decidewhen its retry is refusedclient_seq_reusedafter a lost connection (case 2 above, for a decision). Today
it raises that refusal to the caller, though the decision was recorded and only its verdict is unknown. That needs
its own issue.- The integrations that print their own "not recorded" for an SDK error (
tracekit/integrations/langchain.py:155,
tracekit/integrations/mcp.py:102and:168). They can use the new field in a follow-up.gate.py:116and
crewai.py:77wrapcomplete, which never raises for the signer, so they need no change. - A signer call that looks up whether a record exists.
- The v1 path (tracekitd and
tracekit/agent_sdk.py).
Activity
SDKs: a call on a socket a timeout destroyed sent nothing; docs for r…
SDKs: test a write timed out on the other transport; docs: recorded a…