Skip to content

SDKs: say "may not have been recorded" when a write's outcome is unknown, not "was not recorded" #846

Open

Description

@LoopGlitch26

A follow-up from the review of #837. Line numbers are at afa55b1, the head of #837.

What happens now

When a write call loses its connection, the SDK sends it again with the same request_id (an event call also keeps
its stream and client_seq). If that retry fails too, the SDK reports the call as not recorded. But the first try
may have been written, and in some cases it certainly was.

RunHandle.complete warns the result of <id> was not recorded: <error> for every SignerUnavailable and RPCError
(tracekit/sdk/client.py:726-729; TypeScript sdk/typescript/src/v2/client.ts:690-693). observe, state_write,
model_event and the other write calls raise the same errors, and nothing in them says whether a record exists.

Three cases, each reproduced at afa55b1 with a fake signer (a Unix socket, or _Https.post patched as the
Transports.https test helper does):

  1. The signer dies after writing. It reads complete, writes it, and exits before it answers. The retry can't
    reconnect. The warning is the result of call-1 was not recorded: no signer answering at /tmp/.../c.sock: [Errno 2] No such file or directory. The record may exist.
  2. The signer restarts after writing. The connection drops after the write, and the signer restarts. Its
    request_id cache lives in memory only (tracekit/signer/pipeline.py:757-761), so the retry is refused
    client_seq_reused. The warning is the result of call-1 was not recorded: client_seq_reused: <stream>/0, in Python
    and in TypeScript. The record exists: the signer refuses a client_seq only when it is at or below the last one it
    recorded for that stream (pipeline.py:786-788).
  3. The signer answers unavailable. That is a storage write whose outcome it couldn't tell
    (pipeline.py:848-853). The retry Signer HTTP API: statuses and Request-Id, TRACEKIT_API_KEY, SDK retries of event calls, --signer help #837 adds is then refused client_seq_reused, and call raises the first
    unavailable (client.py:382-385, client.ts:477). complete warns was not recorded: unavailable: storage refused writes. The record exists, for the same reason as in case 2.

Two more paths end as "was not recorded" when the call may have been written: three lost connections
(client.py:394-395, client.ts:487), and a timeout after the request went out (client.py:372-374, and
_Https.post at client.py:235-237).

Someone reading these warnings believes a tool result is missing from the evidence, when it may be there or is there.

What should happen

The SDK says what it knows, in one of three plain forms, and says what to do next.

  • Not recorded. Use this when the signer refused the call and no earlier try of it may have been written
    (invalid_request, run_closed, unknown_tool_call, args_mismatch, forbidden, a size limit). Use it too when
    nothing was sent (no signer at the first connect, a token file that can't be read, the run's previous event still
    unanswered). complete keeps today's wording and adds the next step: the result of call-1 was not recorded: <error>. Once the run closes, its bundle shows call-1 as decided and not completed.
  • May not have been recorded. Use this when a try was sent and got no answer (a lost connection, a timeout), or the
    signer answered unavailable, and no later try settled it. complete warns: the result of call-1 may not have been recorded (<error>). Once the run closes, its bundle has call-1's tool.result if it was, or a decided_not_completed gap if it wasn't. The other write calls raise the same exception as today, and its message adds the same sentence.
    It also says not to send the call again as a new one: if the first try landed, the call would be recorded twice.
  • Recorded. Use this when an event call's retry (decide, complete, observe, state_write, model_event) is
    refused client_seq_reused after a try that may have been written, and the SDK assigned that client_seq (the
    caller didn't pass it in). The earlier try was recorded and only its answer was lost. complete gives no "not
    recorded" warning. It returns None, because there is no answer to return. decide keeps Signer HTTP API: statuses and Request-Id, TRACEKIT_API_KEY, SDK retries of event calls, --signer help #837's behaviour: it raises
    the unavailable, so the caller takes its outage path, because the decision itself is unknown.

Nothing changes about what is raised or when. The fail-mode paths (decide catching SignerUnavailable) stay as they
are. Only the wording changes, plus a field on the error that says which of the three cases it is.

Where to change it

  • tracekit/sdk/client.py, Client.call (lines 348-395). Note when a try of a write call may have reached the
    signer. A write call is one whose request has a request_id (line 351). A try may have reached the signer on a
    _ConnectionLost, on the timeout at line 372, on the SignerUnavailable that _Https.post raises for a timeout,
    or on an unavailable answer. When the call then raises, set a field on the exception, for example e.recorded:
    False, None (unknown) or True. For the unknown case, add the sentence to its message.
  • RunHandle.complete (lines 714-729): word the warning from that field.
  • sdk/typescript/src/v2/client.ts: make the same change in Client.send (lines 415-491) and RunHandle.complete
    (lines 684-696). Add the field to SignerUnavailable (line 50) and RPCError (line 64).
  • The docstrings of complete (client.py:715-718, client.ts:684-685): say which three things the warning can say.

Tests to add (each fails at afa55b1)

In tests/test_sdk_client.py:

  • FakeSigners (a signer on a Unix socket): the signer reads complete, stops listening and shuts the connection
    down. Assert the warning says "may not have been recorded" and points to the bundle. Today it says "was not
    recorded". Use shutdown(socket.SHUT_RDWR): close() alone leaves the descriptor open while the socket's makefile
    reader holds it, and the client never sees the drop.
  • FakeSigners: the signer drops the connection on the first complete and answers the retry with {"error": {"code": "client_seq_reused", "message": "<stream>/0"}}. Assert there is no "not recorded" warning, and that
    Client.call raises with recorded set to True. Today the warning is "was not recorded: client_seq_reused".
  • Transports.https: answer complete with [OUTAGE, REUSED] and assert no "not recorded" warning. Answer it with
    client._ConnectionLost three times and assert "may not have been recorded". Answer it with an args_mismatch error
    and assert "was not recorded" (unchanged).
  • test_a_retry_refused_as_already_recorded_raises_the_first_tries_outage must still pass: decide still raises
    unavailable.

In sdk/typescript/test/v2.test.mjs: add the same three complete cases, using https() and fake(). Capture the
warning with process.on("warning", ...). Its name is TracekitWarning.

Out of scope

  • When the SDK retries, and what decide returns during an outage.
  • decide when its retry is refused client_seq_reused after a lost connection (case 2 above, for a decision). Today
    it raises that refusal to the caller, though the decision was recorded and only its verdict is unknown. That needs
    its own issue.
  • The integrations that print their own "not recorded" for an SDK error (tracekit/integrations/langchain.py:155,
    tracekit/integrations/mcp.py:102 and :168). They can use the new field in a follow-up. gate.py:116 and
    crewai.py:77 wrap complete, which never raises for the signer, so they need no change.
  • A signer call that looks up whether a record exists.
  • The v1 path (tracekitd and tracekit/agent_sdk.py).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions