Replace client-defined tool round-trips with server-side RPC delegation #1414

Background

Working with @cloudflare/ai-chat's client-defined tool flow, I've been tracking a series of small quirks. They're all intermittent and timing/network/model-dependent. Some of what surfaces:

1. After a tool result, the assistant message ends with finishReason: "tool-calls" and no follow-up text is generated. (Filed as Tool part state regresses during client tool round-trip; continuation starts but emits no output #1404, partially addressed in 0.5.4 by making _applyToolResult tolerate output-available as a starting state.)
2. After a tool result, there's a brief window where status flips back to 'ready' and isStreaming/isToolContinuation are both false — between the first continuation stream ending and the next one starting. Anything UI-side gated on those flags briefly snaps to its idle representation.
3. The tool part state itself can be observed transitioning illegally — e.g. output-available → input-streaming → output-available — in chat.messages debug logs. With UIs that render different copy depending on state (e.g. "Preparing…" vs "Done"), users see the regression directly.
4. Continuation responses sometimes appear all at once instead of streaming, as if the SDK briefly thought the turn was done and then had to flush a batch.

All four are intermittent. Cold-start, network jitter, model choice (Anthropic with thinking enabled seems worse), and connection freshness all influence reproducibility. 0.5.4 reduced symptom 1 but the underlying state-machine race appears alive in the others.

The pattern

All four symptoms share a fingerprint: client-tool flows span multiple streamText calls (one to emit the tool call, another after the result). The state machine has to track transitions across that gap, and any race in the gap surfaces as one of the symptoms above.

Server-defined tools have none of these issues. The reason is structural: a server tool's execute runs inside the same streamText call. There is no gap, no transition between streams, no state to reconcile across a pause. One unbroken flow from initial request to final response. The state machine isn't asked to do anything hard.

Proposed solution: client tools as server-side RPC delegation

Make "client-defined tools" behave like server tools from the chat protocol's perspective, while still executing in the browser.

To clarify the difference from today's setup: client tool schemas are already sent to the server (via extractClientToolSchemas/clientTools). The change isn't where tool metadata lives — it's that the server-side tool currently has no execute function, so streamText ends when the model emits the call. In the proposed setup, the server-side tool gets an execute function that delegates to the browser via WS RPC.

Concretely:

1. The tool is registered server-side (just like a normal server tool). Its execute is a thin proxy.
2. When invoked, the proxy generates a requestId, sends a new WS message — e.g. cf_agent_client_tool_invoke { requestId, toolName, input } — and awaits a cf_agent_client_tool_result { requestId, output } from the same connection.
3. The browser receives the invoke message, looks up the user's tool by name, runs the local execute (with full access to browser-side state), and sends back the result.
4. The server's proxy execute resolves with that result, streamText continues without pausing.

From the LLM and from streamText's perspective, it's a server tool. From the consumer's perspective, the tool body still runs in the browser. One unbroken streamText call, no continuation gap, no state to reconcile — the entire class of bugs above goes away. Confirmation/approval continues to work via the existing addToolApprovalResponse mechanism, exactly as for any other server tool.

What this eliminates vs. what's new

Eliminated outright:

• Multi-stream state-machine races (only one stream now)
• The _applyToolResult regression class (no _applyToolResult step exists)
• The isToolContinuation flicker (no continuation as a distinct phase)
• Model-specific oddities around thinking blocks confusing state transitions
• The cold-start resume-wait gap interacting with first-message submits
• The "extra stream" pattern that produces flicker between continuations

New failure modes (simpler, well-understood):

• WebSocket drops mid-invoke → server's execute Promise needs a timeout and error path. streamText would receive an error result, model would respond accordingly.
• Server-side bookkeeping for in-flight calls (Map<requestId, {resolve, reject, timer}>).
• Browser-side execute that throws or hangs is still a problem, but with the same shape as today, addressed by the same timeout pattern.

These are standard async error-handling problems — categorically different from the distributed-state races the current pattern produces.

Suggestion

Once the new pattern is in place and tested, the existing client-tool round-trip mechanism could be deprecated and eventually removed. (I recall this was discussed earlier — at the time I'd argued against deprecation because the round-trip path was the only option. With the RPC pattern in place that argument flips: the round-trip path is strictly worse and adds maintenance burden without unique value.)

Happy to PR a prototype if there's interest in this direction.