Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/codemode-standard-schema-types.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@cloudflare/codemode": patch
---

Declare `~standard.types` on the codemode tool's input schema so Standard Schema consumers such as TanStack AI infer `{ code: string }`.
5 changes: 5 additions & 0 deletions .changeset/tanstack-browser-tool.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"agents": patch
---

Add `browserTool` to `agents/browser/tanstack-ai`, the TanStack AI version of the persistent browser tool. See [Persistent browser](https://github.com/cloudflare/agents/blob/main/docs/agents/browse-the-web.md#persistent-browser).
14 changes: 14 additions & 0 deletions docs/agents/browse-the-web.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,20 @@ export class MyAgent extends Think<Env> {

Only Chromium on a Browser Run binding is supported. It needs the same `LOADER` binding and `CodemodeRuntime` export as `createBrowserTools`.

For TanStack AI, import `browserTool` from `agents/browser/tanstack-ai` instead. It takes the same options plus an optional `name` (default `"browser"`) and returns a `ServerTool`:

```ts
import { browserTool } from "agents/browser/tanstack-ai";

const stream = chat({
adapter,
tools: [browserTool({ browser: this.browser, loader: this.env.LOADER })],
messages
});
```

The TanStack AI tool doesn't support screenshots yet. TanStack AI gives the host and the model the same output, so if the model returns a screenshot, the tool replaces it with a note saying it was left out, and the tool's instructions tell the model to read the page with `Runtime.evaluate` instead.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Documentation uses a contraction

The screenshot explanation says “doesn't,” while the documentation style guide requires “does not.”

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.


## Quick Actions (stateless browsing)

`browser_execute` drives a full, stateful CDP session — the right tool for interactive, multi-step automation. But a lot of agent browsing is really one-shot: _read this page as Markdown_, _extract these fields_, _list the links_. For those, [Quick Actions](https://developers.cloudflare.com/browser-run/quick-actions/) are simpler, faster, and cheaper. They need only the `browser` binding — no Durable Object, Worker Loader, or sandbox — so they work from any Worker.
Expand Down
5 changes: 3 additions & 2 deletions packages/agents/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Each export maps to a public entry point that users `import` from. These are the
| `agents/browser` | `src/browser/index.ts` | Browser Run helpers — CDP sessions, connector, Quick Action primitives |
| `agents/browser/ai` | `src/browser/ai.ts` | AI SDK browser tools — `createBrowserTools` (CDP) + `createQuickActionTools` |
| `agents/browser/ai-sdk` | `src/browser/ai-sdk.ts` | AI SDK `browserTool` for a persistent `Browser` (+ `createQuickActionTools` re-export) |
| `agents/browser/tanstack-ai` | `src/browser/tanstack-ai.ts` | TanStack AI browser tool (`browser_execute`) |
| `agents/browser/tanstack-ai` | `src/browser/tanstack-ai.ts` | TanStack AI `browserTool` for a persistent `Browser` (+ legacy `createBrowserTools`) |
| `agents/voice` | `src/voice/index.ts` | Voice server mixins, contracts, Workers AI providers, text, and SFU helpers |
| `agents/voice/types` | `src/voice/types.ts` | Dependency-light Voice protocol and provider contracts |
| `agents/voice/client` | `src/voice/client.ts` | Framework-neutral browser Voice client |
Expand Down Expand Up @@ -147,8 +147,9 @@ src/
quick-actions.ts # Stateless Quick Action primitives (browserMarkdown, …)
ai.ts # createBrowserTools + createQuickActionTools (AI SDK)
ai-sdk.ts # browserTool for a persistent Browser (AI SDK)
browser-tool.ts # harness-neutral browserTool core shared by ai-sdk.ts and tanstack-ai.ts
tool-helpers.ts # model-output + ctx helpers shared by ai.ts and ai-sdk.ts
tanstack-ai.ts # createBrowserTools for TanStack AI
tanstack-ai.ts # browserTool and createBrowserTools for TanStack AI

voice/ # Voice server, client, React, provider, SFU, and text entries
channels/ # Messaging core, provider adapters, and AI framework adapters
Expand Down
153 changes: 17 additions & 136 deletions packages/agents/src/browser/ai-sdk.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,23 +5,22 @@
*/
import type { FlexibleSchema } from "ai";
import {
createCodemodeRuntime,
DynamicWorkerExecutor,
type ProxyToolOutput
} from "@cloudflare/codemode";
import {
BrowserSessionConnector,
type BrowserExecutionReport,
type BrowserNewTab,
type BrowserSource
} from "./session-connector";
browserReportNotes,
createBrowserToolCore,
type BrowserToolInput,
type BrowserToolOptions,
type BrowserToolOutput
} from "./browser-tool";
import {
browserExecuteModelOutput,
resolveCtx,
transformBrowserResult,
type BrowserModelOutput
} from "./tool-helpers";

export type {
BrowserToolInput,
BrowserToolOptions,
BrowserToolOutput
} from "./browser-tool";
export type { BrowserNewTab, BrowserSource } from "./session-connector";

// Stateless Quick Action tools, so one import covers both.
Expand All @@ -31,52 +30,6 @@ export {
type QuickActionToolName
} from "./ai";

export interface BrowserToolOptions {
/**
* The browser the tool drives — usually a `Browser` installed on the
* host's Lifecycle. Tools built from the same `Browser` share it.
*/
browser: BrowserSource;

/**
* WorkerLoader binding for sandboxed code execution.
*
* Requires `"worker_loaders": [{ "binding": "LOADER" }]` in wrangler.jsonc.
*/
loader: WorkerLoader;

/** Sandbox execution timeout in milliseconds. Defaults to 30000 (30s). */
timeoutMs?: number;

/**
* Durable Object state for the codemode runtime facet. Optional inside an
* Agent (resolved via `getCurrentAgent()`); pass it explicitly elsewhere.
*
* The worker must export the `CodemodeRuntime` class (the
* `@cloudflare/codemode/vite` plugin does this automatically, or add
* `export { CodemodeRuntime } from "@cloudflare/codemode"` to your entry).
*/
ctx?: DurableObjectState;
}

/** The model's input to {@link browserTool}: JavaScript to run. */
export interface BrowserToolInput {
code: string;
}

/**
* The result of one {@link browserTool} run: the codemode execution result,
* plus what happened to the browser.
*/
export type BrowserToolOutput = ProxyToolOutput & {
/** The browser was replaced before this run; earlier tabs are gone. */
restarted?: true;
/** A sentence telling the model what `restarted` means for it. */
notice?: string;
/** Tabs the page opened itself (popups, `target=_blank` links). */
newTabs?: BrowserNewTab[];
};

/**
* The AI SDK tool {@link browserTool} returns. Assignable to the AI SDK's
* `Tool`, and `execute` / `toModelOutput` are always present.
Expand All @@ -91,49 +44,6 @@ export interface BrowserTool {
toModelOutput(options: { output: BrowserToolOutput }): BrowserModelOutput;
}

const RESTARTED_NOTICE =
"The browser was restarted before this run (it expired or was closed): earlier tabs, logins, and page state are gone. Your code ran in a fresh browser — navigate again before relying on page state.";

/**
* A codemode runtime name per browser. Browser names are host-chosen and
* unrestricted; runtime names allow only `[a-zA-Z0-9_.-]`, so anything else
* (including `.`, the escape) is hex-escaped.
*/
function browserRuntimeName(name: string): string {
const escaped = name.replace(
/[^a-zA-Z0-9_-]/g,
(char) => `.${char.codePointAt(0)?.toString(16)}.`
);
return `browser-tool_${escaped}`;
}

/** Add what happened to the browser to the codemode result. */
function withBrowserReport(
output: ProxyToolOutput,
report: BrowserExecutionReport | undefined
): BrowserToolOutput {
if (!report) return output;
return {
...output,
...(report.restarted
? { restarted: true as const, notice: RESTARTED_NOTICE }
: {}),
...(report.newTabs.length > 0 ? { newTabs: report.newTabs } : {})
};
}

/** Report sentences that must survive a screenshot's text summary. */
function browserReportNotes(output: BrowserToolOutput): string[] {
const notes: string[] = [];
if (output.notice) notes.push(output.notice);
if (output.newTabs?.length) {
notes.push(
`New tabs opened by the page: ${JSON.stringify(output.newTabs)}`
);
}
return notes;
}

/**
* Create an AI SDK tool that lets the model drive a persistent browser with
* JavaScript and the Chrome DevTools Protocol.
Expand Down Expand Up @@ -170,43 +80,14 @@ function browserReportNotes(output: BrowserToolOutput): string[] {
* ```
*/
export function browserTool(options: BrowserToolOptions): BrowserTool {
const ctx = resolveCtx(options);
if (!ctx) {
throw new Error(
"browserTool requires a Durable Object 'ctx' — pass it explicitly, or call from within an Agent so it can be resolved via getCurrentAgent()"
);
}

const connector = new BrowserSessionConnector(ctx, {
browser: options.browser
});
const runtime = createCodemodeRuntime({
ctx,
executor: new DynamicWorkerExecutor({
loader: options.loader,
timeout: options.timeoutMs
}),
connectors: [connector],
name: browserRuntimeName(options.browser.name),
transformResult: transformBrowserResult
const core = createBrowserToolCore(options, {
screenshotHint:
"Return screenshots as { type: 'browser_screenshot', mediaType: 'image/png', data: screenshot.data }; the UI keeps the image while the model receives a compact summary."
});

const codemodeTool = runtime.tool({
connectorHints: {
cdp: "A persistent browser over CDP — tabs and logins carry over between runs. Use sessionId: \"active\" for page commands. Return screenshots as { type: 'browser_screenshot', mediaType: 'image/png', data: screenshot.data }; the UI keeps the image while the model receives a compact summary."
}
});

return {
description: codemodeTool.description,
inputSchema: codemodeTool.inputSchema,
execute: async (input, executeOptions) => {
const output = await codemodeTool.execute(input, executeOptions);
return withBrowserReport(
output,
connector.takeReport(output.executionId)
);
},
description: core.description,
inputSchema: core.inputSchema,
execute: core.execute,
toModelOutput: ({ output }) =>
browserExecuteModelOutput(output, browserReportNotes(output))
};
Expand Down
Loading
Loading