Skip to content

feat(agent): add use_app to click and type in the agent's own copy of an app - #3275

Merged
dylan-savage merged 10 commits into
feat/rocket-agent-browserfrom
feat/rocket-agent-use-app
Nov 16, 2026
Merged

dylan-savage merged 10 commits into
feat/rocket-agent-browserfrom
feat/rocket-agent-use-app

Conversation

@dylan-savage

Copy link
Copy Markdown
Collaborator

Summary

The browser agent gets a use_app tool, so it can try the app it builds the way a person would. It opens its own hidden copy of the app, reads the page as a text outline with refs (e1, e2, ...), and acts on those refs. It can click, type, pick an option, press a key, and wait for text. Before this, the agent could check only that an app builds, not that a button or form it changed works.

This PR merges into feat/rocket-agent-browser, not develop.

  • The agent works in its own copy. Each conversation gets a hidden preview for the turn, and the preview closes when the turn ends. The user's own preview is never driven. One call runs up to 20 actions in order and stops at the first failure.
  • Clicking and typing ask first. A read or a wait runs without a card. A call that clicks, types, selects, or presses shows a "Use {app}" card with Always allow using your apps. That grant has its own row in Settings. Don't ask mode runs the call without a card, the same as the other acting tools.
  • The user can watch. While the agent acts, its copy covers the Design preview under a banner with Back to your preview. The chat tab shows a small live view.
  • The page answers dialogs itself. It answers confirm() with OK, cancels prompt(), notes alerts, and holds new windows. Every hidden preview does this, so an app that alerts on load no longer blocks the builder during check_app. When the page closes, it stops the pipelines the app started.
  • check_app borrows the turn's page when use_app has one open, so the agent does not boot a second copy. The finish rule does not change.
  • One list defines the approval grants. The GRANTS list and the GRANT_SETTINGS record make the types, the Settings rows, and the comparisons. Before, samePrefs named each grant by hand, so a new grant that was turned on looked like no change.

Type

Feature

Scope

  • The model reads text only. There are no screenshots and no JS eval.
  • On a host with no preview, such as the eval host, use_app tells the model it is not available there and that check_app still checks the build.
  • The chat live view is small, about 270 by 180 pixels. Its sizes are a first cut.
  • The Settings window does not save checkbox changes today, for the older grant rows as well. So the user cannot turn off Always allow using your apps from Settings. That bug is separate from this PR.

Blast radius

The riskiest change is in the builder's preview sessions. Hidden previews no longer join the page-wide session set, and a session gets settled(). check_app and the Design preview both use these sessions. The other changes stay in the agent module, the agent chat, and a cover in the Design view. use_app can click buttons that run the user's real pipelines. Every such call asks first, unless the user chose Always allow or Don't ask.

Testing

  • The shared suite passes, 342/342, agent tests included. The rocket-ui tests pass, 2/2. Typecheck is clean. Eslint shows only the 4 existing warnings in DesignView.tsx. A new test sends a native use_app call through runAgent and checks that the page gets its actions.
  • Builder end to end in the real Pipeline Builder, with a local engine, Claude Sonnet 5.5, Ask first, and a small diary app. All 4 turns finished.
    • The agent added a Clear all button and an entry count. It then typed entries, clicked, and saw the list and the count change.
    • A card showed for each acting call. After Always allow, no more cards showed.
    • The Design cover and the chat live view showed, and the copy closed at the end of the turn. check_app passed.
  • Before native tool calling, the same run finished only 2 of 4 turns, because Sonnet sometimes answered with tool-call markup instead of the JSON plan. On native tool calling, no turn ended that way.
  • Not tested live: the confirm() and prompt() dialogs, a link that leaves the app, Stop during a call, a pipeline that starts on mount, and an OpenAI model.
  • ./builder test in full: not run.

Checklist

  • Commit messages follow conventional commits
  • No secrets or credentials included. gitleaks passes in the pre-commit hook.
  • Wiki updated (if applicable): n/a
  • Breaking changes documented (if applicable): none. openAppPage is optional on AgentHost, and check_app takes an optional lease.

Linked Issue

None.

🤖 Generated with Claude Code

dylan-savage and others added 10 commits November 15, 2026 17:09
…ettled()

check_app's hidden previews joined the page-wide session set, so
findDevSession could hand one to another conversation's check, and
list_workspace marked an app "open" while a check ran. A session built
with { hidden: true } now stays out of the set.

settled() resolves once the preview shows the latest work: no file
change being adopted, no build running or queued, and the shell has
loaded the newest link (or the build failed). Three counters close the
gaps the watch feed leaves: the time between a write and `building`,
the hot path, and "ok" firing before the shell loads the descriptor.
revision counts links and hot swaps. Nothing calls either yet; use_app
will.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AgentHost gains an optional openAppPage(app, { lifetime, conversationId }),
which returns an AppPage. Only domain types cross the seam: PageAction in,
and a PageUse of outcomes, outline lines and page problems out. use()
throws PageGoneError once the page has closed or left the app.

rocket-ui implements it in appPage.ts. Each page boots its own hidden
preview and closes when `lifetime` aborts, including mid-boot. A use
takes the page lock, waits for settled() (20 s at most) and for the DOM
to go quiet, then reads the outline. A failed build or a page not yet
loaded comes back as a not_ready problem, and console errors and crashes
from the use come back as problems too. A use that runs out of time
waiting for the lock returns "the page was busy" instead of throwing.

pageDom.ts reads the whole document in the frame's realm: visible
elements by role with in-house accessible names, values and states, the
text outside elements that act, and refs (e1, e2, ...) that stay stable
for the page's life. The sentences the page writes for the model live in
pageText.ts.

Hidden preview frames now carry data-rr-agent-preview, so the eval
runner's blob-iframe query can skip them. Nothing opens a page until
use_app lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The agent gets a use_app tool that opens its own hidden copy of an app
for the turn and returns the page's errors and outline with refs. It
reads only; actions come next.

- appUse.ts: the toolkit. It takes the app by folder only, keeps one
  page per app for the turn (racing calls share one open, a failed
  open is retried, nothing opens after the turn ends), gives each call
  a 120 s deadline, hides secrets before it cuts text, and trims its
  view to SUMMARY_BUDGET counted as JSON, with peek offsets at cuts.
  A host without app previews is refused in prepare, before any card.
- run.ts mounts it on runEnded, so the existing finally closes every page.
- loop.ts: use_app holds apps/<folder> in writePaths, so same-wave
  edits of that app and other use_app calls on it run in planned order.
- approval.ts: TOOL_LIST.use_app asks with its card unless the call
  only reads or waits; the card text sits in named constants.
- summary.ts exports SUMMARY_HARD_CAP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
use_app now takes up to 20 actions and runs them in order on the agent's
own copy of the app, stopping at the first failure. Every call that
clicks, types, selects or presses asks first; a read or a wait stays free.

- The page answers confirm() with OK, cancels prompt(), notes alerts, and
  holds window.open and target=_blank links, noting each on the action.
- A failed build acts on nothing: every action is not run, with the
  build's failure as a not_ready problem.
- A link that leaves the app fails that action and closes the page; the
  next call drops it and opens a fresh copy, once, inside its deadline.
- The pipelines the app starts are recorded from shell:devReady (so a
  run started on mount counts), re-armed on every reload, skipping reused
  runs, and stopped within 3 s when the page closes.
- settled() waits for the shell to re-attach after a reload instead of
  reading the half-loaded page as settled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
use_app's card now offers Always allow, backed by a new appUse grant and
its own setting row ("Let the agent click and type in its own copy of
your apps without asking"). Auto runs a click without a card, as before.

The grants are now data. host.ts holds one GRANTS list; Grant, grantsFrom
and every per-grant loop come from it. toolSettings.ts holds one
GRANT_SETTINGS record typed by Grant, which generates the setting rows
and APPROVAL_SETTING_KEYS. rocket-ui reads those keys instead of keeping
its own list, and samePrefs compares over GRANTS. Before, samePrefs named
each grant by hand, so a third grant turned on read as no change: runs
kept asking and a waiting card was never approved.

The model setting moves from order 13 to 14 so the grants stay together.
The Ask texts in Settings and on the chat switch now name clicking and
typing in an app.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the user has no preview open, check_app now builds in the copy of
the app use_app opened this turn, instead of booting a second hidden
preview. The app boots once per turn, and the check builds the copy the
agent goes on using.

check_app gets an optional lease option; run.ts passes use_app's
leaseForCheck, which tries the user's attached preview, then the turn's
page (AppPage.lend, after its boot and any use under way), then the
host's own preview. A page that left the app can't lend, so the host
boots one for the check. lend takes the page's lock like a use does: a
stopped run while the check waits returns ok:false, and dialogs the app
raises during the check show on the next use. agentDevSession is
exported from agentHost.ts for the lent session. check_app's finish rule
is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user now sees the hidden preview use_app acts on. It shows over the
Design preview, with a banner and "Back to your preview", or at the top
of the chat tab, with Larger/Smaller. It is watch-only, and each target
is outlined for 300 ms before its action. It closes when the turn ends.

- watch.ts holds the copies, the places the UI lends and the
  dismissals. Placement is derived on each animation frame. Design
  places go first, and a place with no size or with something drawn
  over its centre shows nothing. The iframe never moves in the DOM;
  only its cssText changes (fixed, 1200x800 scaled to fit and never
  up, pointer-events:none).
- AgentWatchSlot renders both places. IAppBuilderHost.agentWatch feeds
  DesignView and AgentHost.watchFor feeds the chat.
- A chat leaves out a copy a Design area already shows, so its label
  always matches what its stage shows.
- appPage publishes the copy at open, touches it on each use, outlines
  each ref'd target before acting and finds it again after, and
  retracts it first on close.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
use_app waited for quiet only after act() returned. A plain (non-React)
handler changes the DOM synchronously during dispatchEvent, before the
MutationObserver existed, so the action read "no change" though the
page had changed. React's commit lands in a microtask afterwards, which
is why React apps were unaffected.

step() now starts untilQuiet before act(). In headless Chromium, with
a plain onclick that appends a list item, act-then-watch reported
changed:false and watch-then-act reports changed:true.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only use_app's page answered native dialogs. When check_app booted its
own hidden preview (no use_app page this turn), an app that alerts or
confirms on load opened a real dialog. The frame is same-origin in the
user's tab, so that blocked the whole builder.

bootHiddenPreview now answers dialogs itself (pageDom answerDialogs) on
every shell:devReady, so each load and reload is covered, for check_app
and use_app alike. An onDialog option hears what each dialog said;
use_app's DialogLog listens through it, so its dialog notes are
unchanged. The header no longer says the user never sees the frame.

In headless Chromium, with the session and handshake stubbed and an
app that alerts and confirms after devReady, the old code opened 8
native dialogs over two boots and reloads; the new code opens none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under native tool calling, use_app's schema goes to the model as a tool
definition and the model's call comes back as structured arguments. One
turn through runAgent checks both ends: the request carries use_app's
schema whole, nested action items included, and a native call's
actions array reaches the page as typed actions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review: 2

  • develop
  • release/.*

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions github-actions Bot added module:ui Chat UI and Dropper UI tests labels Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor
🤖 Internal: Discord sync marker

Auto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete.

@dylan-savage
dylan-savage merged commit 2267745 into feat/rocket-agent-browser Nov 16, 2026
4 checks passed
@dylan-savage
dylan-savage deleted the feat/rocket-agent-use-app branch November 16, 2026 03:26
dylan-savage added a commit that referenced this pull request Nov 18, 2026
#3275 typed toolSettings.ts's grant rows with `import type { Grant } from
'../modules/agent/host'`. The VS Code extension compiles toolSettings.ts
too, and host.ts pulls in the rest of the agent (turn.ts, the toolkits,
wave/session), which reaches the shell's browser and React code. That put
the agent and the shell into the extension's Node-only type check
(node16 resolution, no DOM lib, no JSX), so vscode:compile-typescript has
failed with about 470 errors on every OS since.

GRANTS and Grant now live in agent/grants.ts, which imports nothing.
toolSettings.ts reads Grant from there, and host.ts re-exports both, so
no other importer changes. The extension's program is back to the shell
files it had before #3275, plus grants.ts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:ui Chat UI and Dropper UI tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant