Repository navigation
feat(agent): add use_app to click and type in the agent's own copy of an app - #3275
Merged
dylan-savage merged 10 commits intoNov 16, 2026
Merged
Conversation
…ettled()
check_app's hidden previews joined the page-wide session set, so
findDevSession could hand one to another conversation's check, and
list_workspace marked an app "open" while a check ran. A session built
with { hidden: true } now stays out of the set.
settled() resolves once the preview shows the latest work: no file
change being adopted, no build running or queued, and the shell has
loaded the newest link (or the build failed). Three counters close the
gaps the watch feed leaves: the time between a write and `building`,
the hot path, and "ok" firing before the shell loads the descriptor.
revision counts links and hot swaps. Nothing calls either yet; use_app
will.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AgentHost gains an optional openAppPage(app, { lifetime, conversationId }),
which returns an AppPage. Only domain types cross the seam: PageAction in,
and a PageUse of outcomes, outline lines and page problems out. use()
throws PageGoneError once the page has closed or left the app.
rocket-ui implements it in appPage.ts. Each page boots its own hidden
preview and closes when `lifetime` aborts, including mid-boot. A use
takes the page lock, waits for settled() (20 s at most) and for the DOM
to go quiet, then reads the outline. A failed build or a page not yet
loaded comes back as a not_ready problem, and console errors and crashes
from the use come back as problems too. A use that runs out of time
waiting for the lock returns "the page was busy" instead of throwing.
pageDom.ts reads the whole document in the frame's realm: visible
elements by role with in-house accessible names, values and states, the
text outside elements that act, and refs (e1, e2, ...) that stay stable
for the page's life. The sentences the page writes for the model live in
pageText.ts.
Hidden preview frames now carry data-rr-agent-preview, so the eval
runner's blob-iframe query can skip them. Nothing opens a page until
use_app lands.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The agent gets a use_app tool that opens its own hidden copy of an app for the turn and returns the page's errors and outline with refs. It reads only; actions come next. - appUse.ts: the toolkit. It takes the app by folder only, keeps one page per app for the turn (racing calls share one open, a failed open is retried, nothing opens after the turn ends), gives each call a 120 s deadline, hides secrets before it cuts text, and trims its view to SUMMARY_BUDGET counted as JSON, with peek offsets at cuts. A host without app previews is refused in prepare, before any card. - run.ts mounts it on runEnded, so the existing finally closes every page. - loop.ts: use_app holds apps/<folder> in writePaths, so same-wave edits of that app and other use_app calls on it run in planned order. - approval.ts: TOOL_LIST.use_app asks with its card unless the call only reads or waits; the card text sits in named constants. - summary.ts exports SUMMARY_HARD_CAP. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
use_app now takes up to 20 actions and runs them in order on the agent's own copy of the app, stopping at the first failure. Every call that clicks, types, selects or presses asks first; a read or a wait stays free. - The page answers confirm() with OK, cancels prompt(), notes alerts, and holds window.open and target=_blank links, noting each on the action. - A failed build acts on nothing: every action is not run, with the build's failure as a not_ready problem. - A link that leaves the app fails that action and closes the page; the next call drops it and opens a fresh copy, once, inside its deadline. - The pipelines the app starts are recorded from shell:devReady (so a run started on mount counts), re-armed on every reload, skipping reused runs, and stopped within 3 s when the page closes. - settled() waits for the shell to re-attach after a reload instead of reading the half-loaded page as settled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
use_app's card now offers Always allow, backed by a new appUse grant and
its own setting row ("Let the agent click and type in its own copy of
your apps without asking"). Auto runs a click without a card, as before.
The grants are now data. host.ts holds one GRANTS list; Grant, grantsFrom
and every per-grant loop come from it. toolSettings.ts holds one
GRANT_SETTINGS record typed by Grant, which generates the setting rows
and APPROVAL_SETTING_KEYS. rocket-ui reads those keys instead of keeping
its own list, and samePrefs compares over GRANTS. Before, samePrefs named
each grant by hand, so a third grant turned on read as no change: runs
kept asking and a waiting card was never approved.
The model setting moves from order 13 to 14 so the grants stay together.
The Ask texts in Settings and on the chat switch now name clicking and
typing in an app.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the user has no preview open, check_app now builds in the copy of the app use_app opened this turn, instead of booting a second hidden preview. The app boots once per turn, and the check builds the copy the agent goes on using. check_app gets an optional lease option; run.ts passes use_app's leaseForCheck, which tries the user's attached preview, then the turn's page (AppPage.lend, after its boot and any use under way), then the host's own preview. A page that left the app can't lend, so the host boots one for the check. lend takes the page's lock like a use does: a stopped run while the check waits returns ok:false, and dialogs the app raises during the check show on the next use. agentDevSession is exported from agentHost.ts for the lent session. check_app's finish rule is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user now sees the hidden preview use_app acts on. It shows over the Design preview, with a banner and "Back to your preview", or at the top of the chat tab, with Larger/Smaller. It is watch-only, and each target is outlined for 300 ms before its action. It closes when the turn ends. - watch.ts holds the copies, the places the UI lends and the dismissals. Placement is derived on each animation frame. Design places go first, and a place with no size or with something drawn over its centre shows nothing. The iframe never moves in the DOM; only its cssText changes (fixed, 1200x800 scaled to fit and never up, pointer-events:none). - AgentWatchSlot renders both places. IAppBuilderHost.agentWatch feeds DesignView and AgentHost.watchFor feeds the chat. - A chat leaves out a copy a Design area already shows, so its label always matches what its stage shows. - appPage publishes the copy at open, touches it on each use, outlines each ref'd target before acting and finds it again after, and retracts it first on close. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
use_app waited for quiet only after act() returned. A plain (non-React) handler changes the DOM synchronously during dispatchEvent, before the MutationObserver existed, so the action read "no change" though the page had changed. React's commit lands in a microtask afterwards, which is why React apps were unaffected. step() now starts untilQuiet before act(). In headless Chromium, with a plain onclick that appends a list item, act-then-watch reported changed:false and watch-then-act reports changed:true. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only use_app's page answered native dialogs. When check_app booted its own hidden preview (no use_app page this turn), an app that alerts or confirms on load opened a real dialog. The frame is same-origin in the user's tab, so that blocked the whole builder. bootHiddenPreview now answers dialogs itself (pageDom answerDialogs) on every shell:devReady, so each load and reload is covered, for check_app and use_app alike. An onDialog option hears what each dialog said; use_app's DialogLog listens through it, so its dialog notes are unchanged. The header no longer says the user never sees the frame. In headless Chromium, with the session and handshake stubbed and an app that alerts and confirms after devReady, the old code opened 8 native dialogs over two boots and reloads; the new code opens none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under native tool calling, use_app's schema goes to the model as a tool definition and the model's call comes back as structured arguments. One turn through runAgent checks both ends: the request carries use_app's schema whole, nested action items included, and a native call's actions array reaches the page as typed actions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dylan-savage
requested review from
Amitabh1998,
Rod-Christensen,
jmaionchi and
stepmikhaylov
as code owners
November 16, 2026 03:23
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review: 2
Please check the settings in the CodeRabbit UI or the
|
Contributor
🤖 Internal: Discord sync markerAuto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete. |
6 of 8 tasks
dylan-savage
added a commit
that referenced
this pull request
Nov 18, 2026
#3275 typed toolSettings.ts's grant rows with `import type { Grant } from '../modules/agent/host'`. The VS Code extension compiles toolSettings.ts too, and host.ts pulls in the rest of the agent (turn.ts, the toolkits, wave/session), which reaches the shell's browser and React code. That put the agent and the shell into the extension's Node-only type check (node16 resolution, no DOM lib, no JSX), so vscode:compile-typescript has failed with about 470 errors on every OS since. GRANTS and Grant now live in agent/grants.ts, which imports nothing. toolSettings.ts reads Grant from there, and host.ts re-exports both, so no other importer changes. The extension's program is back to the shell files it had before #3275, plus grants.ts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The browser agent gets a
use_apptool, so it can try the app it builds the way a person would. It opens its own hidden copy of the app, reads the page as a text outline with refs (e1,e2, ...), and acts on those refs. It can click, type, pick an option, press a key, and wait for text. Before this, the agent could check only that an app builds, not that a button or form it changed works.This PR merges into
feat/rocket-agent-browser, notdevelop.confirm()with OK, cancelsprompt(), notes alerts, and holds new windows. Every hidden preview does this, so an app that alerts on load no longer blocks the builder duringcheck_app. When the page closes, it stops the pipelines the app started.check_appborrows the turn's page whenuse_apphas one open, so the agent does not boot a second copy. The finish rule does not change.GRANTSlist and theGRANT_SETTINGSrecord make the types, the Settings rows, and the comparisons. Before,samePrefsnamed each grant by hand, so a new grant that was turned on looked like no change.Type
Feature
Scope
use_apptells the model it is not available there and thatcheck_appstill checks the build.Blast radius
The riskiest change is in the builder's preview sessions. Hidden previews no longer join the page-wide session set, and a session gets
settled().check_appand the Design preview both use these sessions. The other changes stay in the agent module, the agent chat, and a cover in the Design view.use_appcan click buttons that run the user's real pipelines. Every such call asks first, unless the user chose Always allow or Don't ask.Testing
DesignView.tsx. A new test sends a nativeuse_appcall throughrunAgentand checks that the page gets its actions.check_apppassed.confirm()andprompt()dialogs, a link that leaves the app, Stop during a call, a pipeline that starts on mount, and an OpenAI model../builder testin full: not run.Checklist
openAppPageis optional onAgentHost, andcheck_apptakes an optional lease.Linked Issue
None.
🤖 Generated with Claude Code