Repository navigation
Browser Use 1.0: gate every function tool at BrowserLoop.wrap_tool_execute (#724) - #805
Conversation
browser-use 1.0 removed the action registry that trace_tools wrapped, so trace_tools() raised AttributeError and instrument() crashed while constructing Tools. Probe which seam Tools has, and on 1.0 raise a RuntimeError naming the tested range instead. instrument() reports "not wired" with that range, so a 1.0 user is told the truth before the seam lands in the next commit. Signed-off-by: Koloyaaa <yakamozzz@sjtu.edu.cn>
…#724) 1.0 runs actions as Pydantic AI tools through BrowserLoop, which sees a call's arguments but not the page. Wrap BrowserLoop.wrap_tool_execute once per process so every function tool, the python cell included, is decided as browser_use:<tool> before it runs and recorded after, with 0.13's masking and typed-secret digests. instrument() leaves Browser Use's internal agent to this integration, so a call is decided once. Test the real 1.0 registry and say in the docs and docs/limits.md what a python decision reads (the cell's source only), that page_url is empty on 1.0, and that the seam is one run per process. Closes Cygnux-Labs#724 Signed-off-by: Koloyaaa <yakamozzz@sjtu.edu.cn>
|
I ran this head (0e547a6) in a sealed container against browser-use 1.0.3 (pydantic-ai-slim 2.66.0) and 0.13.11. What holds: a denied custom action or What needs changing before merge:
Smaller:
What I ran: |
|
Thanks. I reproduced both the finish_from_repl gap and the param_model failure at this head, and I agree with the split. This PR takes points 1, 3, 4 and 5, plus the doc and docstring fixes. finish_from_repl gets a decide before it runs, as browser_use:finish_from_repl with its expression as args, and a complete after. I gate only the 'eval' op at Repl.call: the 'exec' op there is the python cell that wrap_tool_execute already decides, so gating both would decide one call twice. Args go through pydantic_core.to_jsonable_python before masking, with a test for a param_model action. For tests: a 1.0 driver for tests/adapter_contract.py (deny, ask, T2, signer down, complete), an instrument() test on 1.0, addCleanup for the BrowserLoop patch, and the stand-in gets act so the except ImportError can go. The T2 digest difference (call_args on 1.0, the signer's copy on 0.13) I'll settle with that contract run instead of picking one by argument. Docs: "browser-use 1.x" instead of (1.0), the module docstring, the cell error status, and a real tracekit last output in the PR body. Point 2 (reading a cell's source with ast) becomes the separate second PR you suggested. The shape you describe is the one I'd write: helper calls resolved to the rule their 0.13 action has; code and command calls read by the shell/code rules when the argument is literal and held when it is not; unparseable cells, or names built at run time, held; and the cell decided at its worst sub-decision. The same reading would apply to finish_from_repl's expression, which this PR records as source but does not yet read. If gating only 'eval' at Repl.call, or the split itself, differs from what you had in mind, say so before I write it. |
|
The split is right, and so is the plan for points 1 and 3-5. On gating only It holds only while that stays true, though. A later version that adds another Everything else as you wrote it. The |
…abs#724) finish_from_repl is an output tool, so it reaches no tool hook and ran unrecorded. Gate Repl.call('eval', ...) and decide it as browser_use:finish_from_repl; a refusal is a ModelRetry the model can answer by delivering another way. Gate 'exec' as well: the python tool marks the cell it already decided in a context variable, so each cell is decided once, and an exec from any other caller is decided as browser_use:python and refused with tracekit.Refused rather than run. Send action args as JSON (pydantic_core.to_jsonable_python) before masking, so a 0.x param_model action no longer reaches the signer as a Python object, and complete a python cell that raised as error. Both 1.0 completions digest the args the signer decided, its copy under a T2 approval: the old call_args digest made the signer reject the completion with args_mismatch. Add a 1.0 contract driver and tests for the delivery, a direct exec, a param_model action and instrument(); restore both class patches after a test; drop the autowire ImportError fallback. Docs and limits.md state the seams, the error status and what a 1.0 decision reads. Signed-off-by: Koloyaaa <yakamozzz@sjtu.edu.cn>
|
Both ops are gated in the third commit.
I added the test you asked for. It calls The other changes are the ones in my earlier reply:
|
|
Thanks for the third commit. Both seams hold: every 1.0 cell and delivery I threw at it was decided before it ran. What I ran at 3256ed6, in a sealed container (browser-use 1.0.3 with pydantic-ai-slim 2.66.0, and 0.13.11), through Browser Use's own
Points 1, 3 and 5 are fixed, and so are the docstring, the
Smaller: the PR body says the 1.0 contract driver covers "signer down". The contract has no signer-down case. That's the separate |
…Labs#724) instrument() left Browser Use's internal agent to this integration whenever its deps_type looked like Browser Use's, even when the 1.0 seam had not been wired; on a failed _wire_1x that agent then had no decision at all. Skip the Pydantic AI capability only while BrowserLoop's wrap_tool_execute carries tracekit_actions, and add it otherwise, because nothing else decides that agent's calls. The test that pinned this branch never ran: it looked up the pydantic-ai distribution, which is pydantic-ai-slim under browser-use. It now checks the pydantic_ai import and fakes a wired seam, so it runs without browser-use installed. A second test fails _wire_1x and asserts the capability is still added. Cover Repl.call's failure paths: a signer-down eval raises ModelRetry and a signer-down exec raises Refused, neither reaching the worker; an exec of other code under a decided cell is decided as browser_use:python; a T2 approval runs the signer's copy and completes with its digest. The docs say the 1.0 contract runs nightly, not on every push, and that on 1.x the shipped packs allow every cell and delivery until a source reader lands, held by a rule on browser_use:python or browser_use:finish_from_repl. Drop the TK-B011 "every input asks" claim and the page_url typing advice for 1.x. Signed-off-by: Koloyaaa <yakamozzz@sjtu.edu.cn>
|
All six are in the fourth commit.
You were right that
The PR body no longer says the contract driver covers signer down. That is On the new head: 106 passed, 47 skipped, 33 subtests on browser-use 1.0.3, and 81 passed, 72 skipped, 33 subtests on 0.13.11, over |
|
Thanks for the quick and thorough fourth commit. All four items are fixed, and from my side this is ready. What I ran at 2e946d2, in a sealed container (browser-use 1.0.3 with pydantic-ai-slim 2.66.0, and 0.13.11):
The end-to-end run through Browser Use's own
Tests:
The |
|
Merged, thanks @Koloyaaa. Browser Use 1.0 is now gated at |
A job in its own environment (browser-use 1.0.x needs pydantic-ai-slim 2.66; .[dev] pins <2.58) runs tests/test_contract_browser_use.py and tests/test_instrument.py, so #805's 1.0 tests gate merges instead of only the nightly @latest run. The doc says where each contract runs. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
What and why
browser-use 1.0 removed the 0.13 action registry (
Tools.registry.execute_action) thattrace_toolswrapped. On1.0,
trace_tools(Tools(), ...)raisedAttributeError: 'Tools' object has no attribute 'execute_action', andinstrument()crashed while constructingTools; an agent that let Browser Use build its own tools ran with nodecision at all.
This adds a 1.0 seam.
_seam()probes which API the installedToolshas; on 1.0,trace_tools()andinstrument()wrapBrowserLoop.wrap_tool_executeonce per process, so every function tool is decided by the signerbefore it runs, recorded as
browser_use:<tool>after, and masked with the same secret handling as 0.13. The 0.13path is unchanged.
instrument()also stops adding Tracekit's generic Pydantic AI capability to Browser Use'sinternal agent while the 1.0 seam is wired, so a call on 1.0 is decided once, under a
browser_use:name. If thewiring failed, the capability stays, because nothing else decides that agent's calls.
Repl.callis the second seam. Both ops it runs are decided there:evalisfinish_from_repl's delivery, an output tool, which Pydantic AI runs without tool hooks. It is decided asbrowser_use:finish_from_replwith itsexpression, and a refusal is aModelRetrythe model can answer bydelivering another way.
execis apythoncell. The cell's tool call is already decided bywrap_tool_execute, which marks that cell'scode in a context variable, so its
execruns without a second decision. Anexecfrom anywhere else is decided asbrowser_use:pythonand refused withtracekit.Refusedunder a denying rule, rather than run. Today the python toolis the only caller of
exec; the gate holds if a later release adds another.Changes since the first review:
finish_from_replis decided and completed (point 1).Repl.calldecides both ops, and a python cell is still decided once (the follow-up you sent).param_modelused to be refused with "the decide request is notJSON: Object of type Item is not JSON serializable";
pydantic_core.to_jsonable_pythonruns before masking (point3).
pythoncell that raises is completed aserror: Browser Use returns the traceback as the cell's text, so thecompletion carries the text and the error status (smaller item).
showed the 1.0 path was wrong before: under T2 it digested the pre-approval args and the signer rejected the
completion with
args_mismatch(point 5).tests/adapter_contract.py) now run on 1.0 throughDriver1xintests/test_contract_browser_use.py,OnRealTools1xrestores both class patches, theautowirestand-in has0.13's
Tools.act, and theexcept ImportErrorfallback is gone: on a real install a missingbrowser_use.agent.loopreports "not wired" (point 4). Aninstrument()test covers 1.0, and another one keepsTracekit's Pydantic AI capability off Browser Use's agent.
docs/limits.mdsays "browser-use 1.x" instead of a(1.0)prefix that read as Tracekit 1.0, and the moduledocstring names both versions.
Changes since the second review:
instrument()skips the Pydantic AI capability only while theBrowserLoopseam is actually wired, meaningBrowserLoop.wrap_tool_executecarriestracekit_actions, which_wire_1xsets on success. If the wiring fails,the capability is added, so that agent's calls are still decided.
pydantic-aidistribution, andbrowser-use installs
pydantic-ai-slim. It now checksimportlib.util.find_spec("pydantic_ai")and fakes a wiredseam, so it runs without browser-use installed. A second test fails
_wire_1xand asserts the capability is stilladded.
OnRealTools1xcovers theRepl.callfailure paths: a signer-downevalraisesModelRetryand a signer-downexecraisesRefused, neither reaching the worker; anexecof other code under a decided cell is decided asbrowser_use:python; and a T2 approval throughRepl.callruns the signer's copy and completes with its digest.until a source reader lands, and a rule on
browser_use:pythonorbrowser_use:finish_from_replholds them. TheTK-B011"every input asks" claim and thepage_urltyping advice for 1.x are gone..github/workflows/isuntouched.
Fixes #724.
How I ran it as a user
Browser Use 1.0.3, Python 3.12, Windows, local dev signer; a scratch Chrome started with
--remote-debugging-port=9339. A small script (not part of this PR) calledtracekit.instrument(), then builtAgent(task=..., llm=FunctionModel(...), tools=Tools())and ran one short task: one custom action, onepythoncell,then
finish_from_repl.tracekit lastprinted, verbatim:The bundle's records name the three decisions:
browser_use:echo,browser_use:pythonandbrowser_use:finish_from_repl, each followed by itstool.result.Contract tests, in both environments:
pyflakes tracekit tracekit_sdk.py tests eval examplesprints nothing.Checklist
make checkpasses. Not completed locally: on this Windows machine the full suite stops at 8% (the fivefailures before the stop are pre-existing Gemini/
httpxenvironment failures, reproduced at the base commit).Linux CI is the merge gate. (
sdk/typescript/unchanged, somake test-tsdoes not apply.)classes;
OnRealTools1xcovers the tool hook, theRepl.callfailure paths and T2 throughRepl.call;instrument()is covered on both seams, including a failed 1.0 wiring.docs/integrations/browser-use.mdanddocs/limits.mdgit commit -s)Security invariants
execution and
Repl.callthe existing
_ActionspathEvidence format