Skip to content

feat(macos): add query, a targeted accessibility lookup - #73

Open
hyprcat wants to merge 3 commits into
iFurySt:mainfrom
hyprcat:feat/targeted-ax-query
Open

hyprcat wants to merge 3 commits into
iFurySt:mainfrom
hyprcat:feat/targeted-ax-query

Conversation

@hyprcat

@hyprcat hyprcat commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Find macOS controls by text and/or role without first taking a screenshot or rendering the full AX tree. This retains hyprcat’s original implementation commit and integrates current main, including persistent image configuration, hardware/SCK capture and background/agent-display behavior.

var safari = await cua.getApp("Safari", { initialState: false });
var result = await safari.query({ text: "Compose", role: "button", exact: true });
nodeRepl.write(result);
// After checking the result, use its index in this same session.
  • Native query and JS app.query() return matches with truncated, stop_reason, visited_nodes, window_id and effective budgets. Exact matching never silently retries a substring.
  • A single bounded BFS applies consistent matching and geometry rules: default 500 nodes, capped at 5000; default 20 matches, capped at 100; bounded child pages/queue/text and a two-second cooperative traversal deadline with AX messaging timeouts. AX failures and long text are reported as incomplete results.
  • Query resolves already-running apps without activation, raise or capture. Invalid window IDs/types produce tool errors instead of numeric-conversion traps.
  • Handles are bound to the app process and window, expire after 120 seconds, retain at most 5000 entries and are cleared at session end. Before acting, revalidate window ownership, control metadata/query criteria and fresh geometry; stale handles never fall back to cached coordinates.
  • Existing getApp() behavior is preserved. { initialState: false } skips its initial snapshot. Query then action requires the same persistent MCP/ocu repl session; separate CLI calls cannot reuse indexes.
  • macOS adds the tenth native tool. Windows/Linux retain the nine core tools and report query as unsupported. The model-facing plugin still advertises only js / js_reset.

Usage and option tables: English, 中文. Architecture, skill guidance, release notes and history are updated.

Validation: swift test passes 207 tests (7 skipped); all 33 Node tests pass; make check-docs and whitespace checks pass. A direct native CLI negative-window-ID regression returns a structured error and exit 1 without trapping.

Live desktop timing and stale-window scenarios have not been exercised interactively in this follow-up. The cooperative traversal deadline cannot interrupt an in-flight system call, and query only sees AX controls the app exposes.

Clicking one button should not cost a whole AX tree and a screenshot. When the
caller already knows what the control says, `query` returns just the matches —
a cost that barely moves with how complex the window is.

    open-computer-use call query --args '{"app":"Safari","text":"Compose","role":"button"}'

Each match carries an `index` the existing element actions accept, so click,
set_value, scroll and perform_secondary_action can act on a queried control
with no snapshot in between. It also finds a control that appeared after an
earlier action, without a fresh get_app_state.

The search uses the app's own `AXUIElementsForSearchPredicate` where it offers
one. Otherwise a bounded breadth-first walk that matches as it goes and stops at
`limit`, prunes off-window subtrees by geometry, walks only `AXVisibleChildren`
of tables, outlines and lists so it does not drown in scrolled-away rows, and
reads every node's attributes in one batched AX call. When the node cap stops
the walk with nothing found it says so, rather than reporting the control
absent. An `exact` miss is retried as a substring and marked `match: "contains"`,
since a label off by one word is the common miss.

Window resolution is read-only: no activation, no raise, no capture. A lookup
must not steal the foreground, or looking would change what is being looked at.

Queried indexes start at 1_000_000, rise monotonically and are never reissued,
so a later snapshot can never point an index at a different control — the
failure mode where an action silently hits the wrong thing.

Supporting changes: `ElementRecord` gains defaulted `title`/`value`;
`WindowCapture` gains exact binding by CGWindowID and geometry-only reads;
`SkyLightSPI` binds `_AXUIElementGetWindow`.

This is a tenth tool beyond the nine the MCP surface deliberately mirrors from
official computer-use. docs/ARCHITECTURE.md now records that divergence and why.
Windows and Linux are untouched and still expose the aligned nine.
@hyprcat

hyprcat commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

@iFurySt checking in: query looks up specific AX elements without a full snapshot, which pairs well with the JS REPL for cheap targeted reads. It conflicts with main now; happy to rebase onto the REPL work if you want it, or close it if it doesn't fit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants