Repository navigation
Conversation
Clicking one button should not cost a whole AX tree and a screenshot. When the
caller already knows what the control says, `query` returns just the matches —
a cost that barely moves with how complex the window is.
open-computer-use call query --args '{"app":"Safari","text":"Compose","role":"button"}'
Each match carries an `index` the existing element actions accept, so click,
set_value, scroll and perform_secondary_action can act on a queried control
with no snapshot in between. It also finds a control that appeared after an
earlier action, without a fresh get_app_state.
The search uses the app's own `AXUIElementsForSearchPredicate` where it offers
one. Otherwise a bounded breadth-first walk that matches as it goes and stops at
`limit`, prunes off-window subtrees by geometry, walks only `AXVisibleChildren`
of tables, outlines and lists so it does not drown in scrolled-away rows, and
reads every node's attributes in one batched AX call. When the node cap stops
the walk with nothing found it says so, rather than reporting the control
absent. An `exact` miss is retried as a substring and marked `match: "contains"`,
since a label off by one word is the common miss.
Window resolution is read-only: no activation, no raise, no capture. A lookup
must not steal the foreground, or looking would change what is being looked at.
Queried indexes start at 1_000_000, rise monotonically and are never reissued,
so a later snapshot can never point an index at a different control — the
failure mode where an action silently hits the wrong thing.
Supporting changes: `ElementRecord` gains defaulted `title`/`value`;
`WindowCapture` gains exact binding by CGWindowID and geometry-only reads;
`SkyLightSPI` binds `_AXUIElementGetWindow`.
This is a tenth tool beyond the nine the MCP surface deliberately mirrors from
official computer-use. docs/ARCHITECTURE.md now records that divergence and why.
Windows and Linux are untouched and still expose the aligned nine.
This was referenced Sep 15, 2026
Contributor
Author
|
@iFurySt checking in: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Find macOS controls by text and/or role without first taking a screenshot or rendering the full AX tree. This retains hyprcat’s original implementation commit and integrates current main, including persistent image configuration, hardware/SCK capture and background/agent-display behavior.
queryand JSapp.query()return matches withtruncated,stop_reason,visited_nodes,window_idand effective budgets. Exact matching never silently retries a substring.getApp()behavior is preserved.{ initialState: false }skips its initial snapshot. Query then action requires the same persistent MCP/ocu replsession; separate CLI calls cannot reuse indexes.js/js_reset.Usage and option tables: English, 中文. Architecture, skill guidance, release notes and history are updated.
Validation:
swift testpasses 207 tests (7 skipped); all 33 Node tests pass;make check-docsand whitespace checks pass. A direct native CLI negative-window-ID regression returns a structured error and exit 1 without trapping.Live desktop timing and stale-window scenarios have not been exercised interactively in this follow-up. The cooperative traversal deadline cannot interrupt an in-flight system call, and query only sees AX controls the app exposes.