Skip to content

fix(emulator): expose iOS accessibility tree - #9924

Closed
5Hyeons wants to merge 2 commits into
stablyai:mainfrom
5Hyeons:fix/ios-emulator-ax
Closed

5Hyeons wants to merge 2 commits into
stablyai:mainfrom
5Hyeons:fix/ios-emulator-ax

Conversation

@5Hyeons

@5Hyeons 5Hyeons commented Jul 22, 2026 •

Copy link
Copy Markdown
Contributor

Closes #9923

Summary

Enable orca emulator ax for active iOS Simulator sessions.

  • advertise the iOS accessibility-tree capability
  • derive /ax beside the detached serve-sim stream URL when axUrl is omitted
  • fetch and validate the raw AX tree through Electron net.fetch
  • keep target and active-session resolution inside EmulatorBridge, with a clear device-mismatch error
  • leave the Android accessibility path unchanged

This gives agents structured labels, roles, and frames for iOS controls instead of relying only on screenshot-based coordinate estimation.

Root cause

serve-sim already exposed the iOS accessibility tree, but Orca declared the capability unsupported and never connected the active session's endpoint to the command runtime.

Screenshots

No visual change.

Manual QA used the returned AX frames to open Safari, dismiss an unexpected Safari popover, follow an Apple.com link, and navigate back on a separate iPhone Simulator.

Testing

  • pnpm lint — repository-wide lint reaches a pre-existing non-exhaustive switch at src/renderer/src/components/skills/skill-freshness-group.tsx:101; oxlint and oxfmt --check pass for every changed file
  • pnpm typecheck
  • pnpm test — 3,217 files passed; 34,131 tests passed; 58 skipped
  • pnpm build
  • Added regression coverage for capability advertisement, endpoint derivation, raw AX responses, unavailable helpers, active-session routing, and device mismatch

Baseline reproduction on main:

emulator_unsupported: accessibilityTree is not supported by the ios emulator backend
exit 1

Fixed development build:

{"ok":true,"resultType":"array","roots":1,"rootType":"Application","childCount":18}

AI Review Report

The review checked correctness, cross-platform behavior, local/SSH boundaries, agent and integration compatibility, performance, UI impact, and basic security.

  • It initially flagged duplicated iOS target/session resolution in the runtime as a possible session-switch race. The final change moves that decision into EmulatorBridge.accessibilityTree and adds a device-mismatch regression test.
  • macOS: verified with a real development Orca and separate iPhone Simulator.
  • Linux/Windows: the iOS backend remains host-gated; no shortcuts, labels, filesystem paths, or shell behavior changed.
  • Remote/SSH: emulator control remains local-host-only; no remote workspace behavior changed.
  • Agents/integrations: the public CLI response remains raw platform accessibility data; no provider-specific behavior changed.
  • Performance: one bounded local helper request occurs only when emulator ax is invoked; it is not on a render or polling hot path.
  • UI: no UI code or visual behavior changed.

Final independent re-review found no remaining blocking, P1, or P2 findings.

Security Audit

  • No new dependency, auth flow, secret handling, shell command construction, filesystem write, or renderer IPC surface was added.
  • The request URL comes from the active locally managed serve-sim session.
  • Requests have a five-second timeout; non-2xx responses are surfaced, error bodies are truncated to 512 characters, and successful bodies are parsed and shape-checked before being returned.
  • No security follow-up was identified.

Notes

  • The behavior is macOS/iOS-specific; Android remains unchanged.
  • X (Twitter) handle: Not provided.

ELI5

orca emulator ax works for iOS Simulator sessions, not only Android. Agents can read accessibility labels, roles, and frames from the active iOS sim the same way.

@5Hyeons
5Hyeons marked this pull request as ready for review July 22, 2026 11:10
@coderabbitai

coderabbitai Bot commented Jul 22, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 4dae572a-4fc8-4c04-a694-0c83f5cff5bf

📥 Commits

Reviewing files that changed from the base of the PR and between a6aaca8 and 7b65c2b.

📒 Files selected for processing (2)
  • src/main/emulator/emulator-bridge.test.ts
  • src/main/emulator/emulator-bridge.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/main/emulator/emulator-bridge.ts

📝 Walkthrough

Walkthrough

The emulator accessibility command now supports iOS accessibility-tree retrieval. Serve-sim session parsing provides explicit or derived AX endpoints. A request helper fetches and validates AX JSON responses with timeout and error handling. The iOS backend advertises and implements the capability. EmulatorBridge validates the active worktree session before routing requests, and RuntimeEmulatorCommands uses the new bridge method. Tests cover URL derivation, fetching, errors, routing, and device mismatches.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately summarizes the main change: exposing the iOS accessibility tree.
Description check ✅ Passed The description includes the required sections and adequately covers summary, testing, AI review, security, and notes.
Linked Issues check ✅ Passed The PR implements #9923 by enabling iOS AX support, deriving /ax, routing the active session, and leaving Android behavior unchanged.
Out of Scope Changes check ✅ Passed The changes appear scoped to iOS AX support and regression coverage, with no clear unrelated additions.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 63edec76-4ded-409d-88a7-9ec2734aade4

📥 Commits

Reviewing files that changed from the base of the PR and between 39b124c and a6aaca8.

📒 Files selected for processing (10)
  • src/cli/specs/emulator.ts
  • src/main/emulator/backends/emulator-backend.ts
  • src/main/emulator/backends/ios-emulator-backend.test.ts
  • src/main/emulator/backends/ios-emulator-backend.ts
  • src/main/emulator/emulator-bridge.test.ts
  • src/main/emulator/emulator-bridge.ts
  • src/main/emulator/serve-sim-accessibility-tree.ts
  • src/main/emulator/serve-sim-detached-session.test.ts
  • src/main/emulator/serve-sim-detached-session.ts
  • src/main/runtime/orca-runtime-emulator.ts

Comment thread src/main/emulator/emulator-bridge.ts Resolved
@AmethystLiang
AmethystLiang requested a review from OrcaWin July 22, 2026 17:03
AmethystLiang added a commit that referenced this pull request Jul 23, 2026
) (#10029)

* Revert "Enable accessibility tree (`ax`) command on iOS emulator sessions (#10007)"

This reverts commit 43ae014.

* fix(emulator): expose iOS accessibility tree

* fix(emulator): support device-only iOS AX

* fix(emulator): normalize iOS ax to 0..1 and heal missing axUrl

serve-sim's helper /ax reports element frames in absolute pixels, but
tap/gesture take normalized 0..1 coords. Normalize the raw AX node tree
into a compact nested shape whose frames are 0..1 over the device screen
(first root's frame), mirroring serve-sim's own normalizeAxTree, so agents
can feed ax output straight back into input commands.

Also heal sessions that were registered without an axUrl: #9924 only
derived /ax at parse time, so already-active sessions had no endpoint.
The bridge now derives it from the session's mjpeg stream URL, guarded to
the /stream.mjpeg suffix so a non-mjpeg URL never fabricates a bogus /ax.

* docs(emulator): mark ax working on iOS with correct raw-AX-tree shape

Both skill guides and the CLI summary described iOS ax as unsupported (or,
via the reverted #10007, as a normalized "screen + elements" shape that
never matched the endpoint). ax works on both backends: Android via
uiautomator, iOS via the serve-sim helper. Document the real iOS output —
a raw AX node tree (labels, roles, nested children) with frames normalized
to 0..1 — and regenerate the bundled skill guides.

* chore(skills): regenerate skill bundle manifests

CI verify failed because generated skill artifacts were stale after version/skill revision bumps.

* fix(emulator): read ax from explicit device without active session

Fall back to udid-keyed session lookup when a worktree has no active emulator,
allowing `--device` targeting to work the same way for ax as it does for tap/type.
Also clarify in docs that AX frames are normalized 0..1 with top-left origin,
and show how to tap an element at its frame center (x+width/2, y+height/2).

* fix(emulator): cap iOS AX tree at 500 nodes

Unbounded accessibility trees can flood agent output. Enforce a 500-node limit (matching serve-sim's snapshot cap) and mark truncated parents so consumers know the tree was cut.

---------

Co-authored-by: 5Hyeons <ohs2251@naver.com>
gal064 pushed a commit to gal064/orca that referenced this pull request Jul 25, 2026
…blyai#10007) (stablyai#10029)

* Revert "Enable accessibility tree (`ax`) command on iOS emulator sessions (stablyai#10007)"

This reverts commit 43ae014.

* fix(emulator): expose iOS accessibility tree

* fix(emulator): support device-only iOS AX

* fix(emulator): normalize iOS ax to 0..1 and heal missing axUrl

serve-sim's helper /ax reports element frames in absolute pixels, but
tap/gesture take normalized 0..1 coords. Normalize the raw AX node tree
into a compact nested shape whose frames are 0..1 over the device screen
(first root's frame), mirroring serve-sim's own normalizeAxTree, so agents
can feed ax output straight back into input commands.

Also heal sessions that were registered without an axUrl: stablyai#9924 only
derived /ax at parse time, so already-active sessions had no endpoint.
The bridge now derives it from the session's mjpeg stream URL, guarded to
the /stream.mjpeg suffix so a non-mjpeg URL never fabricates a bogus /ax.

* docs(emulator): mark ax working on iOS with correct raw-AX-tree shape

Both skill guides and the CLI summary described iOS ax as unsupported (or,
via the reverted stablyai#10007, as a normalized "screen + elements" shape that
never matched the endpoint). ax works on both backends: Android via
uiautomator, iOS via the serve-sim helper. Document the real iOS output —
a raw AX node tree (labels, roles, nested children) with frames normalized
to 0..1 — and regenerate the bundled skill guides.

* chore(skills): regenerate skill bundle manifests

CI verify failed because generated skill artifacts were stale after version/skill revision bumps.

* fix(emulator): read ax from explicit device without active session

Fall back to udid-keyed session lookup when a worktree has no active emulator,
allowing `--device` targeting to work the same way for ax as it does for tap/type.
Also clarify in docs that AX frames are normalized 0..1 with top-left origin,
and show how to tap an element at its frame center (x+width/2, y+height/2).

* fix(emulator): cap iOS AX tree at 500 nodes

Unbounded accessibility trees can flood agent output. Enforce a 500-node limit (matching serve-sim's snapshot cap) and mark truncated parents so consumers know the tree was cut.

---------

Co-authored-by: 5Hyeons <ohs2251@naver.com>
@nwparker nwparker added the bug Something isn't working label Jul 27, 2026
enrick-dev pushed a commit to enrick-dev/orca that referenced this pull request Oct 15, 2026
…blyai#10007) (stablyai#10029)

* Revert "Enable accessibility tree (`ax`) command on iOS emulator sessions (stablyai#10007)"

This reverts commit 9011129.

* fix(emulator): expose iOS accessibility tree

* fix(emulator): support device-only iOS AX

* fix(emulator): normalize iOS ax to 0..1 and heal missing axUrl

serve-sim's helper /ax reports element frames in absolute pixels, but
tap/gesture take normalized 0..1 coords. Normalize the raw AX node tree
into a compact nested shape whose frames are 0..1 over the device screen
(first root's frame), mirroring serve-sim's own normalizeAxTree, so agents
can feed ax output straight back into input commands.

Also heal sessions that were registered without an axUrl: stablyai#9924 only
derived /ax at parse time, so already-active sessions had no endpoint.
The bridge now derives it from the session's mjpeg stream URL, guarded to
the /stream.mjpeg suffix so a non-mjpeg URL never fabricates a bogus /ax.

* docs(emulator): mark ax working on iOS with correct raw-AX-tree shape

Both skill guides and the CLI summary described iOS ax as unsupported (or,
via the reverted stablyai#10007, as a normalized "screen + elements" shape that
never matched the endpoint). ax works on both backends: Android via
uiautomator, iOS via the serve-sim helper. Document the real iOS output —
a raw AX node tree (labels, roles, nested children) with frames normalized
to 0..1 — and regenerate the bundled skill guides.

* chore(skills): regenerate skill bundle manifests

CI verify failed because generated skill artifacts were stale after version/skill revision bumps.

* fix(emulator): read ax from explicit device without active session

Fall back to udid-keyed session lookup when a worktree has no active emulator,
allowing `--device` targeting to work the same way for ax as it does for tap/type.
Also clarify in docs that AX frames are normalized 0..1 with top-left origin,
and show how to tap an element at its frame center (x+width/2, y+height/2).

* fix(emulator): cap iOS AX tree at 500 nodes

Unbounded accessibility trees can flood agent output. Enforce a 500-node limit (matching serve-sim's snapshot cap) and mark truncated parents so consumers know the tree was cut.

---------

Co-authored-by: 5Hyeons <ohs2251@naver.com>
@OrcaWin

OrcaWin commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

Thanks for adding the missing iOS accessibility-tree path, @5Hyeons, and for following up on device-only targeting. This is now available on main: emulator ax reaches the iOS backend, the bridge resolves the attached device’s AX endpoint, and the HTTP reader validates the response and handles helper failures. Main also derives the endpoint for older registered sessions.

I checked the current bridge routing against this PR and am closing it because the feature is already in place. Thank you for the implementation, regression tests, and Simulator verification!

@OrcaWin OrcaWin closed this Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: iOS emulator AX command is disabled despite serve-sim support

3 participants