Repository navigation
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThe emulator accessibility command now supports iOS accessibility-tree retrieval. Serve-sim session parsing provides explicit or derived AX endpoints. A request helper fetches and validates AX JSON responses with timeout and error handling. The iOS backend advertises and implements the capability. EmulatorBridge validates the active worktree session before routing requests, and RuntimeEmulatorCommands uses the new bridge method. Tests cover URL derivation, fetching, errors, routing, and device mismatches. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 63edec76-4ded-409d-88a7-9ec2734aade4
📒 Files selected for processing (10)
src/cli/specs/emulator.tssrc/main/emulator/backends/emulator-backend.tssrc/main/emulator/backends/ios-emulator-backend.test.tssrc/main/emulator/backends/ios-emulator-backend.tssrc/main/emulator/emulator-bridge.test.tssrc/main/emulator/emulator-bridge.tssrc/main/emulator/serve-sim-accessibility-tree.tssrc/main/emulator/serve-sim-detached-session.test.tssrc/main/emulator/serve-sim-detached-session.tssrc/main/runtime/orca-runtime-emulator.ts
) (#10029) * Revert "Enable accessibility tree (`ax`) command on iOS emulator sessions (#10007)" This reverts commit 43ae014. * fix(emulator): expose iOS accessibility tree * fix(emulator): support device-only iOS AX * fix(emulator): normalize iOS ax to 0..1 and heal missing axUrl serve-sim's helper /ax reports element frames in absolute pixels, but tap/gesture take normalized 0..1 coords. Normalize the raw AX node tree into a compact nested shape whose frames are 0..1 over the device screen (first root's frame), mirroring serve-sim's own normalizeAxTree, so agents can feed ax output straight back into input commands. Also heal sessions that were registered without an axUrl: #9924 only derived /ax at parse time, so already-active sessions had no endpoint. The bridge now derives it from the session's mjpeg stream URL, guarded to the /stream.mjpeg suffix so a non-mjpeg URL never fabricates a bogus /ax. * docs(emulator): mark ax working on iOS with correct raw-AX-tree shape Both skill guides and the CLI summary described iOS ax as unsupported (or, via the reverted #10007, as a normalized "screen + elements" shape that never matched the endpoint). ax works on both backends: Android via uiautomator, iOS via the serve-sim helper. Document the real iOS output — a raw AX node tree (labels, roles, nested children) with frames normalized to 0..1 — and regenerate the bundled skill guides. * chore(skills): regenerate skill bundle manifests CI verify failed because generated skill artifacts were stale after version/skill revision bumps. * fix(emulator): read ax from explicit device without active session Fall back to udid-keyed session lookup when a worktree has no active emulator, allowing `--device` targeting to work the same way for ax as it does for tap/type. Also clarify in docs that AX frames are normalized 0..1 with top-left origin, and show how to tap an element at its frame center (x+width/2, y+height/2). * fix(emulator): cap iOS AX tree at 500 nodes Unbounded accessibility trees can flood agent output. Enforce a 500-node limit (matching serve-sim's snapshot cap) and mark truncated parents so consumers know the tree was cut. --------- Co-authored-by: 5Hyeons <ohs2251@naver.com>
…blyai#10007) (stablyai#10029) * Revert "Enable accessibility tree (`ax`) command on iOS emulator sessions (stablyai#10007)" This reverts commit 43ae014. * fix(emulator): expose iOS accessibility tree * fix(emulator): support device-only iOS AX * fix(emulator): normalize iOS ax to 0..1 and heal missing axUrl serve-sim's helper /ax reports element frames in absolute pixels, but tap/gesture take normalized 0..1 coords. Normalize the raw AX node tree into a compact nested shape whose frames are 0..1 over the device screen (first root's frame), mirroring serve-sim's own normalizeAxTree, so agents can feed ax output straight back into input commands. Also heal sessions that were registered without an axUrl: stablyai#9924 only derived /ax at parse time, so already-active sessions had no endpoint. The bridge now derives it from the session's mjpeg stream URL, guarded to the /stream.mjpeg suffix so a non-mjpeg URL never fabricates a bogus /ax. * docs(emulator): mark ax working on iOS with correct raw-AX-tree shape Both skill guides and the CLI summary described iOS ax as unsupported (or, via the reverted stablyai#10007, as a normalized "screen + elements" shape that never matched the endpoint). ax works on both backends: Android via uiautomator, iOS via the serve-sim helper. Document the real iOS output — a raw AX node tree (labels, roles, nested children) with frames normalized to 0..1 — and regenerate the bundled skill guides. * chore(skills): regenerate skill bundle manifests CI verify failed because generated skill artifacts were stale after version/skill revision bumps. * fix(emulator): read ax from explicit device without active session Fall back to udid-keyed session lookup when a worktree has no active emulator, allowing `--device` targeting to work the same way for ax as it does for tap/type. Also clarify in docs that AX frames are normalized 0..1 with top-left origin, and show how to tap an element at its frame center (x+width/2, y+height/2). * fix(emulator): cap iOS AX tree at 500 nodes Unbounded accessibility trees can flood agent output. Enforce a 500-node limit (matching serve-sim's snapshot cap) and mark truncated parents so consumers know the tree was cut. --------- Co-authored-by: 5Hyeons <ohs2251@naver.com>
…blyai#10007) (stablyai#10029) * Revert "Enable accessibility tree (`ax`) command on iOS emulator sessions (stablyai#10007)" This reverts commit 9011129. * fix(emulator): expose iOS accessibility tree * fix(emulator): support device-only iOS AX * fix(emulator): normalize iOS ax to 0..1 and heal missing axUrl serve-sim's helper /ax reports element frames in absolute pixels, but tap/gesture take normalized 0..1 coords. Normalize the raw AX node tree into a compact nested shape whose frames are 0..1 over the device screen (first root's frame), mirroring serve-sim's own normalizeAxTree, so agents can feed ax output straight back into input commands. Also heal sessions that were registered without an axUrl: stablyai#9924 only derived /ax at parse time, so already-active sessions had no endpoint. The bridge now derives it from the session's mjpeg stream URL, guarded to the /stream.mjpeg suffix so a non-mjpeg URL never fabricates a bogus /ax. * docs(emulator): mark ax working on iOS with correct raw-AX-tree shape Both skill guides and the CLI summary described iOS ax as unsupported (or, via the reverted stablyai#10007, as a normalized "screen + elements" shape that never matched the endpoint). ax works on both backends: Android via uiautomator, iOS via the serve-sim helper. Document the real iOS output — a raw AX node tree (labels, roles, nested children) with frames normalized to 0..1 — and regenerate the bundled skill guides. * chore(skills): regenerate skill bundle manifests CI verify failed because generated skill artifacts were stale after version/skill revision bumps. * fix(emulator): read ax from explicit device without active session Fall back to udid-keyed session lookup when a worktree has no active emulator, allowing `--device` targeting to work the same way for ax as it does for tap/type. Also clarify in docs that AX frames are normalized 0..1 with top-left origin, and show how to tap an element at its frame center (x+width/2, y+height/2). * fix(emulator): cap iOS AX tree at 500 nodes Unbounded accessibility trees can flood agent output. Enforce a 500-node limit (matching serve-sim's snapshot cap) and mark truncated parents so consumers know the tree was cut. --------- Co-authored-by: 5Hyeons <ohs2251@naver.com>
|
Thanks for adding the missing iOS accessibility-tree path, @5Hyeons, and for following up on device-only targeting. This is now available on main: I checked the current bridge routing against this PR and am closing it because the feature is already in place. Thank you for the implementation, regression tests, and Simulator verification! |
Closes #9923
Summary
Enable
orca emulator axfor active iOS Simulator sessions./axbeside the detachedserve-simstream URL whenaxUrlis omittednet.fetchEmulatorBridge, with a clear device-mismatch errorThis gives agents structured labels, roles, and frames for iOS controls instead of relying only on screenshot-based coordinate estimation.
Root cause
serve-simalready exposed the iOS accessibility tree, but Orca declared the capability unsupported and never connected the active session's endpoint to the command runtime.Screenshots
No visual change.
Manual QA used the returned AX frames to open Safari, dismiss an unexpected Safari popover, follow an Apple.com link, and navigate back on a separate iPhone Simulator.
Testing
pnpm lint— repository-wide lint reaches a pre-existing non-exhaustive switch atsrc/renderer/src/components/skills/skill-freshness-group.tsx:101;oxlintandoxfmt --checkpass for every changed filepnpm typecheckpnpm test— 3,217 files passed; 34,131 tests passed; 58 skippedpnpm buildBaseline reproduction on
main:Fixed development build:
{"ok":true,"resultType":"array","roots":1,"rootType":"Application","childCount":18}AI Review Report
The review checked correctness, cross-platform behavior, local/SSH boundaries, agent and integration compatibility, performance, UI impact, and basic security.
EmulatorBridge.accessibilityTreeand adds a device-mismatch regression test.emulator axis invoked; it is not on a render or polling hot path.Final independent re-review found no remaining blocking, P1, or P2 findings.
Security Audit
serve-simsession.Notes
ELI5
orca emulator ax works for iOS Simulator sessions, not only Android. Agents can read accessibility labels, roles, and frames from the active iOS sim the same way.