Skip to content

One SwiftUI toolbar control reaches press/wait as two actionable elements (AMBIGUOUS_MATCH) #2480

Closed

Description

@prateekranka

One SwiftUI toolbar control reaches press/wait as two actionable elements (AMBIGUOUS_MATCH)

agent-device 0.21.0, macOS 26.6, Xcode 26.6 (17F113), iOS 26.5 simulator (iPhone 15), SwiftUI toolbar.

Observed

$ agent-device snapshot -i --json   # nodes with id scoring_home_button
  Other  "scoring_home_button"  rect [20, 63, 36, 36]        (no hittability evidence)
  Button "scoring_home_button"  rect [20.667, 63, 35, 36]    (no hittability evidence)

$ agent-device press 'id="scoring_home_button"' --settle
Error (AMBIGUOUS_MATCH): Selector matched 2 distinct actionable elements: id="scoring_home_button"
Candidates: @e3 [other] "Home"  @e4 [button] "Home"

$ agent-device wait 'id="scoring_home_button"' 3000
Error (AMBIGUOUS_MATCH): selector matched multiple elements

Root cause (TypeScript)

packages/selectors/src/interaction-targeting.ts:

  • candidatesFormSingleAncestryChain passes - the pair really is one ancestor-descendant chain.
  • findPreferredActionableDescendant filters same-rect children with a verified-hittability requirement. Regular iOS snapshots omit hittability evidence, so that branch can never fire on iOS.
  • The rects differ by <= 1 pt, beyond RECT_EQUALITY_TOLERANCE = 0.5, so areRectsApproximatelyEqual is false too.

So the wrapper resolves to itself, the control resolves to the semantic target, the classifier calls the pair ambiguous, and the documented "collapse duplicate accessibility wrappers" guarantee cannot hold for the most common SwiftUI toolbar shape.

Root cause (runner side). The native wait refusal is produced by apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift:1683 and RunnerTests+Interaction.swift:238, which refuse multi-match without the shared structural-equivalence rules.

Impact

An agent's existence check reports "the screen never appeared", and its press refuses, for a control that is uniquely addressable for a human. In our harness this produced false scenario failures that looked like app bugs.

Fix

Branch fix/unverified-hittability-wrapper-chain (fork prateekranka/agent-device) adds packages/selectors/src/interaction-targeting-wrapper-chain.ts plus a small classifier change: when every candidate lacks hittability evidence and all rects agree within 1 pt, collapse to the deepest semantic touch target. Candidates carrying any hittability fact keep today's rules. Verified red -> green on a simulator; gate 304 files / 2010 tests.

Runner question. Should the runner delegate multi-match to the shared equivalence rules, or mirror them? I can prepare that patch; it needs a simulator lane run.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions