Skip to content

iOS Simulator bridge and web content: follow-ups to the #2484 fence (content-class gate, probe circuit, WebContent spike) #2492

Open

Description

@thymikee

Follow-up to #2484 / #2486. That PR is a fence: the Simulator AX bridge refuses a tree that ends at a web view's AXRemoteElement leaf (remote-content-boundary) and the route serves XCTest for the rest of the app generation. This issue scopes what makes the class of defect impossible or cheaper, in the order the design consultation ranked them (two adversarial reviews plus a codex exec architecture pass; verdicts below are the agreed ones).

Why we regressed

  • feat(ios): route Simulator snapshots through AX bridge #2279 judged the bridge on node/depth parity, ownership, and latency against native apps. The one fact that mattered — the reader snapshots one process (apple/snapshot-bridge/README.md) — was never a declared capability or a test.
  • The failure was silent: a well-formed tree with a viewport and no truncation passes every protocol check, and the quality verdict knows sparse roots, not an on-screen web view with nothing inside.
  • No gate drove web content through the default route. The WebView lab (feat: add WebView accessibility lab #1397) was a manual lab. http(s) URLs open Safari by design, but Safari was never in the acquisition validation set.

Tasks, ranked

1. Content-class gate (high value, moderate cost)

Extend the fixture-backed iOS E2E lane with a fixed screen set captured through the default route, asserting coarse content presence, backend-independent: native, WKWebView (exists: smoke:webview-remote-content), Safari on a harness-served local page (Node http server in the scenario; no network), a system sheet. Each row opens a fresh app generation so an earlier fallback cannot mask a broken bridge path. Add one deterministic page action with an observable result (a page ref press/fill) to the WebView row — today it checks content and warnings only (test/integration/ios-simulator-e2e/live-webview-remote-content.ts). Keep fallback-policy assertions (remote-content-boundary / circuit-disabled) separate from content assertions.

2. Launch-observation probe opens the circuit (high value, low cost)

snapshot-observability.ts reads the full tree (its depth: 1 regular hint resolves to full depth in adapter.ts), answers unobservable on a web-first app, and cannot record the refusal: only fallbackAfterFailure in snapshot-route.ts opens the circuit. So a Safari open pays the 300 ms fixed settle and the first snapshot pays one more refused bridge round trip. Pass a route-owned failure-recording hook into the probe and open the circuit for remote-content-boundary. Keep the verdict unobservable; mapping it to observable contradicts the contract in snapshot-observability.ts ("the host AX bridge published its tree").

3. Public help text (modest value, very low cost)

snapshot --help prints the internal backend capability table (src/commands/capture/snapshot.ts, snapshotBackendCapabilityHelp). The #2484 reporter read known gap deep-extension as this bug. Drop the internal table from ordinary help; keep the user-facing depth and fallback semantics.

4. Spike: bounded WebContent acquisition through the bridge's remote-access client (high information value, timeboxed)

Problem. The bridge cannot establish page content. Before any cross-process traversal is designed, prove whether its XCTAccessibilityFramework initForRemoteAccess client (SnapshotBridgeRuntime.m) can read an associated WebContent document while keeping foreground ownership, bounded capture, and actionable node identity.

Blocker to clear first. snapshotForProcess: checks the read pid against primaryApp before capture, during continuations, and after (SnapshotBridgeRuntime.m:319/326/337). A describe request with the WebContent pid is rejected by our own gate, which proves nothing about Apple's AX. The spike must separate the foreground-owner pid from the read pid in disposable code and keep owner checks against Safari or the fixture app.

Prove, in order.

  1. The experiment reaches the AX call (owner/read pid split; verified WebContent pid supplied manually the first time).
  2. Remote access returns page semantics: a deterministic local page with unique body text, a link, and an input; require roles and content, not a non-null root or the document title. Record native error domain/code, elapsed time, automation state, bounds. Keep a successful app-pid read as the control.
  3. The boundary supplies a generic target: inspect UIAccessibilitySnapshotKeyElement for the AXRemoteElement before JSON conversion; establish whether its pid denotes the host proxy or WebContent; try direct element-rooted resolution (SnapshotBridgeCapture.m continuation seam; note the runner re-roots retained elements via a different request API in RunnerAXSnapshotBridge.m). A Safari TabDocument … WebViewProcessID= string is discovery evidence only, never a production association contract.
  4. The target identifies the correct document: two WKWebViews, a second Safari tab, and a navigation that replaces the WebContent process.
  5. Composition preserves identity and geometry: acquisition stamps every node with target.pid (adapter.ts); prove node attribution, index/parent normalization, coordinate space, clipping, and a page-ref interaction through the existing runner path.
  6. Recovery stays bounded: share the existing node/request/deadline budgets across all reads; keep the fence when resolution fails; cover duplicate boundaries, process disappearance, and navigation during capture.

Non-goals. Shipping grafting; changing default routing or the circuit; removing the fence; Safari-specific pid parsing in production; new quality vocabulary; physical devices.

Exit criteria.

  • No-go: controlled reads reach AX but cannot obtain page semantics within bounds → record the exact failure, keep XCTest fallback.
  • Partial: manually selected WebContent reads succeed but generic association or safe composition fails → report transport feasibility only.
  • Go: both apps yield correctly associated page content, composition survives a document replacement, a page ref acts within shared bounds → open a separate implementation issue.
  • Timebox: two engineering days; unresolved prerequisites end as inconclusive with the missing proof named.

Report: runtime/Xcode versions, native errors, call counts, elapsed time, process identities.

5. Truncation completeness policy (decision, then small change)

A tree cut by the bridge's 1500-node cap (snapshot-source/limits.ts) above the web view carries no remote leaf: the page is missing and only truncated residue discloses it. Decide when an involuntarily capped regular acquisition is publishable versus requiring fallback, for native controls beyond the cap as well as web views. Use the existing truncation residue; do not infer missing content from surviving shapes.

Rejected

  • A producer-independent "on-screen WebView with no descendants" degraded-output verdict: empty web views are legitimate, depth/filtering can remove descendants, and truncation above the web view leaves nothing to classify. At most a qualified "content unverified" observation, not degradation or fallback.
  • A dedicated SnapshotSourceFailureKind for per-screen refusals: changes no policy today; introduce it only when a consumer needs distinct recovery behavior, and migrate the producers that already use unsupported for transient states together.
  • Re-asking the bridge per capture instead of the per-generation circuit: charges a refused round trip to every wait poll on a web screen. Revisit only with measurements of native-screen cost in hybrid apps.

Related

  • open <http url> on a simulator does not bind Safari to the session (src/platform-runtime-open-target.ts gates the fallback to physical devices); tracked separately.

Activity

  1. thymikee commented on Sep 25, 2026

    @thymikee
    MemberAuthor

    New evidence: E2E smoke:regular-visible-depth-frontier failed in run 36015557135. The regular depth-1 snapshot came back XCTest-served (backend: "tree", circuit-disabled), so the test never exercised the bridge depth contract (live-snapshot-depth-frontier.ts ~127-136). The scenario runs open --relaunch, which should start a new app generation (new pid and start time) and clear the old circuit (snapshot-target.ts ~126-137, snapshot-route.ts ~518-527). Before weakening the assertion, diagnose why this capture still used a disabled circuit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions