Repository navigation
Diagnose recurring multi-minute Settings replay click in iOS Smoke #2948
Description
Activity
The first-click delay is predominantly runner startup, not tap synthesis. In the linked #2936 job, the runner was preflighted (97.0 s), then the workflow ran
pnpm clean:daemon, releasing it. The first Settings click took 143.8 s; its runner log shows an interrupted initial launch, a fullbuild-for-testing, another interrupted launch, then a ready runner. Once ready, snapshot and tap completed in under 3 s; the tap was synthesized at(201, 406)and reported success, but the next wait still timed out. That does not prove navigation occurred.#2949 keeps the prepared runner alive through the replay. A local dedicated simulator passed the replay in 10.0 s with the same daemon (1.3 s click), versus 12.0 s after cleanup (1.4 s click); the local cache was warm. I am waiting for exact-head CI to measure the cold-run benefit. This issue remains open for the failed post-tap destination check and any remaining startup delay.
Exact-head #2949 iOS job passed and proves the prepared runner was reused: one preflight
xcodebuildlaunch, runner PID 30721 serving both preflight and Settings commands, and no rebuild before replay. The Settings step fell from 4m50s to 1m28s; first click fell from 143.840 s to 18.189 s.The remaining first-attempt failure is real.
openreportedpostOpenObservation: unobservable; first click spent 13.1 s in a runner snapshot after a 4 s system-modal probe timeout and private-AX fallback, then the tap command completed in 1.1 s. The subsequent wait failed after 9.0 s; attempt 2 passed with a 4.7 s click. Runner reuse removes the multi-minute restart/build penalty but does not establish that the first tap navigated. Keep this issue open for the residual snapshot and post-tap behavior.Follow-up on the residual after #2949: the failed attempt does not prove a missed tap. The first runner snapshot took 13.09 s (4 s system-modal probe timeout, drain, then private-AX fallback); the tap took 1.07 s. The next wait received no readable capture before its deadline: its bridge acquisition was canceled, and the runner fallback snapshot completed after the wait. The failed attempt tapped
(201,319)and the passing retry(201,406), but the older #2936 failure tapped(201,406)too, so coordinate difference is not causal proof.A separate local replay passed; its 19.5 s click was cold runner startup and did not reproduce this wait failure. The next diagnostic run should enable global
--debugfor the Settings test and retain an on-failure screen image or video plus the selected-node rectangle and capture backend. That would tell us whether navigation missed or the wait simply lacked a readable capture. No interaction fix is justified from the current artifacts.Phase-attribution report — diagnostic plan executed live, 2026-10-08 (iPhone Duo, iOS 27.1, UDID 4F879835)
Ran the owning lane's real harness (
node src/bin.ts test test/integration/replays/ios/simulator/01-settings.ad --udid … --retries 2, i.e. the ios.yml "Run iOS Settings replay smoke test" command shape,prepare ios-runnerfirst, same shared state dir), 20 replay cycles = 44 attempts plus a cold-daemon no-preflight class and 10+13 focused probe iterations. Evidence:replay-timing.ndjsonper attempt, per-request daemon logs (sessions/<s>/requests/<req>.ndjsonunder--debug+ explicit--state-dir), sessionrunner.log, and today'smainCI artifactios-artifactsfrom run37815426076(which reproduced the original signature: attempt 1 failed at step 4, total 57.8s flaky).Observed distribution (44 local attempts + 1 live CI failing attempt)
- 0/44 click (step 3) failures; 0/44 step-4 wait failures locally; 42/44 attempts failed at step 7
back— a distinct iOS-27.1 defect, filed as iOS 27.1: back always refuses on the SwiftUI floating toolbar (top-band frame rule rejects the real control; leading fallback taps empty space) #3333 (runnerbackInApprejects the real_UIButtonBarButton identifier=BackButtonbecause iOS 27's floating toolbar sits below the top-band frame rule, then the leading fallback taps empty space; 12/13 direct failures;click @refon the same button navigates fine). - Step-4 wait failure class seen live today on CI (run
37815426076, 1 of the last 6 main iOS runs; the other 5 passed first attempt 22.7–31.2s). click Generalwall-clock: warm p50 2.64 s (n=44), cold-first-click 15.5 s locally with warm Xcode cache; CI pre-perf(ios): reuse prepared runner in Settings smoke #2949 that same slot was 143–191 s.
Phase attribution of the multi-minute click (cold-1 attempt-1, 15.5 s local; same shape as the historic 143–191 s, minus Xcode-cache warmth)
From the click's own request ndjson: bridge target readiness 0.24 s; runner demand inside the click:
build-for-testing13.1 s (exec_commanddur), xctestrun prep 0.4 s,test-without-buildinglaunch + port-connect retries 4.9 s (firstuptimeround-tripdur=4926), readiness preflight, then tap synthesisios_runner_command_send cmd=tap dur=888 ms. i.e. the multi-minute window is runner build+launch inside the first runner-demanding command — the #2949 preflight-reuse class, already merged; the diagnostic reproduces its shape. Warm click phases: bridge probe/fallback ~0.2 s + readiness snapshot 2.0 s (snapshot_capture dur=2001 backend=xctest) + tap 0.59 s ≈ 2.6 s.Did
Generalactually activate before click success?- Locally (iOS 27.1): yes, verified.
click Generalsucceeded, and the failure-time screen at step 7 is the General detail page (nav title General,cell About); direct probes: 10/10is exists "label=About"pass immediately after click. The 42 step-7 failures are the back-control defect, not a missed tap. The retry's success here is coincidental floating-toolbar flake, not proof. - On the live CI failing attempt: unanswerable from retained evidence, and likely not activated. Attempt 1's tap was synthesized at
(201,319)~0.5 s after a 2.8 s capture of a still-settling root list (that y hits the Apple-Account band); the passing attempt 2 tapped the settled(201,406). The click reported success from the tap ack (tap dur≈0.8 s, ok=1), and the wait then got zero readable captures — its runner snapshot was onlyCOMMAND_ACCEPTEDat 17:41:47.2, after the wait had already failed at 17:41:46.4, and completed ok=1 1.6 s later against a dead request. Success was reported without any landing observation. Per the issue's own rule ("file it as a new issue with evidence rather than fixing it here"), that owning-path gap is filed as click and press report dispatch, not landing: state the contract and track the outcomeObservation gap #3335.
Bridge-circuit correlation test (the #2491/#3328 signature)
Does not correlate with the multi-minute window. In every local attempt the daemon emitted
ios_snapshot_route_fallback(foreground-owner-unverifiedat open →circuit-disabledfor the app generation; 45+96 events), and each fallback cost ≤ ~2.2 s on that capture, never more; clicks stayed at 2.5–3 s throughout. In the CI failing window (17:41–17:42) the replay session's runner log has zeroPRIVATE_AX_SNAPSHOT_FAILED/SNAPSHOT_BACKEND_FAILEDlines — those signatures in the same job log belong to the runner-XCTest/E2E sessions at 17:24–17:34. The failing wait's 10.4 s against a 5 s budget is a capture that never reached the runner in budget (dispatched after the wait failed), i.e. the observation-stall class of #2491/#3328 — not circuit-consumed time.Disposition
- No focused fix PR from this row: the multi-minute click is already owned by merged perf(ios): reuse prepared runner in Settings smoke #2949 (reproduced as the cold-start shape above); the step-4 wait-stall is owned by ci(ios): the iOS simulator smoke lane is failing on main across several rotating signatures #2491/ci(ios): a regular depth-1 snapshot carries XCTest tree quality metadata when the AX bridge probe circuit is open #3328; the two defects my evidence establishes are filed as new owned issues: iOS 27.1: back always refuses on the SwiftUI floating toolbar (top-band frame rule rejects the real control; leading fallback taps empty space) #3333 (iOS 27.1 floating-toolbar
back, blocks this replay's step 7 on Duo/27.1, with runner-frame evidence and the exactisTopNavigationControlFrame/fallback-point mechanism) and click and press report dispatch, not landing: state the contract and track the outcomeObservation gap #3335 (click success carries no activation/landing evidence; the required "did it activate" question was unrecoverable on the live failing CI attempt, with the guarantee-matrixverifyEvidencecell as the owning seam). - Retry kept as containment throughout; no successful retry was used as proof of click correctness.
Cleanup:
diag2948session closed, daemon retired viapnpm clean:daemon, iPhone Duo 4F879835 shut down; no other device touched.- 0/44 click (step 3) failures; 0/44 step-4 wait failures locally; 42/44 attempts failed at step 7
Closing as completed. Each completion clause now has an owner:
- Multi-minute first click: attributed to the runner
build-for-testingand launch inside the first command that needs the runner. Merged perf(ios): reuse prepared runner in Settings smoke #2949 fixes it by reusing the prepared runner: on CI the first click fell from 143.8 s to 18.2 s and the Settings step from 4m50s to 1m28s. Locally the cold first click took 15.5 s and warm clicks about 2.6 s at p50. - Failed wait after the click: an observation stall where the wait gets no readable capture before its deadline. Owned by ci(ios): the iOS simulator smoke lane is failing on main across several rotating signatures #2491/ci(ios): a regular depth-1 snapshot carries XCTest tree quality metadata when the AX bridge probe circuit is open #3328.
- Did
Generalactivate before click success: locally yes (10/10 probes). On the live CI failure (run37815426076) the retained evidence cannot answer it, because click success has no landing evidence. Moved to click and press report dispatch, not landing: state the contract and track the outcomeObservation gap #3335. - iOS 27.1
backfailures seen during the diagnostic runs (42/44 attempts): owned by iOS 27.1: back always refuses on the SwiftUI floating toolbar (top-band frame rule rejects the real control; leading fallback taps empty space) #3333.
The retry is still in place as containment. No retry success was treated as proof that the click worked.
- Multi-minute first click: attributed to the runner
Purpose
Reduce a recurring multi-minute delay in the iOS Smoke Settings replay and determine whether its successful
click Generalresponse reflects a completed interaction.Evidence
Three independent exact-head PR runs on 2026-09-24 show the same pattern in
01-settings.ad: the first attempt reports step 3click Generalas successful after 143.840 s, 191.076 s, or 165.128 s; step 4 then times out after 5 s waiting forAbout,Software Update, or the expected Settings text. The retry succeeds in 30.5 s, 23.6 s, or 14.8 s respectively.The recorded replay timing establishes the delay and failed wait; it does not establish which internal click phase consumed the time or whether the tap landed. The #2936 failed wait request log includes a
simulator-target-discovery-pendingbridge fallback, but successful click requests did not retain equivalent phase diagnostics.Required behavior
Generalwas actually activated before reporting click success; repair the owning path if success can be reported without the action, or bound an optional pre-action probe beneath the operation it precedes.Completion
Depends on no current PR; this is separate from #2936's depth-frontier diagnostic.