Skip to content

ios(ci): cold toolchain probes time out below the first-exec signature stall (10 s / 5 s budgets) #2422

Description

@thymikee

Symptom

The iOS smoke lane fails on cold macOS runners with a toolchain probe timeout, unrelated to the PR under test:

  • Verify clean-installed Simulator snapshot bridge preparation: AppError: xcrun timed out after 10000ms (from verify-installed-snapshot-bridge.ts → ensureSnapshotBridgeBinary).
  • Preflight iOS runner through public CLI: Could not read the Xcode toolchain versions the Apple runner cache is keyed on (xcodebuild -version: … timed out after 5000ms).

Occurrences in the last three days, all on the same step and with no code in the failing path changed by the branch: refactor/core-replay-divergence-* (2026-09-07 16:06), claude/2339-contracts-single-owner-moves (09-06 10:23), claude/wait-poll-timeline (09-06 09:07), PR #2418 three consecutive runs on 2026-09-09 (09:50, 10:43 preflight variant, 10:51), PR #2420 (10:49, 45 s variant). Six other branches' iOS runs passed in the same window, so it is per-runner cold state, not a fleet outage.

Cause

Two cold-toolchain probes carry budgets below the first-exec stall of Apple's signature verification on a fresh host (the first xcodebuild/xcrun/large-binary exec after boot blocks roughly 18 to 19 s at 0% CPU while syspolicyd scans; the second call is instant):

  • packages/platform-apple/src/snapshot-source/cache-identity.ts:87: toolOutput runs xcodebuild -version, sw_vers, uname, xcrun --sdk iphonesimulator --show-sdk-version with timeoutMs: Math.min(10_000, remaining), under a 120 s overall deadline set by the verify script.
  • packages/platform-apple/src/runner/runner-cache-metadata.ts:27: TOOLCHAIN_PROBE_TIMEOUT_MS = 5_000 for xcodebuild -version in the runner cache key.

Both budgets were sized for a warm toolchain. The overall deadlines (120 s snapshot-source, the runner preflight budget) have room; only the per-call budget trips.

Required behavior

  • A toolchain identity probe tolerates one cold-start stall: per-call budget of at least 30 s bounded by the remaining deadline, and one retry after a timeout while the deadline allows, because the retry is instant once the signature check has run.
  • Both sites read the same constant and carry one comment naming the stall, so the two do not drift again.
  • The timeout error keeps its current text and code so existing tests and the runner-cache hint stay valid.
  • Unit tests: a fake host whose first probe call exceeds the budget and whose second returns immediately passes; a host that never returns still fails at the deadline with the same error.

Non-goals

  • Raising the snapshot-bridge native build timeout (BUILD_TIMEOUT_MS) or the overall deadlines.
  • Retrying the compile step.

Activity

  1. thymikee commented on Sep 9, 2026

    @thymikee
    MemberAuthor

    Third site, seen on PR #2421 (run 34366211699) and earlier on #2420 (10:49): the same CI step fails inside packages/platform-apple/src/snapshot-source/native-runtime.test.ts:13-32, which compiles the bridge with xcrun --sdk iphonesimulator clang … under a hard timeoutMs: 45_000, and the runner reports xcrun timed out after 45000ms. That is the native compile, not the identity probe, so PR #2423 does not cover it. On a cold runner the first xcrun here pays the same signature stall plus the compile itself. Candidate fix, separate from #2423: budget that compile from the test's own deadline (the production path uses BUILD_TIMEOUT_MS bounded by the 120 s snapshot-source deadline) rather than a fixed 45 s, or skip the compile in the unit lane when the CI step already verified it against the clean-installed package moments earlier.

  2. added 3 commits that reference this issue on Sep 10, 2026
    cd34fe5
    ae8a123
    da829e2
  3. thymikee commented on Sep 10, 2026

    @thymikee
    MemberAuthor

    Split: the third site (the fixed 45 s clang compile budget in native-runtime.test.ts) is now tracked as #2439. PR #2423 closes this issue for the two identity-probe sites it names.

  4. added 6 commits that reference this issue on Sep 10, 2026
    420f623
    dc4d5e1
    fb221cc
    44496c2
    ab8130a
    4372274
  5. added a commit that references this issue on Sep 10, 2026
    bbd53d6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions