[thymikee]

Summary

Experiment with two isolated simulator processes for the iOS PR XCTest selection. Alternate generated method IDs; retain every test, assertion, and timeout. Wait for both workers, preserve failures, upload separate results, and verify exact method coverage. Four files; follow-up to #3802.

Validation

Tested head: 027b43734b26134459be39310eeef0e61a1fd1ab.

• pnpm check:affected --run --head HEAD: 36 checks passed; receipt run-jsvaj0, exactCommit=true, checkoutChanged=false. Focused reporter/selection tests: 51 passed. Independent review: no findings.
• Local pilot: 48/48 executions passed at 9beae21eaf; warm 12-test pair improved 39.7%, excluding boot.
• Hosted CI: all 125 selected methods executed exactly once; 124 passed, one failed (iOS runner: isolate collapsed-tab occupancy test from prerequisite timeouts #3800). Worker exits 65/0 propagated; cleanup passed. Later CLI/replay/E2E steps were skipped.
• Hosted targeted step: 23m02s, plus 3m34s second-simulator boot. Earlier serial run at 9beae21eaf: 21m40s for 124 methods. Different commits/runs prevent a controlled speedup claim. This experiment shows no hosted benefit.
• Coverage, Integration Tests, repo guards, and all other executable checks passed. Cache matrix skipped; review bot neutral. Not merge-ready: iOS runner: isolate collapsed-tab occupancy test from prerequisite timeouts #3800 remains open. Do not adopt this configuration based on the local pilot.

[github-actions]

Size Report

Metric	Base	Current	Diff
Installed (including dependencies)	5.41 MB	5.41 MB	0 B
Package (unpacked)	5.41 MB	5.41 MB	0 B
Package (download)	1.63 MB	1.63 MB	+3 B

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	40.4 ms	39.8 ms	-0.7 ms
CLI --help	75.5 ms	73.1 ms	-2.4 ms

[thymikee]

Hosted experiment result at 027b43734b26134459be39310eeef0e61a1fd1ab: no demonstrated CI speed benefit; iOS remains blocked.

The iOS job confirms both independent simulators booted and both workers finished:

• Shard 0: 63 methods, one failure, native exit 65.
• Shard 1: 62 methods, zero failures, native exit 0.
• Union: 125/125 selected methods exactly once, 124 passed, one failed, zero skipped. The failing native exit was preserved after waiting for both workers. Simulator cleanup, selection enforcement, artifact upload, and host-focus check passed.
• The sole assertion is testPlanMovesPastATreeTierWhoseCollapsedTabQueryCannotStart, RunnerTests+SnapshotCapturePlanOccupancyTests.swift:529: actual capture-failed, expected budget. It matches the existing blocker iOS runner: isolate collapsed-tab occupancy test from prerequisite timeouts #3800, also seen in the earlier serial test(ios): stabilize text clearing and snapshot budget regressions #3802 run. This is not an established infrastructure fault, so I did not rerun it.

The performance result does not reproduce the local pilot:

Measurement	Hosted two-shard run	Earlier serial run
Targeted XCTest step	23m02s	21m40s
Additional simulator boot	3m34s	none
Completed methods	125	124
Common 124 methods, summed case durations	1963.278s	1120.270s

Serial reference: 9beae21eaf, run 39151402218. Both used Xcode 26.6 / iOS 26.5, but these are different commits, hosts/runs, and selections; this is an observational comparison, not a controlled estimate. Slower common test bodies are consistent with contention, but do not prove its cause.

Recommendation: do not merge the same-host two-simulator configuration on this evidence. A further parallelization experiment should compare shards on separate runners against a serial run at the same source SHA, including setup cost. Keep the #3800 fixture fix separate; do not change assertions, timeouts, or selection to make this experiment green.

Coverage, Integration Tests, Repo Guards, Android/Linux/macOS smoke, and all other executable checks passed at this head. The cache matrix is skipped and the review bot neutral. Downstream iOS CLI/replay/E2E checks were skipped after XCTest failed, so they are not validated. The PR remains ready for review and is not merge-ready.

[thymikee]

The code is sound for an experiment, but the hosted run at 027b437 shows no gain, so please do not merge this as is. The sharded lane took 23m02s, plus 3m34s to boot the second simulator, against 21m40s for the earlier serial run of 124 methods. The two shards still took 944.8s and 1034.2s of test-case time. Both simulators share one hosted macOS runner, so the local 39.7% gain (warm, 12 tests, 12-core host) did not carry over. Merging would make the required iOS PR lane about 5 minutes slower, add a second simulator to create and delete, and add roughly 80 lines of shell around https://github.com/callstack/agent-device/blob/027b437/.github/workflows/ios.yml#L140. Please keep this as an experiment, and close it or hold it until a controlled A/B on the same commit and runner shows a real wall-time gain. If you keep it, the rule should be that the shard count and its per-shard resources (simulator, log, exit file, result bundle) all come from one SHARD_COUNT-driven loop.

Not blocking, take or leave: the shard count 2 is spelled out at five sites in https://github.com/callstack/agent-device/blob/027b437/.github/workflows/ios.yml#L175 (both for SHARD_INDEX in 0 1 loops, % 2, the SIMULATORS array and the shard-0.log shard-1.log arguments), so one SHARD_COUNT or SIMULATORS array should derive them all; the header of https://github.com/callstack/agent-device/blob/027b437/scripts/xctest-run-summary.ts#L1 still says "Two jobs in one place" and "Reads xcresulttool ... JSON", yet the new mode is a third job that scrapes Test Case lines from the xcodebuild log, so a log format change would give "shard executed no identifiable methods" instead of a parse error, and if this ever merges it should read identities from xcresulttool get test-results tests on the existing shard-N.xcresult bundles; and https://github.com/callstack/agent-device/blob/027b437/docs/evidence/ios-xctest-sharding-pilot-2026-10-18.md#L57 says hosted timing "still need CI evidence", but that run now exists and shows no benefit, so the doc should record it (23m02s plus 3m34s boot against 21m40s) or stay in the PR body, and its fault-injection table has no script or test in the repo.

Would it be simpler to land nothing and keep the serial lane? If faster PR feedback is still wanted, could we cut the work first, by shortening the PR-reach list or moving slow occupancy tests to the nightly lane? Or run the A/B on one commit and runner pair, and adopt sharding only if step time plus boot time wins by a clear margin, with one SHARD_COUNT array and identities read from the existing xcresult summary path so the exact-once check extends livenessFailure. Fixing #3800 comes first, so the lane can be measured green on both shapes, then a same-commit hosted A/B that includes the second simulator boot.

The only failing check is Smoke Tests, on testPlanMovesPastATreeTierWhoseCollapsedTabQueryCannotStart (expected 'budget', got 'capture-failed' at RunnerTests+SnapshotCapturePlanOccupancyTests.swift:529). It is the same assertion #3800 records on three other heads, and this diff touches no Swift file, so it is likely unrelated. Running two simulators on one host could raise the flake rate if #3800 is a timing issue, but that is not shown. The other 16 checks pass. I did not re-run the fault-injection cases or the unit tests, and I could not compare timings across different commits and runs, as you note. Until #3800 is fixed and the hosted run shows a gain, this should not merge.

[thymikee]

Closing this experiment as requested. The hosted run showed no benefit: 23m02s for XCTest plus 3m34s for the additional simulator boot, versus the earlier 21m40s serial run. The runs were not a controlled same-SHA comparison, so they do not establish causation; they also do not support adopting same-host sharding. Keep the existing serial lane. The performance and exact-coverage evidence remains in the PR discussion. The #3800 fixture fix is being handled separately in #3802.

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-10-19 05:49 UTC