Repository navigation
test(ios): establish snapshot convergence baselines and permanent evidence #2189
Copy link
Copy link
Closed
Labels
Description
Parent: #2188
Outcome
Check in the reproducible evidence needed to judge every later iOS snapshot-convergence change.
A third party must be able to produce current-HEAD local, proxy, compute, and package-size baselines
from documented commands without reconstructing artifacts from an investigation session.
Exact scope
- Add benchmark definitions, a versioned raw-result schema, and one documented reproduction entry
point under repository tooling rather than a runtime package. - Measure current runner-backed iOS Simulator snapshots in explicitly named states:
- cold-cold: Simulator off, daemon off, no prepared runner/bridge cache;
- cold: Simulator booted, daemon/helper processes absent;
- warm: Simulator, app, daemon, runner, and target generation ready;
- relaunch: same Simulator/tooling with a new app process/generation.
- Include representative quiet, list, nested-scroll, alert/system-surface, and XCTest-stress screens.
- Report daemon-resident request time and fresh-process CLI wall time separately.
- Record direct-proxy baselines for persistent-client and fresh-process CLI execution at controlled
approximately 0 ms, 20 ms, and 80 ms RTT. Record configured bandwidth/packet loss, response
bytes, median, p95, failures, and raw samples. - Record current packed, clean-installed, and bundled npm sizes using the repository's existing size
workflow. - Promote iOS: skip private-AX deep extension on observation-only captures (settle, stabilization, corroboration) #1626's deep-button false-failure into permanent, implementation-independent fixture
evidence: a shallow surface observation must not prove that a full action had no effect when the
changed descendant was omitted. - Link App-mount race on fast/cold opens: first capture can see an unreadable or empty tree (cross-platform) #1571 in the result schema and classify an unreadable/empty first tree separately from bridge,
runner, timeout, and stale-generation failures.
Use at least 20 samples per warm cell and 10 per cold cell unless the checked-in harness documents a
larger statistically justified minimum. Store raw samples; summaries alone are insufficient.
Non-goals
- No production routing, snapshot semantics, bridge, runner-demand, or fallback change.
- Do not implement iOS: skip private-AX deep extension on observation-only captures (settle, stabilization, corroboration) #1626 or expose acquisition intent publicly.
- Do not claim the investigation's provisional numbers as reproduced until this issue's commands
produce them on current HEAD. - No physical-iPhone measurement; that track is separate.
Acceptance and evidence
- One documented command reproduces each local measurement cell and writes schema-valid raw output.
- One documented command reproduces each proxy cell without changing application semantics.
- Repeated runs identify machine/toolchain, package revision, screen fixture, target generation,
sample count, failures, and outliers. - The deep-button fixture is observed failing under the invalid shallow-corroboration rule and green
under the current safe/full requirement. Record the exact planted-red command and failing assertion. - Package-size output reports packed, installed, and bundled values from the same revision.
- Focused tooling/schema tests and
pnpm check:affected --runpass. Native and proxy measurements
are attached as evidence rather than represented by fixture-only tests.
Worker stop conditions
- Stop rather than substitute ad-hoc timing if a required state cannot be produced deterministically.
- Report infrastructure contention separately; do not convert changing timeout sets into product
regressions without an origin/main control under the same load. - Do not modify or close iOS: skip private-AX deep extension on observation-only captures (settle, stabilization, corroboration) #1626 or App-mount race on fast/cold opens: first capture can see an unreadable or empty tree (cross-platform) #1571.
Reactions are currently unavailable
Activity