Skip to content

test(e2e): agent-device through a simlock worker on real devices - #509

Merged
V3RON merged 18 commits into
task/475from
task/477
Oct 13, 2026
Merged

V3RON merged 18 commits into
task/475from
task/477

Conversation

@V3RON

@V3RON V3RON commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Closes #477

Status

Implement: done Review: round 1 + one review of the hardware fix loop, passed (text fixes unreviewed) Hardware: pass (run 20261013-164634, 30/30) Gate: waiting for the stack
Slow lane: the maintainer allowed it for every PR of this run.
Next: gh stack sync, then merge-pr.sh 509 for stack #508

  • On a Mac with an iOS runtime, Apple silicon, at least 7 GiB of RAM and SIMLOCK_E2E_AGENT_DEVICE set as the toolchain row says: scripts/slow-e2e.sh e2e/slow-agent-device.test.ts passes every test in the iOS describe. Checked: run 20261013-164634, 15/15.
  • On a machine with emulator and adb, at least 12 GiB of RAM and SIMLOCK_E2E_AGENT_DEVICE set the same way: the same command passes every test in the Android describe. Checked: run 20261013-164634, 15/15.
  • With SIMLOCK_E2E_AGENT_DEVICE unset, the same command writes <log>.exit 0 and shows both describes skipped. Checked: run 20261013-150830.
  • With SIMLOCK_E2E_AGENT_DEVICE=/nonexistent, the same command fails, naming /nonexistent in the beforeAll failure. Checked: run 20261013-150900.

The real-device runs found five simlock defects, fixed here: the front refused the backend that connect proxy sends; it read agent-device's replies one level too high (result.data), which also affects #474's lease-end code; /health answered 503, so the CLI hid simlock's reason; a sharded test named devices by id, not name; upstream failures were not logged. The agent-device build also needs pnpm build:android.

Assumptions

  • Assumption: release-all ordering check — a session sample is a session list CLI call of about a second, so "the first no-session sample comes before the wipe" would fail on correct behaviour; the test asserts no sample that saw the session started after the wipe began, plus the first and last samples, which fails the same wrong behaviour.
  • Assumption: the running-command test allows 1 s between the CLI's exit and the wipe start — the CLI process exits a few ms after the daemon answers.
  • Assumption: JSON shapes of agent-device (data.sessions, data.devices[].id) and simlock record fields (requesterId, deviceId) are read as the code shows them; the green stage confirms them on hardware.
  • Assumption: each agent runs connect proxy --daemon-base-url <worker>/v1/agent-device once, with its own AGENT_DEVICE_STATE_DIR and token in env, and every command with --json; the output envelope {success,data,error} is read as agent-device's source shows it.
  • Assumption: waitForAgentDevice polls GET /v1/agent-device/health and accepts status 200 with ok: true; the front relays the daemon's own /health and adds no upstream field in code, so the spec's upstream check cannot be made.
  • Assumption: backend on allocate — accept only the backend connect proxy sends for the platform (ios-instance on iOS, android-instance on Android) and forward it unchanged; any other backend is still refused.

Review

Spec review: 2 blocking, 2 fixed, 1 notes. Code review: 4 blocking, 4 fixed, 3 notes. Claims review: 2 blocking, 2 fixed, 1 notes.
Rounds: 1. Tokens: 20.8M.

Fixed unreviewed: the round 1 test fixes, and the review of the fix loop's text fixes (run artifacts removed, .agent-device ignored, two tests, one comment). The fix loop's commits had no review per fix, at the maintainer's request; one review covered them all.

Rejected:

  • code: e2e/helpers/env.ts restart() has no timeout of its own — duplicate of a spec finding, fixed under it
  • spec: e2e/slow-agent-device.test.ts short sample gap stays green — not blocking: Assumption 1 accepts it
  • code: e2e/slow-agent-device.test.ts session list exit code unchecked — not blocking: weakens no stated check
  • claims: toolchain.md package.json names only a short hash — not blocking: still true via vendor/README.md
  • code: e2e/helpers/env.ts:384 restartSteps keeps running after the 90 s limit — not blocking: slow-lane helper, teardown only
  • code: src/agent-device/front/request-rules.ts:236 emulator name fallback may match another agent's emulator — not blocking: speculative

Written by an agent.

@V3RON

V3RON commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

Review notes

Not blocking, not verified. Each is one reviewer's claim.

  • spec: e2e/slow-agent-device.test.ts:375 a session ending within one ~1.25 s sample period after the wipe begins stays green (Assumption 1)
  • code: e2e/slow-agent-device.test.ts:359 no sample checks the session list exit code, so a failed sample counts as "no session"
  • claims: docs/internal/agent-rules/toolchain.md:77 says rev-parse HEAD prints the commit in package.json's @agent-device/proxy dependency; package.json holds a short hash, the full commit is in vendor/README.md
  • code: e2e/helpers/env.ts:384 when the 90 s restart limit wins, restartSteps keeps running; a late startDaemon can leave a daemon behind after teardown
  • code: src/agent-device/front/request-rules.ts:236 if agent-device cannot read an emulator's AVD name it falls back to the shared model name, so that --device name could match another agent's emulator

Written by an agent.

@V3RON
V3RON marked this pull request as ready for review October 13, 2026 15:26
@V3RON
V3RON added this pull request to stack #508 October 13, 2026 15:26
V3RON added 18 commits October 13, 2026 17:27
…low lane (#477)

Implements withSharedDaemon, makeAgent, newAgentToken, cleanupAgents and
waitForAgentDevice. 15 failing -> 15 unproven: no agent-device build here,
so the hardware run is left to the slow lane.
…n restart waits for exit; teardown kills leftover by marker; health wait (#477)
… backstop; 60 s agent CLI default; 90 s restart limit (#477)
…; e2e: pinned build check, fail fast on open, show health reply (#477)
…esult.data, as agent-device sends them; log an unusable or failed upstream reply (#477)
…SON; one progress line per test; last observed state on a free-wait timeout (#477)
…ble, so the CLI runs its first command and prints the reason (#477)
…as an open agent-device session requires; e2e: find a lease by its owner (#477)
…p check names a missing Android helper; the build step is documented (#477)
@V3RON
V3RON merged commit 744ea5c into main Oct 13, 2026
22 checks passed
@V3RON
V3RON deleted the task/477 branch October 13, 2026 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

End-to-end tests: agent-device through a simlock worker on real iOS and Android devices

1 participant