Skip to content

Add UI tests for the Jetpack app, with a fixtures backend that runs in CI - #26119

Open
jkmassel wants to merge 37 commits into
jkmassel/fix-capabilities-callback-priorityfrom
jkmassel/ui-tests-targets-structure
Open

jkmassel wants to merge 37 commits into
jkmassel/fix-capabilities-callback-priorityfrom
jkmassel/ui-tests-targets-structure

Conversation

@jkmassel

@jkmassel jkmassel commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Adds a JetpackUITests target with 116 UI tests for the Jetpack app, replacing the tests removed in #25399. 37 of them run against HTTP fixtures and now run in CI; the other 79 sign in to a real WordPress.com account and are for running locally.

Stacked on #26121. The branch also carries the 13 commits of #25801, which adds the -wpcom-token launch argument these tests sign in with and hasn't merged yet; they drop out of this diff when it does.

What's in it

  • Add the JetpackUITests target and test plan to the Jetpack scheme, in Tests/JetpackUITests. The scheme's test action still pointed at the target Remove UI tests #25399 deleted. Tests are written against screen objects, and JetpackUITestCase launches the app already signed in.
  • Add HTTPFixtures, a module with a URLProtocol that answers every URLSession request in the app from WireMock-style stub mappings and can log each request it sees. The app installs it when it's launched with -ui-test-http-fixtures.
  • Add 185 fixture files: the WireMock mappings the removed tests ran against, restored from before Clean up UI tests infrastructure #25673 deleted them, with stats responses that are generated relative to today's date.
  • Add 8 suites that run against the fixtures (37 tests): the account and site picker, Me, Notifications, the Reader, Stats, liking a post, analytics events and request counts.
  • Add 7 suites that run against a real account (79 tests): My Site, the dashboard cards, the site menu and the lists behind it, the editor, and Stats.
  • Add accessibility identifiers to the screens the tests reach: 28 lines across 15 files. The Stats chart's metric tabs also gain the .isSelected trait, which they didn't expose.
  • Add a "UI Tests" CI step that depends on the existing Jetpack build for testing and runs the fixture-backed suites. test_without_building infers the scheme from the xctestrun name again, as it did before Clean up UI tests infrastructure #25673, and takes an only_testing list.
  • Add docs/ui-tests.md, which covers running the tests, writing one, writing a fixture, and what the two kinds of suite do and don't check.

Why two backends

Some screens change the account just by being shown. Opening the Notifications tab marks every notification as seen, opening one marks it as read, and opening a post in the Reader counts as a view of it. Those can't be tested against a real account without altering it, so Notifications, the Reader and Me run against the fixtures.

The fixture account never changes, which also lets those suites assert on content and on what the app sends. ReaderActionTests likes a post, then checks that the app sent POST /rest/v1.1/sites/70135762/posts/125073/likes/new exactly once and that the post then appears in the Likes stream.

The real-account suites check the opposite half: that the app works against live responses. They assert that screens open and controls change state, never on content, so they aren't tied to one account. When the account's site has nothing to open (no recent activity, no referrers this week), a test is skipped with a message naming what was missing rather than failed.

What a reviewer will want to know

The interceptor can't reroute requests in a release build. FixtureURLProtocol and the call that installs it are inside #if DEBUG.

The app-side changes are identifiers and one trait. Nothing else in the app's screens changes. The trait means the selected Stats metric tab now reports itself as selected to assistive technology; That was confirmed in the accessibility hierarchy, not with VoiceOver itself.

CI needs no token or secret. The script picks the suites that declare override class var backend: Backend { .fixtures }, so a new fixture-backed suite is included without editing the pipeline, and a real-account suite never runs there.

The "UI Tests" step runs as one CI job per area, five in all. Its commit status isn't a required check.

Some taps retry. On CI's iOS 27.0 simulators, soon after a launch, XCUITest sometimes can't compute an element's hit point and doesn't tap at all. Opening a tab, the site picker or a quick action on My Site taps the middle of the element until its screen appears, up to three times. We haven't found why the hit point can't be computed.

Not in this PR

  • Running the real-account suites in CI. That needs a WordPress.com account kept for testing, and its token as a secret.
  • Uploading UI test results to Test Engine.
  • A few screens the fixtures can't serve yet, which show an error state under them and so have no test: the Reader's Discover stream, a post's comments, Notification Settings, and the "new post" notification.

Test plan

  • All 37 fixture-backed tests pass on CI (#36351), and locally on an iPhone 18 Pro simulator running iOS 27.2. The Reader suites also passed three runs in a row.
  • HTTPFixturesTests passes with swift test --filter HTTPFixturesTests: 52 tests in 5 suites, including one that loads every fixture file.
  • The 79 real-account tests ran against a WordPress.com account on an iOS 27.2 simulator: 78 passed and none were skipped. That run came before the last three commits, which change how tabs, the site picker and quick actions are opened, and what the Help test expects. MySiteTests and the Locations test passed again after them. StatsTests.testTopListCardMenu failed once, not finding the card menu's Delete Card item, then passed three runs in a row.
  • The request assertion in ReaderActionTests fails when it should: a copy that waited for a different endpoint failed and listed the requests the app had sent.
  • .buildkite/pipeline.yml passes Buildkite's schema validation.
  • A fixture-backed suite runs locally with no token: xcodebuild -workspace WordPress.xcworkspace -scheme Jetpack -testPlan JetpackUITests -destination 'platform=iOS Simulator,name=iPhone 18 Pro' test -only-testing:JetpackUITests/ReaderActionTests SWIFT_ACTIVE_COMPILATION_CONDITIONS='$(inherited) UI_TEST_HTTP_FIXTURES' ended with Executed 1 test, with 0 failures. The last argument compiles the fixtures in; docs/ui-tests.md explains why the command line is the only place to set it.
  • A real-account suite runs locally: the same command with -only-testing:JetpackUITests/MySiteTests and no last argument passed all 5 tests, none skipped. It needs a WordPress.com token in ~/.wpcom-token and signs the simulator in to that account. These suites only read; the ones that open the editor close it without saving. Without a token the suite reports 5 skipped tests.

@jkmassel jkmassel added Testing Unit and UI Tests and Tooling Tooling Build, Release, and Validation Tools labels Oct 5, 2026
@jkmassel jkmassel self-assigned this Oct 5, 2026
@jkmassel jkmassel added this to the 27.4 milestone Oct 5, 2026
@dangermattic

dangermattic commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator
4 Warnings
⚠️ Modules/Package.swift was changed without updating its corresponding Package.resolved.

If the change includes adding, removing, or editing a dependency please resolve the Swift packages as appropriate to your project setup (e.g. in Xcode or by running swift package resolve).

If the change to the Package.swift did not modify dependencies, ignoring this warning should be safe, but we recommend double checking and running the package resolution just in case.
.

⚠️ Package.swift was changed without updating its corresponding Package.resolved.

If the change includes adding, removing, or editing a dependency please resolve the Swift packages as appropriate to your project setup (e.g. in Xcode or by running swift package resolve).

If the change to the Package.swift did not modify dependencies, ignoring this warning should be safe, but we recommend double checking and running the package resolution just in case.
.

⚠️ View files have been modified, but no screenshot or video is included in the pull request. Consider adding some for clarity.
⚠️ This PR is larger than 500 lines of changes. Please consider splitting it into smaller PRs for easier and faster reviews.

Generated by 🚫 Danger

@wpmobilebot

wpmobilebot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor
App Icon📲 You can test the changes from this Pull Request in Jetpack by scanning the QR code below to install the corresponding build.
App NameJetpack
ConfigurationRelease-Alpha
Build Number34854
VersionPR #26119
Bundle IDcom.jetpack.alpha
Commitb12ef60
Installation URL3fgg14jcsh5h8
Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

@wpmobilebot

wpmobilebot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor
App Icon📲 You can test the changes from this Pull Request in WordPress by scanning the QR code below to install the corresponding build.
App NameWordPress
ConfigurationRelease-Alpha
Build Number34854
VersionPR #26119
Bundle IDorg.wordpress.alpha
Commitb12ef60
Installation URL77kfsghcf6f0g
Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

@wpmobilebot

wpmobilebot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

🤖 Build Failure Analysis

This build has failures. Claude has analyzed them - check the build annotations for details.

@jkmassel
jkmassel force-pushed the jkmassel/ui-tests-targets-structure branch from 8c4b66c to 06ef515 Compare October 6, 2026 17:39
@jkmassel
jkmassel changed the base branch from jkmassel/fix-urlsession-lazy-race to jkmassel/fix-capabilities-callback-priority October 6, 2026 17:39
@jkmassel
jkmassel force-pushed the jkmassel/fix-capabilities-callback-priority branch from 3b2d9a8 to f02c27a Compare October 7, 2026 02:26
@jkmassel
jkmassel force-pushed the jkmassel/ui-tests-targets-structure branch from 06ef515 to b12ef60 Compare October 7, 2026 02:26
@wpmobilebot wpmobilebot modified the milestones: 27.4, 27.5 Oct 20, 2026
@wpmobilebot

Copy link
Copy Markdown
Contributor

Version 27.4 has now entered code-freeze, so the milestone of this PR has been updated to 27.5.

@jkmassel
jkmassel force-pushed the jkmassel/ui-tests-targets-structure branch from b12ef60 to 8c4fae5 Compare November 6, 2026 20:27
@jkmassel
jkmassel force-pushed the jkmassel/fix-capabilities-callback-priority branch from f02c27a to 87a9fca Compare November 6, 2026 20:27
@wpmobilebot wpmobilebot modified the milestones: 27.5, 27.6 Nov 20, 2026
@wpmobilebot

Copy link
Copy Markdown
Contributor

Version 27.5 has now entered code-freeze, so the milestone of this PR has been updated to 27.6.

@jkmassel
jkmassel force-pushed the jkmassel/ui-tests-targets-structure branch from 8c4fae5 to 99d5b33 Compare November 20, 2026 23:13
@jkmassel
jkmassel force-pushed the jkmassel/fix-capabilities-callback-priority branch from 87a9fca to 1781b55 Compare November 20, 2026 23:13
Add `Scripts/sim-signin.sh`, a one-command wrapper around `xcrun simctl
launch` that signs a Simulator into WordPress.com with a bearer token, and
make the existing `-ui-test-wpcom-token` launch argument finish sign-in
automatically instead of requiring a tap on "Continue with WordPress.com".

`autoSignInUITestWPComAccountIfNeeded()` mirrors the existing
`autoSignInUITestSite()` hook: it runs from `showSignInUI()`, no-ops unless
the token argument is present and no account is signed in, then runs the
real sign-in flow and swaps the window root with `windowManager.showUI()`.
Add a named `-t, --wpcom-token <token>` option alongside the positional
form, and reject a missing option value instead of shifting past the end.
Introduce `WordPressDotComAuthenticator.launchArgumentToken`, which reads
the new `-wpcom-token` launch argument and falls back to the legacy
`-ui-test-wpcom-token`. Route both read sites — `attemptSignIn` and
`autoSignInUITestWPComAccountIfNeeded()` — through it, and switch the helper
script to pass `-wpcom-token`.
The launch argument is `-wpcom-token`, not UI-test-scoped, so
`autoSignInUITestWPComAccountIfNeeded()` was misleading. Rename it to
`autoSignInWPComAccountFromLaunchArgumentIfNeeded()`.
So the token can be set once instead of pasted every launch, sim-signin.sh
falls back to the WPCOM_TOKEN environment variable, then a ~/.wpcom-token
file, when neither --wpcom-token nor a positional token is given.
Precedence: flag > env var > file.
Wraps `Scripts/sim-signin.sh` behind `make sim-login`, forwarding optional
`DEVICE`, `APP`, and `RESET=1` variables plus an `ARGS` passthrough. The
token still resolves from `--wpcom-token`, `WPCOM_TOKEN`, or `~/.wpcom-token`.
The site menu rows, the dashboard cards, their headers and menu buttons, the
Stats cards and date controls, domain rows, and the subscriber and user
details. The chart's metric tabs also gain the selected trait, which they
were missing for VoiceOver.
A URLProtocol, compiled into debug builds only, that answers every
URLSession request from WireMock-style stub mappings and can log each
request it sees. The app installs it when it's launched with
-ui-test-http-fixtures.
The target replaces the UI tests removed in #25399. JetpackUITestCase
launches the app signed in with the -wpcom-token launch argument, against
either a real WordPress.com account or the fixtures.
The WireMock mappings the removed UI tests used, restored from before #25673
deleted them, with stats responses that follow today's date.
My Site, the dashboard cards, the site menu and its lists, the editor, and
Stats. They check that screens open and controls change state, and skip
rather than fail when the account's site has nothing to open.
The account, Me, Notifications, the Reader, Stats, liking a post, analytics
events and request counts. The fixture account never changes, so these
assert on what the screens show and on what the app sends.
A UI Tests step that depends on the Jetpack build for testing and runs the
suites that declare the fixtures backend, which need no token.
test_without_building infers the scheme from the xctestrun name again and
takes an only_testing list.
The module that answers the app's requests from fixtures, and the code in
the app that installs it, were compiled into every debug build. Now every
file in HTTPFixtures and the hook in UITestConfigurator are wrapped in
`#if UI_TEST_HTTP_FIXTURES`, so a build that doesn't set that condition has
none of it: the module compiles to an object with no code in it.

No configuration or scheme sets the condition. fastlane passes it to
xcodebuild as a build setting, which is the only way to reach a package
module as well as the app's targets: the `test` lane does for the Jetpack
scheme, and `build_jetpack_for_testing` does when given `http_fixtures:true`,
which is how CI now builds the app the UI tests run against.

A fixture-backed test fails as soon as the app has launched, with a message
naming the condition, when the app wasn't built with it. Without this the
app would send the test's requests to WordPress.com.

HTTPFixturesTests is compiled out with the module. CI's package tests set
the condition; without it the target reports one skipped test that says
how to run the rest. It leaves WordPressUnitTests.xctestplan, whose build
doesn't set the condition.

A SwiftLint rule fails a file in the module that doesn't start with the
condition.
The 37 fixture-backed tests took 954 seconds on one Simulator; they now
take 398.

- Poll for elements instead of using XCTest's waits. `waitForExistence`
  and `wait(for:toEqual:timeout:)` check once a second and not before the
  first second is up, so each took a second even when the element was
  already there, and a test makes dozens of them.
- Launch the app with `-ui-test-disable-animations`, which turns off
  UIKit's animations. XCTest waits for them before every tap and query.
  The argument is the one #25673 removed when nothing passed it.
- Send analytics events once a second when the app runs against the
  fixtures, instead of every 15 seconds. The Stats suite slept 17 seconds
  per test to be sure of having them.
- Stop the app asking to send push notifications, with a launch argument,
  instead of dismissing the question. The sheet appears a moment after the
  list does, so a faster test could get to the list first.

The suites that run against a real account keep the second before a wait's
first check. Their content arrives over the network after its screen does,
and without that second five of the 79 tests failed on lists that were
still loading. Three of those dependencies are fixed here, where the cause
was clear:

- Account Settings is empty until its request returns, so the assertions
  that something is shown now wait for it.
- The Social list has its "Connect a New Account" button before it has
  the connections, so "the first row" opened that instead.
- The Insights tab's first card loads after the tab appears, and scrolling
  before then scrolled past where it was about to be.
The suites move into a folder per area of the app (Me, MySite,
Notifications, Reader and Stats), and the UI Tests step becomes a matrix of
those folders, so the areas run at the same time on separate agents.

Every job fails if it finds a fixture-backed suite in a folder the matrix
doesn't list, so a new area can't be left out of CI unnoticed.

`test_without_building` also takes `concurrent_workers`, to spread suites
over clones of the Simulator. CI doesn't use it: clones start cold, and
with four starting at once the first test on each timed out.
On CI, with `-ui-test-disable-animations`, three of the five UI Tests jobs
failed their first test: the app had signed in and fetched the account's
sites, but stayed on the sign-in screen behind its progress indicator.
In one of them the next test then found My Site without its create
button. The same jobs passed their other tests, and the run before the
argument was added passed all 37.

Remove the argument and the code behind it. XCTest waiting for
animations costs the fixture-backed tests about two minutes in nine on
one Simulator, which is less than a job that has to be retried.

Also report one commit status for the UI Tests step. GitHub receives the
context as it's written in the pipeline, so "UI Tests ({{matrix}})"
arrived with the braces in it.
With a job per area, five of ten jobs over two builds failed their first
test and passed the rest: My Site hadn't appeared 60 seconds after the
app launched. The app was still on the sign-in screen, behind its
progress indicator, with the account and its sites already fetched.

CI runs each job on a Simulator it has just erased and booted, and the
first launch there is slow in a way no later one is: over 76 seconds to
sign in, against under 10. One job a build could get by on that; five
starting together can't.

Wait up to three minutes for My Site after the first launch of a run,
before the test starts its own 60-second wait.

The previous commit blamed these failures on `-ui-test-disable-animations`.
They happened again without it, so that was wrong, and the docs no longer
say so.
XCTest waits for the app's animations to finish before every tap and
query. Launching with `-ui-test-disable-animations`, which calls
`UIView.setAnimationsEnabled(false)`, takes the 37 fixture-backed tests
from 519 seconds to 402 on one Simulator. Each of them passed three
times in a row with it, and a swipe scrolls a list just as far.

This is the argument #25673 removed when nothing passed it. An earlier
commit here added it back and the next took it out again, over first-test
failures on CI that turned out to be the cold first launch, which has
its own wait now.
Two changes to how the test jobs set up.

The unit test job now names the Simulator it runs on: "iPhone 17 Pro
(26.5)", which is the one fastlane had been choosing for it. With a
device named, `test_without_building` doesn't resolve the Swift packages
or read the build settings first, which was 147 seconds of that job.

Both test jobs start booting their Simulator as their first step, so
that it boots while the job downloads its build and installs its gems.
A UI test job spent 53 seconds getting its Simulator ready once
everything else was. The lane takes `reset_simulator:false` so that it
doesn't erase the Simulator that's booting. A CI job runs in a VM made
for it, so there's nothing on the Simulator to erase.
@jkmassel
jkmassel force-pushed the jkmassel/fix-capabilities-callback-priority branch from 1781b55 to e70c5a9 Compare November 20, 2026 23:16
@jkmassel
jkmassel force-pushed the jkmassel/ui-tests-targets-structure branch from 99d5b33 to 239cada Compare November 20, 2026 23:16
The bell asks `/me` for its `has_unseen_notes` field since #26076, so
sign-in requests `/me` twice: once for the account, once for that one
field. The test counted both and failed. It now counts only the
requests that ask for the whole account.
Since #26372 the Saved stream is a SwiftUI list that loads from
`/wpcom/v2/read/saved/keys` and `/rest/v1.2/read/saved`. No fixture
answered either, so the stream showed an error, and `testSaved` looked
for the table the stream used to be.

Add fixtures for an account with no saved posts, and an accessibility
identifier on the list's empty state for the test to wait for.
The menu can appear before its navigation bar has either button. The
test then took the Done branch, and failed waiting for a Done button
that was never going to appear.
Since #26026 the site menu is a SwiftUI list: a collection view whose
rows are buttons carrying the row identifiers, inside cells that have
none. The list only creates the rows near the screen, so a row further
down is scrolled to before it exists.
Since #25961 the screen that lists all of the Locations card's items is
titled with the level the card shows, such as Countries, not Locations.
Since #26381, CI builds the app without its production credentials,
and without Zendesk's the Help screen offers the community forum, not
Zendesk support. The test now expects the forum rows in such a build,
and waits for the Version row, which every build shows.
@jkmassel
jkmassel marked this pull request as ready for review November 21, 2026 02:32
@jkmassel
jkmassel requested a review from a team as a code owner November 21, 2026 02:32
@jkmassel
jkmassel requested a review from crazytonyli November 21, 2026 02:32
On CI's iOS 27.0 simulators, soon after a launch, XCUITest sometimes
can't compute a tab's hit point and doesn't tap it, and the test then
fails waiting for the tab's screen. Opening a tab now taps the middle
of the tab's frame until the tab bar shows it selected, up to three
times. ScreenObject gains tap(_:until:), which does the same for any
element, for the App Store screenshots to use.
My Site is often the first screen a test taps after a launch, when on
CI's iOS 27.0 simulators a tap sometimes doesn't happen. Opening the
site picker or a quick action now taps until its screen appears, up to
three times.
@jkmassel
jkmassel force-pushed the jkmassel/ui-tests-targets-structure branch from 3bd7d91 to d41ce3d Compare November 21, 2026 02:58
A retried tap now shows up as a step, so a test run shows how often a
first tap didn't open its screen.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Testing Unit and UI Tests and Tooling Tooling Build, Release, and Validation Tools

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants