Skip to content

Clean up UI tests infrastructure - #25673

Merged
crazytonyli merged 4 commits into
trunkfrom
task/clean-up-ui-tests
Jun 22, 2026
Merged

crazytonyli merged 4 commits into
trunkfrom
task/clean-up-ui-tests

Conversation

@crazytonyli

Copy link
Copy Markdown
Contributor

Note

I recommend reviewing commit by commit.

The UI tests were removed in #25399, but there are some surrounding areas (mocks, screen objects, etc) left in the codebase.

The screenshot generation pipeline uses UI tests. This PR removes the screenshot generation automation. We can figure out a new way to do it next time we need new screenshots. I imagine it'd be easier to use LLM for that than UI tests + mocks.

UI tests were removed in #25399. This drops the launch-argument flags that nothing passes anymore (logging, prompts, migration, autofill, animations, mock data, screenshot generation) along with their app-side conditionals, reducing UITestConfigurator to the simulator sign-in reset path. The sign-in launch arguments are unaffected.
The WordPress and Jetpack screenshot-generation targets are XCUITest targets that relied on the now-removed UI-test infrastructure. This deletes the targets, their schemes and sources, and the fastlane raw-screenshot lanes, while keeping the promo-screenshot composition and GlotPress download lanes.
With the screenshot-generation targets gone, nothing consumes the UITestsFoundation module or its XcodeTarget wrappers. This removes them and the ScreenObject package dependency they pulled in.
The WireMock mock server, its rake task, the Buildkite UI-test runner, and the Jetpack UI-test branch of the test_without_building lane only supported the removed UI tests. This deletes them along with the obsolete UI-tests documentation.
@crazytonyli crazytonyli added this to the 27.1 milestone Jun 19, 2026
@crazytonyli
crazytonyli requested review from a team and jkmassel June 19, 2026 01:14
@dangermattic

Copy link
Copy Markdown
Collaborator
1 Warning
⚠️ This PR is larger than 500 lines of changes. Please consider splitting it into smaller PRs for easier and faster reviews.

Generated by 🚫 Danger

@crazytonyli crazytonyli changed the title Clean UI tests infrastructure Clean up UI tests infrastructure Jun 19, 2026
@wpmobilebot

Copy link
Copy Markdown
Contributor
App Icon📲 You can test the changes from this Pull Request in WordPress by scanning the QR code below to install the corresponding build.
App NameWordPress
ConfigurationRelease-Alpha
Build Number32719
VersionPR #25673
Bundle IDorg.wordpress.alpha
Commitc0fcbc5
Installation URL5d2qe8hi5psig
Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

@wpmobilebot

Copy link
Copy Markdown
Contributor
App Icon📲 You can test the changes from this Pull Request in Jetpack by scanning the QR code below to install the corresponding build.
App NameJetpack
ConfigurationRelease-Alpha
Build Number32719
VersionPR #25673
Bundle IDcom.jetpack.alpha
Commitc0fcbc5
Installation URL3fh3e9eiflrsg
Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

@mokagio mokagio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bitter sweet, but no point in carrying around unused code such as this.

On that note, I had a look locally and found this still in the branch:

Going by how the code checks for launch arguments that are no longer in the repo, I think it can be removed.

Additionally, there's this code in the app delegate:

// MARK: - UI Test Support
extension WordPressAppDelegate {
func autoSignInUITestSite() {
guard let wpComSiteAddress = UserDefaults.standard.string(forKey: "ui-test-select-wpcom-site") else {
return
}
let service = WordPressComSyncService()
service.syncWPCom(
authToken: "valid_token",
isJetpackLogin: false,
onSuccess: { _ in
if let blog = try? BlogQuery().hostname(containing: wpComSiteAddress)
.blog(in: ContextManager.shared.mainContext)
{
self.windowManager.showUI(for: blog)
} else {
fatalError("Can't find blog: \(wpComSiteAddress)")
}
},
onFailure: {
fatalError("Can't sync blogs: \($0)")
}
)
}
}

The option in the first guard is also unused, which again suggests this can be removed.

If my assumptions are correct, I opened #25679 to merge the removals against this.

This PR leaves the screenshots composition lanes in place. I say we could get rid of them as well (and of rmagick with them!). The same rational of using AI to help if the time comes to generate more screenshots applies to that part of the automation. And, as always, Git remembers.


Update: I got AI to investigate the validity of the testing skill after this update and #25679 and it turns out that the latter would require updating the skill as well. I did not implement those updates.

@crazytonyli up to you whether to remove the additional UI tests references or not. If you decide to keep them, then I'd suggest maybe tidying the code so that it's clearer it's not really about UI testing in the XCUITest sense.

@mokagio mokagio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant to approve earlier.

@crazytonyli

Copy link
Copy Markdown
Contributor Author

Yes, the type names should be updated now. But I wanted to keep the addition number in the diff low, to make reviewing easier. I can do that in a future PR.

@crazytonyli
crazytonyli added this pull request to the merge queue Jun 22, 2026
Merged via the queue into trunk with commit e07839a Jun 22, 2026
24 checks passed
@crazytonyli
crazytonyli deleted the task/clean-up-ui-tests branch June 22, 2026 09:01
jkmassel added a commit that referenced this pull request Oct 6, 2026
The 37 fixture-backed tests took 954 seconds on one Simulator; they now
take 398.

- Poll for elements instead of using XCTest's waits. `waitForExistence`
  and `wait(for:toEqual:timeout:)` check once a second and not before the
  first second is up, so each took a second even when the element was
  already there, and a test makes dozens of them.
- Launch the app with `-ui-test-disable-animations`, which turns off
  UIKit's animations. XCTest waits for them before every tap and query.
  The argument is the one #25673 removed when nothing passed it.
- Send analytics events once a second when the app runs against the
  fixtures, instead of every 15 seconds. The Stats suite slept 17 seconds
  per test to be sure of having them.
- Stop the app asking to send push notifications, with a launch argument,
  instead of dismissing the question. The sheet appears a moment after the
  list does, so a faster test could get to the list first.

The suites that run against a real account keep the second before a wait's
first check. Their content arrives over the network after its screen does,
and without that second five of the 79 tests failed on lists that were
still loading. Three of those dependencies are fixed here, where the cause
was clear:

- Account Settings is empty until its request returns, so the assertions
  that something is shown now wait for it.
- The Social list has its "Connect a New Account" button before it has
  the connections, so "the first row" opened that instead.
- The Insights tab's first card loads after the tab appears, and scrolling
  before then scrolled past where it was about to be.
jkmassel added a commit that referenced this pull request Oct 6, 2026
XCTest waits for the app's animations to finish before every tap and
query. Launching with `-ui-test-disable-animations`, which calls
`UIView.setAnimationsEnabled(false)`, takes the 37 fixture-backed tests
from 519 seconds to 402 on one Simulator. Each of them passed three
times in a row with it, and a swipe scrolls a list just as far.

This is the argument #25673 removed when nothing passed it. An earlier
commit here added it back and the next took it out again, over first-test
failures on CI that turned out to be the cold first launch, which has
its own wait now.
jkmassel added a commit that referenced this pull request Oct 6, 2026
The WireMock mappings the removed UI tests used, restored from before #25673
deleted them, with stats responses that follow today's date.
jkmassel added a commit that referenced this pull request Oct 6, 2026
The 37 fixture-backed tests took 954 seconds on one Simulator; they now
take 398.

- Poll for elements instead of using XCTest's waits. `waitForExistence`
  and `wait(for:toEqual:timeout:)` check once a second and not before the
  first second is up, so each took a second even when the element was
  already there, and a test makes dozens of them.
- Launch the app with `-ui-test-disable-animations`, which turns off
  UIKit's animations. XCTest waits for them before every tap and query.
  The argument is the one #25673 removed when nothing passed it.
- Send analytics events once a second when the app runs against the
  fixtures, instead of every 15 seconds. The Stats suite slept 17 seconds
  per test to be sure of having them.
- Stop the app asking to send push notifications, with a launch argument,
  instead of dismissing the question. The sheet appears a moment after the
  list does, so a faster test could get to the list first.

The suites that run against a real account keep the second before a wait's
first check. Their content arrives over the network after its screen does,
and without that second five of the 79 tests failed on lists that were
still loading. Three of those dependencies are fixed here, where the cause
was clear:

- Account Settings is empty until its request returns, so the assertions
  that something is shown now wait for it.
- The Social list has its "Connect a New Account" button before it has
  the connections, so "the first row" opened that instead.
- The Insights tab's first card loads after the tab appears, and scrolling
  before then scrolled past where it was about to be.
jkmassel added a commit that referenced this pull request Oct 6, 2026
XCTest waits for the app's animations to finish before every tap and
query. Launching with `-ui-test-disable-animations`, which calls
`UIView.setAnimationsEnabled(false)`, takes the 37 fixture-backed tests
from 519 seconds to 402 on one Simulator. Each of them passed three
times in a row with it, and a swipe scrolls a list just as far.

This is the argument #25673 removed when nothing passed it. An earlier
commit here added it back and the next took it out again, over first-test
failures on CI that turned out to be the cold first launch, which has
its own wait now.
jkmassel added a commit that referenced this pull request Oct 7, 2026
The WireMock mappings the removed UI tests used, restored from before #25673
deleted them, with stats responses that follow today's date.
jkmassel added a commit that referenced this pull request Oct 7, 2026
The 37 fixture-backed tests took 954 seconds on one Simulator; they now
take 398.

- Poll for elements instead of using XCTest's waits. `waitForExistence`
  and `wait(for:toEqual:timeout:)` check once a second and not before the
  first second is up, so each took a second even when the element was
  already there, and a test makes dozens of them.
- Launch the app with `-ui-test-disable-animations`, which turns off
  UIKit's animations. XCTest waits for them before every tap and query.
  The argument is the one #25673 removed when nothing passed it.
- Send analytics events once a second when the app runs against the
  fixtures, instead of every 15 seconds. The Stats suite slept 17 seconds
  per test to be sure of having them.
- Stop the app asking to send push notifications, with a launch argument,
  instead of dismissing the question. The sheet appears a moment after the
  list does, so a faster test could get to the list first.

The suites that run against a real account keep the second before a wait's
first check. Their content arrives over the network after its screen does,
and without that second five of the 79 tests failed on lists that were
still loading. Three of those dependencies are fixed here, where the cause
was clear:

- Account Settings is empty until its request returns, so the assertions
  that something is shown now wait for it.
- The Social list has its "Connect a New Account" button before it has
  the connections, so "the first row" opened that instead.
- The Insights tab's first card loads after the tab appears, and scrolling
  before then scrolled past where it was about to be.
jkmassel added a commit that referenced this pull request Oct 7, 2026
XCTest waits for the app's animations to finish before every tap and
query. Launching with `-ui-test-disable-animations`, which calls
`UIView.setAnimationsEnabled(false)`, takes the 37 fixture-backed tests
from 519 seconds to 402 on one Simulator. Each of them passed three
times in a row with it, and a swipe scrolls a list just as far.

This is the argument #25673 removed when nothing passed it. An earlier
commit here added it back and the next took it out again, over first-test
failures on CI that turned out to be the cold first launch, which has
its own wait now.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants