Repository navigation
Clean up UI tests infrastructure - #25673
Conversation
UI tests were removed in #25399. This drops the launch-argument flags that nothing passes anymore (logging, prompts, migration, autofill, animations, mock data, screenshot generation) along with their app-side conditionals, reducing UITestConfigurator to the simulator sign-in reset path. The sign-in launch arguments are unaffected.
The WordPress and Jetpack screenshot-generation targets are XCUITest targets that relied on the now-removed UI-test infrastructure. This deletes the targets, their schemes and sources, and the fastlane raw-screenshot lanes, while keeping the promo-screenshot composition and GlotPress download lanes.
With the screenshot-generation targets gone, nothing consumes the UITestsFoundation module or its XcodeTarget wrappers. This removes them and the ScreenObject package dependency they pulled in.
The WireMock mock server, its rake task, the Buildkite UI-test runner, and the Jetpack UI-test branch of the test_without_building lane only supported the removed UI tests. This deletes them along with the obsolete UI-tests documentation.
Generated by 🚫 Danger |
|
| App Name | WordPress | |
| Configuration | Release-Alpha | |
| Build Number | 32719 | |
| Version | PR #25673 | |
| Bundle ID | org.wordpress.alpha | |
| Commit | c0fcbc5 | |
| Installation URL | 5d2qe8hi5psig |
|
| App Name | Jetpack | |
| Configuration | Release-Alpha | |
| Build Number | 32719 | |
| Version | PR #25673 | |
| Bundle ID | com.jetpack.alpha | |
| Commit | c0fcbc5 | |
| Installation URL | 3fh3e9eiflrsg |
mokagio
left a comment
There was a problem hiding this comment.
Bitter sweet, but no point in carrying around unused code such as this.
On that note, I had a look locally and found this still in the branch:
Going by how the code checks for launch arguments that are no longer in the repo, I think it can be removed.Additionally, there's this code in the app delegate:
WordPress-iOS/WordPress/Classes/System/WordPressAppDelegate.swift
Lines 882 to 909 in b497299
The option in the first guard is also unused, which again suggests this can be removed.
If my assumptions are correct, I opened #25679 to merge the removals against this.
This PR leaves the screenshots composition lanes in place. I say we could get rid of them as well (and of rmagick with them!). The same rational of using AI to help if the time comes to generate more screenshots applies to that part of the automation. And, as always, Git remembers.
Update: I got AI to investigate the validity of the testing skill after this update and #25679 and it turns out that the latter would require updating the skill as well. I did not implement those updates.
@crazytonyli up to you whether to remove the additional UI tests references or not. If you decide to keep them, then I'd suggest maybe tidying the code so that it's clearer it's not really about UI testing in the XCUITest sense.
mokagio
left a comment
There was a problem hiding this comment.
I meant to approve earlier.
|
Yes, the type names should be updated now. But I wanted to keep the addition number in the diff low, to make reviewing easier. I can do that in a future PR. |
The 37 fixture-backed tests took 954 seconds on one Simulator; they now take 398. - Poll for elements instead of using XCTest's waits. `waitForExistence` and `wait(for:toEqual:timeout:)` check once a second and not before the first second is up, so each took a second even when the element was already there, and a test makes dozens of them. - Launch the app with `-ui-test-disable-animations`, which turns off UIKit's animations. XCTest waits for them before every tap and query. The argument is the one #25673 removed when nothing passed it. - Send analytics events once a second when the app runs against the fixtures, instead of every 15 seconds. The Stats suite slept 17 seconds per test to be sure of having them. - Stop the app asking to send push notifications, with a launch argument, instead of dismissing the question. The sheet appears a moment after the list does, so a faster test could get to the list first. The suites that run against a real account keep the second before a wait's first check. Their content arrives over the network after its screen does, and without that second five of the 79 tests failed on lists that were still loading. Three of those dependencies are fixed here, where the cause was clear: - Account Settings is empty until its request returns, so the assertions that something is shown now wait for it. - The Social list has its "Connect a New Account" button before it has the connections, so "the first row" opened that instead. - The Insights tab's first card loads after the tab appears, and scrolling before then scrolled past where it was about to be.
XCTest waits for the app's animations to finish before every tap and query. Launching with `-ui-test-disable-animations`, which calls `UIView.setAnimationsEnabled(false)`, takes the 37 fixture-backed tests from 519 seconds to 402 on one Simulator. Each of them passed three times in a row with it, and a swipe scrolls a list just as far. This is the argument #25673 removed when nothing passed it. An earlier commit here added it back and the next took it out again, over first-test failures on CI that turned out to be the cold first launch, which has its own wait now.
The WireMock mappings the removed UI tests used, restored from before #25673 deleted them, with stats responses that follow today's date.
The 37 fixture-backed tests took 954 seconds on one Simulator; they now take 398. - Poll for elements instead of using XCTest's waits. `waitForExistence` and `wait(for:toEqual:timeout:)` check once a second and not before the first second is up, so each took a second even when the element was already there, and a test makes dozens of them. - Launch the app with `-ui-test-disable-animations`, which turns off UIKit's animations. XCTest waits for them before every tap and query. The argument is the one #25673 removed when nothing passed it. - Send analytics events once a second when the app runs against the fixtures, instead of every 15 seconds. The Stats suite slept 17 seconds per test to be sure of having them. - Stop the app asking to send push notifications, with a launch argument, instead of dismissing the question. The sheet appears a moment after the list does, so a faster test could get to the list first. The suites that run against a real account keep the second before a wait's first check. Their content arrives over the network after its screen does, and without that second five of the 79 tests failed on lists that were still loading. Three of those dependencies are fixed here, where the cause was clear: - Account Settings is empty until its request returns, so the assertions that something is shown now wait for it. - The Social list has its "Connect a New Account" button before it has the connections, so "the first row" opened that instead. - The Insights tab's first card loads after the tab appears, and scrolling before then scrolled past where it was about to be.
XCTest waits for the app's animations to finish before every tap and query. Launching with `-ui-test-disable-animations`, which calls `UIView.setAnimationsEnabled(false)`, takes the 37 fixture-backed tests from 519 seconds to 402 on one Simulator. Each of them passed three times in a row with it, and a swipe scrolls a list just as far. This is the argument #25673 removed when nothing passed it. An earlier commit here added it back and the next took it out again, over first-test failures on CI that turned out to be the cold first launch, which has its own wait now.
The WireMock mappings the removed UI tests used, restored from before #25673 deleted them, with stats responses that follow today's date.
The 37 fixture-backed tests took 954 seconds on one Simulator; they now take 398. - Poll for elements instead of using XCTest's waits. `waitForExistence` and `wait(for:toEqual:timeout:)` check once a second and not before the first second is up, so each took a second even when the element was already there, and a test makes dozens of them. - Launch the app with `-ui-test-disable-animations`, which turns off UIKit's animations. XCTest waits for them before every tap and query. The argument is the one #25673 removed when nothing passed it. - Send analytics events once a second when the app runs against the fixtures, instead of every 15 seconds. The Stats suite slept 17 seconds per test to be sure of having them. - Stop the app asking to send push notifications, with a launch argument, instead of dismissing the question. The sheet appears a moment after the list does, so a faster test could get to the list first. The suites that run against a real account keep the second before a wait's first check. Their content arrives over the network after its screen does, and without that second five of the 79 tests failed on lists that were still loading. Three of those dependencies are fixed here, where the cause was clear: - Account Settings is empty until its request returns, so the assertions that something is shown now wait for it. - The Social list has its "Connect a New Account" button before it has the connections, so "the first row" opened that instead. - The Insights tab's first card loads after the tab appears, and scrolling before then scrolled past where it was about to be.
XCTest waits for the app's animations to finish before every tap and query. Launching with `-ui-test-disable-animations`, which calls `UIView.setAnimationsEnabled(false)`, takes the 37 fixture-backed tests from 519 seconds to 402 on one Simulator. Each of them passed three times in a row with it, and a swipe scrolls a list just as far. This is the argument #25673 removed when nothing passed it. An earlier commit here added it back and the next took it out again, over first-test failures on CI that turned out to be the cold first launch, which has its own wait now.


Note
I recommend reviewing commit by commit.
The UI tests were removed in #25399, but there are some surrounding areas (mocks, screen objects, etc) left in the codebase.
The screenshot generation pipeline uses UI tests. This PR removes the screenshot generation automation. We can figure out a new way to do it next time we need new screenshots. I imagine it'd be easier to use LLM for that than UI tests + mocks.