Repository navigation
Run the nightly integration tier on 2 xdist workers - #970
Merged
Merged
Conversation
The integration tier takes 59 min serially on ubuntu and 47 min on macOS, which leaves no room inside its 80 min budget once setup-proteus is slow. Its tests each write into their own temporary directory, so they run on one xdist worker per core, with work stealing to balance the few long runs against the many short schema tests.
With psutil installed, xdist's auto mode counts physical cores, which on a runner with hyperthreading is half its vCPUs. The logical mode uses every vCPU, and without psutil it falls back to the same affinity count as auto.
The ubuntu runner has 4 vCPUs on 2 physical cores. With 4 workers the CPU-bound tests slow down past their per-test timeouts, while xdist's auto mode (psutil physical cores) gives 2 workers on ubuntu and 3 on macOS and keeps both green. The job timeout moves to 90 min so a cold FWL_DATA download still fits next to the measured test time.
xdist's auto mode counts physical cores only when psutil imports, and falls back to the logical count otherwise, which oversubscribes the ubuntu runner. The develop extra now requests pytest-xdist with its psutil extra instead of relying on a transitive install, and the integration step prints the 25 slowest tests so the headroom to the per-test timeouts shows in every nightly.
On three workers the macOS runner brings the heaviest integration tests to 92 % of their timeouts, because the outgas cross-checks and the JANUS smoke test end up running together. The integration matrix now carries a worker count per OS: physical cores on ubuntu, two on macOS.
Both runners take a literal `-n 2` in place of the per-OS worker matrix, so the worker count no longer depends on psutil or on the runner topology, and the psutil extra goes. A test reads the integration-tier pytest command from the workflow and requires `-n 2 --dist worksteal`, and tests/AGENTS.md states that the tier runs on 2 workers.
The guard reads every -n and --dist value from the shell-split pytest command, so a second -n later in the line, or the -n2 and --numprocesses=2 spellings, are read the way pytest reads them.
The guard now parses the integration-tier command with pytest and pytest-xdist loaded, so clustered short flags such as -vn3, the -d shortcut for --dist=load, --maxprocesses and --tx are read as the nightly reads them. A parametrized test pins the helper on literal command lines.
The guard requires exactly one -n in the integration-tier command, followed by 2, so a later -n in the same command fails it.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #970 +/- ##
==========================================
- Coverage 93.97% 93.65% -0.33%
==========================================
Files 128 128
Lines 20891 20891
Branches 3515 3515
==========================================
- Hits 19633 19566 -67
- Misses 908 914 +6
- Partials 350 411 +61
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
IKisvardai
added a commit
that referenced
this pull request
Oct 9, 2026
Brings in the prescribed orbital migration (#937), the CALLIOPE cold-start seed (#958), the fwl-io dataset fetch and stellar spectra changes (#960, #964, #967, #968), the PETSc download (#965), Zalmoxis Anderson acceleration with fwl-aragog 26.10.08 (#971) and the CI changes (#955, #970). The schema invariant tests keep both the trapping and the migration validators, and the golden trajectory is main's re-recorded one with the trapping columns added.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The nightly integration tier now runs on 2 pytest-xdist workers per runner (
-n 2 --dist worksteal) instead of serially. The test step drops from about 59 min to 19 to 35 min on ubuntu and from about 47 min to 29 to 35 min on macOS, with the same 206 passed and 11 skipped on both. The job timeout goes from 80 to 90 min, which covers setup, the tests and a cold FWL_DATA download, and the job reports its 25 slowest tests.The worker count is fixed at 2 on both runners rather than
-n auto: more workers push the heaviest integration tests close to their per-test timeouts (4 workers on ubuntu gave 5 pytest-timeout failures). A test intests/tools/test_ci_tier_coverage.pyfails if the invocation changes the worker count or drops the work-stealing scheduler, andtests/AGENTS.mdsays that integration tests must not share files or state with each other.Validation of changes
Branch nightlies with 2 workers, all green:
smoke_janus, which runs slow on main for a separate reason.-n auto,-n 4, a second-n, and the flag removed.Tested on GitHub Actions (ubuntu-latest, macos-latest) and locally on macOS with Python 3.12.
Checklist