Repository navigation
Make command line tests faster #5966
Description
Activity
Can I take this?
Yes, just open a PR!
Is this still the case? Runing
pytest .\mypy\test\testcmdline.pyruns in 51s for me; what would be considered fast? Also please let me know if this issue refers to something else than what runs whenpytest .\mypy\test\testcmdline.py.Yes, that's right / they're still slow. Due to
pytest-xdist, how long they take is a function of number of cores on your machine. In CI, we have 2 cores, sopytest mypy/test/testcmdline.py -n 2would give you a better sense of how much time these tests contribute when you open a PROh hmm, actually I think Github Actions updated its runners to have 4 cores. That should be an easy win, #17185
Oh hmm, actually I think Github Actions updated its runners to have 4 cores. That should be an easy win.
Thanks, but what should be an easy win? Are they not being run with four workers on the GitHub instances? It's running with 8 workers on my machine even though it has 16 logical cores; do you mean tweaking that setting?
Yes, we currently run with
-n 2, so with a maximum concurrency of 2. I think we set it explicitly because the value reported byos.cpu_count()on GH Actions doesn't match the actual number of cores the action has available.Reacted by Manuel MartinezGotcha. Edit: I see @hauntsaninja submitted a PR to set
n 4But beyond that, how can we get away from running the CLI as a sub process and still test some of the stdout from the CLI? The original comment mentions converting some of the tests into regular check tests or even unit tests. Could somebody point me to examples of both?
Yeah, Jukka's main idea is to just make tests that don't actually need to be run as a subprocess into other tests. For example, I think a lot of the "config file" tests in
cmdline.testcould be made into normal tests (e.g. move them intocheck-flags.test)Reacted by Manuel Martinez- added a commit that references this issue
on Jul 22, 2024 - added a commit that references this issue
on Oct 17, 2024
Command line tests in
test-data/unit/cmdline.testare very slow since they use full typeshed stubs and run mypy in a subprocess. It looks like many of them would be easy to migrate to be normal type checker tests, or some of then could be even regular Python unit tests. Alternatively, we could perhaps use test stubs with them.Speeding up report generation tests may also be easy.