Split deterministically regardless of test order Fix #23#52
Conversation
| (2, 2, "duration_based_chunks", ["test_8", "test_9", "test_10"]), | ||
| (2, 1, "least_duration", ["test_3", "test_5", "test_6", "test_8", "test_10"]), | ||
| (2, 2, "least_duration", ["test_1", "test_2", "test_4", "test_7", "test_9"]), | ||
| (2, 1, "least_duration", ["test_3", "test_5", "test_7", "test_9", "test_10"]), |
There was a problem hiding this comment.
The split changing here is a bit unfortunate, but happens because "test_10" < "test_2" which shuffles everything around a bit, but the contract for least_duration should still be upheld.
|
Sorry about that failing test, I had changed around the code a bit when adding types and missed re-running it 🙃 |
|
I've now run this on a "real" test-suite together with |
|
Great! Could you also add a line to the changelog 🙏 And does this make any difference wrt the mention about |
|
Done! Would it be possible to do a release relatively soon? 😃 |
|
Gimme 15 minutes :) |
|
Needed only 6 minutes, it's available in 0.8.0. Thanks for the contribution! 🏅 |
2 participants
Description
Hi, I believe this should fix the issues with running this together with test-suite-randomization tools such as
pytest-randomlyorpytest-random-orderwhen using theleast_durationalgorithm, I don't think the problem is possible to solve forduration_based_chunks.The idea here is that if the tests always are in the same order when splitting (by sorting by str-representation) then the split should always be done the same way.
Opening this before fixing documentation + changelog in case that I've missed something obvious and this being rejected. 😅
Checklist
CHANGELOG.mdOR the changes are insignificant