Test names were changed

[ego-thales]

Hello,

I noticed that since version >=9, test names were changed, maybe due to different parametrization loading.
My CI depends on "expectation" files, which are used as reference from commit to commit, to check that end result does no change. These may contain plot results, tensors/arrays et simply console output.

Is this something that could be considered a bug and thus to be fixed?

Otherwise, I'm facing the problem of renaming every file and forcing users to use pytest>=9, which seems a be premature to date.

---

Concretely, you can view here some expectation files and for example, from pytest<9 to pytest>=9, one test went from being named

test_knn_label_count_expectations[cpu-half-ip-1-1-self_query=True]

to being named

test_knn_label_count_expectations[half-cpu-ip-1-1-self_query=True]

---

Thanks in advance!
All the best :)

[ego-thales]

Unsure, but this may be related to #13913?

[The-Compiler]

I don't think #13913 is related, and I don't think we've ever made a guarantee of test names / IDs being stable.

[bluetech]

Do you think you can provide a reproducible example? That will allow us to see why it changed and decide if it was intentional or not.

[eliegoudout]

Thank you for both your answers!

I'll try to provide a MRE. In the mean time you can refer to the (clearly nonminimal) example provided above, obtained with this test function:
https://github.com/ThalesGroup/scio/blob/1e9ee4343abb181afab0d1ccd231fb7d5fa1c0b9/test/scio/scores/utils/utils/test_utils.py#L325

[The-Compiler]

Bisected via pytest --collect-only test/scio/scores/utils/utils/test_utils.py | grep test_knn_label_count_expectations to:

• fixtures: fix closure computation with indirect override chains #13789

Also didn't manage to create a minimal example so far, though.

[eliegoudout]

Thanks for the bisect! I need to learn how to do that! It makes sense since it influences BFS/DFS behaviour in discovery.

Here's the current MRE I got:

import pytest

@pytest.fixture(params=["a1", "a2"])
def A(): ...

@pytest.fixture(params=["b1", "b2"])
def B(): ...

@pytest.fixture
def C(): ...

@pytest.fixture
def A_B(A, B): ...

@pytest.fixture
def B_C(B, C): ...

@pytest.fixture
def A_B_C(A_B, C): ...


def test(A_B_C, B_C): ...

====================== test session starts =======================
platform win32 -- Python 3.14.0, pytest-8.4.2, pluggy-1.6.0
cachedir: .pytest_cache
rootdir: C:\Users\Elie\workspace\tmp
configfile: pyproject.toml
collected 4 items

mre.py::test[b1-a1] PASSED                                  [ 25%]
mre.py::test[b1-a2] PASSED                                  [ 50%]
mre.py::test[b2-a1] PASSED                                  [ 75%]
mre.py::test[b2-a2] PASSED                                  [100%]

======================= 4 passed in 0.03s ========================
====================== test session starts =======================
platform win32 -- Python 3.14.0, pytest-9.0.1, pluggy-1.6.0
cachedir: .pytest_cache
rootdir: C:\Users\Elie\workspace\tmp
configfile: pyproject.toml
collected 4 items

mre.py::test[a1-b1] PASSED                                  [ 25%]
mre.py::test[a1-b2] PASSED                                  [ 50%]
mre.py::test[a2-b1] PASSED                                  [ 75%]
mre.py::test[a2-b2] PASSED                                  [100%]

======================= 4 passed in 0.02s ========================

Note: C cannot be trivially removed neither from A_B_C nor B_C in test.

[bluetech]

Thanks @The-Compiler for bisecting and @eliegoudout for creating an MRE.

I'd say the new behavior seems more intuitive? The new getfixtureclosure follows the runtime behavior more closely.

@ego-thales / @eliegoudout I don't know if it would work in your cases, but maybe you can tweak the order in which you request fixtures to get back the previous order, e.g. in the example change def test(A_B_C, B_C) to def test(B_C, A_B_C).

It might also be possible to write a hook which forces the order, but I'm not sure (need to check).

[eliegoudout]

I'd say the new behavior seems more intuitive?

I think I agree, without having a very strong opinion

@ego-thales / @eliegoudout I don't know if it would work in your cases, but maybe you can tweak the order in which you request fixtures to get back the previous order, e.g. in the example change def test(A_B_C, B_C) to def test(B_C, A_B_C).

It might work, but I actually have many fixtures and the test signature follows a structured ordering so for my project, I'd rather rename my expectation files and force pytest>=9 if it comes to that.

It might also be possible to write a hook which forces the order, but I'm not sure (need to check).

I think it does not really make sense for me since I should not really care about the ordering. And also, I have a hard time imagining a developer thinking "f*ck, I truly wanted pytest 8.4.2" ^^

Should this issue be closed? Or maybe wait if pytest considers documenting the ordering (I can't imagine I'm the only idiot in the world resting on this)?

I'll let you decide!

[bluetech]

OK, I'll close the issue. Since I think the new behavior is better, and stems for a real bugfix, I don't think we should revert it. As @The-Compiler said, pytest doesn't guarantee the exact order between versions (that would be very restricting), though hopefully there won't be a reason to change it again.