How to make testing utilities importable when using importlib mode

[nicoddemus]

The --import-mode=importlib mode does not change sys.path in order to import test modules and conftest.py files, which has many benefits but also some drawbacks.

Often in test suites there are functions and classes used only for testing. This is not a problem when the tests are embedded in the source, but for layouts where the tests are in a separate directory, users don't have many options where to put those testing-only functions/classes in a way that is importable when using --import-mode=importlib (see #7245 (comment)).

One solution might be to have a new option that appends one or more directories to sys.path when running the tests only:

[pytest]
addopts = --import-mode=importlib
sys_path_append = tests

This at first might seem similar to just going ahead and using --import-mode=append, however this has the advantage that any directory can be added, instead of sys.path being implicitly changed just because tests/conftest.py exists with --import-mode=append.

This is just an idea, opening this issue to discuss the problem of how can we import testing utilities when using imporlib mode in layouts where tests are outside the source code.

[RonnyPfannschmidt]

I would propose that we investigate how to consider root paths in the scheme

With pep 420 and various other details we should consider test to be in packages with surrounding Code, not free-standing files

[nicoddemus]

I would propose that we investigate how to consider root paths in the scheme

By "root paths" do you mean our own rootdir/rootpath, or something else?

Either way, could you please elaborate more on your underlying idea?

[RonnyPfannschmidt]

the idea would be that testsuites have "root folders" that either locate inside a python package, or inside a "testpath"

that way by writing down the "testpaths" in pytest.ini, we could determine which modules to consider as toplevel modules

also we could supply a custom import to path based testsuites

i would like to elaborate the idea of "testusites" anyway, as it would be nice to be able to reuse/parametrize "testsuites" a different package ships, but thats out of scope for here

bascially i would like us to enable the importlib import to "seem" like the testpath which is a parent folder of the acted like a sys.path entry for enabling imports, while distinctively not being in sys.path

[pmeier]

Coming from #9109 (comment), would it be possible to import the test path in case it is a package? For example running pytest --import-mode=importlib tests on

tests/
├── __init__.py
├── test_foo.py
└── utils.py

could try to import tests (probably with setting an additional flag), which in turn would enable us to from .utils import bar in test_foo.py.

[nicoddemus]

Meanwhile I think you can use https://github.com/okken/pytest-srcpaths for that.

[pmeier]

@nicoddemus If I understand pytest-srcpaths correctly it won't help us.

One of the reasons we wanted to use --import-mode=importlib is that adding the project root to sys.path has unintended side effects. I'm aware that the usual way out is to put our package into a src/ folder, but unfortunately we can't go that route.

So if we add the project root with pytest-srcpaths we are back at our original problem.

[nicoddemus]

Unfortunately something needs to be added to sys.path for a test module to import from another test module... other import modes change sys.path implicitly, that's why one is able to import from other test modules.

As a workaround, you might consider moving your tests to a separate folder, and add that to sys.path using pytest-srcpaths.

To exemplify, consider you have this today:

app/
tests/
  __init__.py
  test_foo.py
  test_bar.py

Inside test_bar, you want to be able to do from tests.test_foo import some_utility.

You could create a new tests-root directory, and configure pytest-srcpaths to add that to sys.path:

app/
tests-root/
  tests/
    __init__.py
    test_foo.py
    test_bar.py

[pmeier]

Unfortunately something needs to be added to sys.path for a test module to import from another test module

I'm a little rusty on the details, but can't we use importlib to import packages without adding anything to sys.path. In my example above, if we import tests before we start the collection we should be good, right?

I'll write a small plugin as proof-of-concept if I find the time. What would be a good place to perform that import? pytest_sessionstart?

[nicoddemus]

if we import tests before we start the collection we should be good, right?

Only if you import it and add to sys.modules... the thing is, importlib is not recursive, it will import a single module without touching sys.modules or sys.path, but if the imported module itself imports other modules, those will get added to sys.modules.

I'll write a small plugin as proof-of-concept if I find the time. What would be a good place to perform that import? pytest_sessionstart?

I'm not entirely sure what you want to do, but that seems like a good hook to do stuff before test modules themselves are imported.

[kalekundert]

Just to add my 2 cents to this discussion: I wish that pytest had a way to treat each test script in the same way that python treats standalone scripts. In other words, for the directory layout below, I wish that running pytest basically behaved like python tests/test_foo.py; python tests/test_bar.py (in terms of how things are imported, at least):

tests/
  test_foo.py
  test_bar.py

Here's some pseudocode outlining how I imagine this working more specifically:

• For each test script found:
  • Add the directory containing the script to the beginning of sys.path (as documented here).
  • "Import" the script using importlib (without actually relying on the script being an actual python module, so it wouldn't matter if it had a non-python extension, or dots/hyphens in its name, or whatever).
  • Store the current value of sys.modules with each test loaded from the script, so that the correct modules can be used when the test is being executed.
  • Reset sys.path.
  • Reset sys.modules.

I think this approach would handle a lot of use cases very intuitively, although I admit that I haven't spent much time thinking about this issue, and I'm sure there are corner cases I haven't considered.

---

For what it's worth, I run into problems with pytest's import rules when I try to write tests for my documentation examples. Briefly, I try to include a lot of examples in my documentation, and I usually make a directory for each example. Each directory includes the example code itself, any secondary files necessary to run the example code (e.g. images, data sets, etc.), metadata relating to how to display the example, and tests to make sure the example actually works. (Doctest works for simple examples, but pytest is better for more complicated ones.) I run into problems when I inevitably want to give the test script the same name (e.g. test_example.py) for each example. I can work around this by just coming up with unique names for each test, but it feels like a limitation in pytest.

[pmeier]

@nicoddemus

Only if you import it and add to sys.modules... the thing is, importlib is not recursive, it will import a single module without touching sys.modules or sys.path, but if the imported module itself imports other modules, those will get added to sys.modules.

I've hacked together https://github.com/pmeier/pytest-import. It needs a lot of refinement, but for now it does what I want it to do: with this plugin installed, I can use common utilities while running with --import-mode=importlib.