Tests fail on pytest 3.8 and 3.9 with pytest 4

[diazona]

We've started getting errors in CI, as shown for example here on PR 96:

$ tox -vv -e py --force-dep pytest==4
[...]
py: 11260 W commands[0]> pytest -v --junitxml=junit-py.xml [tox/tox_env/api.py:426]
/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/_pytest/config/__init__.py:195: PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
Plugin: helpconfig, Hook: pytest_cmdline_parse
AssertionError: 
For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning
  return pluginmanager.hook.pytest_cmdline_parse(
Traceback (most recent call last):
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/bin/pytest", line 8, in <module>
    sys.exit(main())
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/_pytest/config/__init__.py", line 58, in main
    config = _prepareconfig(args, plugins)
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/_pytest/config/__init__.py", line 195, in _prepareconfig
    return pluginmanager.hook.pytest_cmdline_parse(
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/pluggy/_callers.py", line 156, in _multicall
    teardown[0].send(outcome)
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/_pytest/helpconfig.py", line 93, in pytest_cmdline_parse
    config = outcome.get_result()
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/_pytest/config/__init__.py", line 675, in pytest_cmdline_parse
    self.parse(args)
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/_pytest/config/__init__.py", line 845, in parse
    self._preparse(args, addopts=addopts)
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/_pytest/config/__init__.py", line 799, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/pluggy/_manager.py", line 422, in load_setuptools_entrypoints
    self.register(plugin, name=ep.name)
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/_pytest/config/__init__.py", line 306, in register
    ret = super(PytestPluginManager, self).register(plugin, name)
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/pluggy/_manager.py", line 169, in register
    hook._maybe_apply_history(hookimpl)
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/pluggy/_hooks.py", line 581, in _maybe_apply_history
    res = self._hookexec(self.name, [method], kwargs, False)
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/setuptools/_vendor/typeguard/_pytest_plugin.py", line 53, in pytest_addoption
    add_ini_option("string")
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/setuptools/_vendor/typeguard/_pytest_plugin.py", line 22, in add_ini_option
    parser.addini(
  File "/home/runner/work/pytest-localserver/pytest-localserver/.tox/py/lib/python3.9/site-packages/_pytest/config/argparsing.py", line 130, in addini
    assert type in (None, "pathlist", "args", "linelist", "bool")
AssertionError
py: 11978 C exit 1 (0.72 seconds) /home/runner/work/pytest-localserver/pytest-localserver> pytest -v --junitxml=junit-py.xml pid=2070 [tox/execute/api.py:286]
.pkg: 11979 D delete package /home/runner/work/pytest-localserver/pytest-localserver/.tox/.tmp/package/1/pytest_localserver-0.9.1.dev1+ge6f077a.tar.gz [tox/tox_env/python/virtual_env/package/pyproject.py:224]
  py: FAIL code 1 (11.74=setup[11.03]+cmd[0.72] seconds)
  evaluation failed :( (11.84 seconds)

[diazona]

This proved a little tricky to reproduce locally, but I managed it in Docker: first build an image from this

FROM python:3.9

RUN useradd --uid 1000 -m -s /bin/bash ubuntu
USER ubuntu
WORKDIR /home/ubuntu
RUN python -m venv venv && . venv/bin/activate && python -m pip install --upgrade pip && python -m pip install tox
WORKDIR /code
ENTRYPOINT /bin/bash

and then run

$ docker run --rm -it -v .:/code <image>
ubuntu@9ba8592d6d01:/code$ source ~/venv/bin/activate
ubuntu@9ba8592d6d01:/code$ tox -vv -e py --force-dep pytest==4
[...]
Plugin: helpconfig, Hook: pytest_cmdline_parse
AssertionError: 
[...]
  File "/code/.tox/py/lib/python3.9/site-packages/_pytest/config/argparsing.py", line 130, in addini
    assert type in (None, "pathlist", "args", "linelist", "bool")
AssertionError
py: 21148 C exit 1 (1.00 seconds) /code> pytest -v --junitxml=junit-py.xml pid=337 [tox/execute/api.py:286]
.pkg: 21149 D delete package /code/.tox/.tmp/package/6/pytest_localserver-0.9.0.tar.gz [tox/tox_env/python/virtual_env/package/pyproject.py:224]
  py: FAIL code 1 (20.52=setup[19.52]+cmd[1.00] seconds)
  evaluation failed :( (20.71 seconds)

This is on master, incidentally, so the error has nothing to do with the PR on which it first showed up (#96).

[diazona]

Ah I think I figured it out: in setuptools commit 00384a5f4fd2 (v71.0.0), it added typeguard==4.3.0 as a vendored dependency. typeguard includes a pytest plugin, and as of typeguard commit ded1a04f0c99 (v4.2.0), that plugin calls pytest's parser.addini() with type = "string". But pytest didn't support "string" as a type until commit af3759a50372 (v6.2.0). So essentially this is a conflict between typeguard>=4.2 (via setuptools>=71) and pytest<6.2.

I suppose the logical fix is to bump the minimum required version of pytest up to 6.2 - the alternative is enforcing a maximum version of typeguard or perhaps setuptools, and enforcing a maximum version is probably not a great idea. I want to explore the dependency space a bit to confirm, but I can make the change soon.