Incremental version 24.7.1 still breaks some projects's setuptools behaviour

[potiuk]

The #100 in incremental 24.7.0 has been solved, but there is another breaking change remaining in 24.7.1.

In airlfow we are using virtualenv to create venvs and it seems that incremental initialization during virtualenv's intialization also break something by attempting to read pyproject.toml. Attempting to parse and fail in i__init__ where some inconsistency occurs is generally a bad idea when incremental is apprently somewhat plugged into setuptools/distutils (it looks like at least - why otherwise it would be initialized when python -m virtualenv got initialized?)

The error can be very easily reproduced using our images that we build in CI:

Run:

docker run -it  ghcr.io/apache/airflow/main/ci/python3.8:4535e08b862e2b7ff4f2a76de7124983d4efe9db

And follow it up with python -m virtualenv (without changing the current directory from /opt/airflow - which is where airlfow sources are present). Then it will fail with:

root@4051201568b7:/opt/airflow# python -m virtualenv
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/runpy.py", line 185, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/local/lib/python3.8/runpy.py", line 144, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/usr/local/lib/python3.8/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/usr/local/lib/python3.8/site-packages/virtualenv/__init__.py", line 3, in <module>
    from .run import cli_run, session_via_cli
  File "/usr/local/lib/python3.8/site-packages/virtualenv/run/__init__.py", line 15, in <module>
    from .plugin.creators import CreatorSelector
  File "/usr/local/lib/python3.8/site-packages/virtualenv/run/plugin/creators.py", line 7, in <module>
    from virtualenv.create.via_global_ref.builtin.builtin_way import VirtualenvBuiltin
  File "/usr/local/lib/python3.8/site-packages/virtualenv/create/via_global_ref/builtin/builtin_way.py", line 5, in <module>
    from virtualenv.create.creator import Creator
  File "/usr/local/lib/python3.8/site-packages/virtualenv/create/creator.py", line 13, in <module>
    from virtualenv.discovery.cached_py_info import LogCmd
  File "/usr/local/lib/python3.8/site-packages/virtualenv/discovery/cached_py_info.py", line 25, in <module>
    _CACHE[Path(sys.executable)] = PythonInfo()
  File "/usr/local/lib/python3.8/site-packages/virtualenv/discovery/py_info.py", line 100, in __init__
    self.distutils_install = self._distutils_install().copy()
  File "/usr/local/lib/python3.8/site-packages/virtualenv/discovery/py_info.py", line 182, in _distutils_install
    d = dist.Distribution({"script_args": "--no-user-cfg"})  # conf files not parsed so they do not hijack paths
  File "/usr/local/lib/python3.8/site-packages/setuptools/dist.py", line 475, in __init__
    _Distribution.__init__(
  File "/usr/local/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 283, in __init__
    self.finalize_options()
  File "/usr/local/lib/python3.8/site-packages/setuptools/dist.py", line 904, in finalize_options
    ep(self)
  File "/usr/local/lib/python3.8/site-packages/incremental/__init__.py", line 402, in _get_setuptools_version
    config = _load_pyproject_toml("./pyproject.toml")
  File "/usr/local/lib/python3.8/site-packages/incremental/__init__.py", line 518, in _load_pyproject_toml
    path=_findPath(os.path.dirname(toml_path), package),
  File "/usr/local/lib/python3.8/site-packages/incremental/__init__.py", line 370, in _findPath
    raise ValueError(
ValueError: Can't find the directory of package apache-airflow: I looked in ./src/apache-airflow and ./apache-airflow

The pyproject.toml seems to be pretty correct, yet incremental fails.

Incremental is installed there in latest version.

root@141711ea4b00:/opt# pip freeze | grep incremental
incremental==24.7.1

Running the same command from a different directory works:

root@141711ea4b00:/opt# python -m virtualenv
usage: virtualenv [--version] [--with-traceback] [-v | -q] [--read-only-app-data] [--app-data APP_DATA] [--reset-app-data] [--upgrade-embed-wheels] [--discovery {builtin}] [-p py] [--try-first-with py_exe]
                  [--creator {builtin,cpython3-posix,venv}] [--seeder {app-data,pip}] [--no-seed] [--activators comma_sep_list] [--clear] [--no-vcs-ignore] [--system-site-packages] [--symlinks | --copies] [--no-download | --download]
                  [--extra-search-dir d [d ...]] [--pip version] [--setuptools version] [--wheel version] [--no-pip] [--no-setuptools] [--no-wheel] [--no-periodic-update] [--symlink-app-data] [--prompt prompt] [-h]
                  dest
virtualenv: error: the following arguments are required: dest

Also downgrading it to 22.10.0 works also from /opt/airflow:

root@141711ea4b00:/opt/airflow# pip install incremental==22.10.0
Collecting incremental==22.10.0
  Downloading incremental-22.10.0-py2.py3-none-any.whl.metadata (6.0 kB)
Downloading incremental-22.10.0-py2.py3-none-any.whl (16 kB)
Installing collected packages: incremental
  Attempting uninstall: incremental
    Found existing installation: incremental 24.7.0
    Uninstalling incremental-24.7.0:
      Successfully uninstalled incremental-24.7.0
Successfully installed incremental-22.10.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
root@141711ea4b00:/opt/airflow# python -m virtualenv
usage: virtualenv [--version] [--with-traceback] [-v | -q] [--read-only-app-data] [--app-data APP_DATA] [--reset-app-data] [--upgrade-embed-wheels] [--discovery {builtin}] [-p py] [--try-first-with py_exe]
                  [--creator {builtin,cpython3-posix,venv}] [--seeder {app-data,pip}] [--no-seed] [--activators comma_sep_list] [--clear] [--no-vcs-ignore] [--system-site-packages] [--symlinks | --copies] [--no-download | --download]
                  [--extra-search-dir d [d ...]] [--pip version] [--setuptools version] [--wheel version] [--no-pip] [--no-setuptools] [--no-wheel] [--no-periodic-update] [--symlink-app-data] [--prompt prompt] [-h]
                  dest
virtualenv: error: the following arguments are required: dest
SystemExit: 2

[twm]

Thank you for the thorough report, and apologies for the breakage.

It looks like this was introduced in #94, so it would also affect the yanked 24.7.0 release. I am currently working up a patch.

[potiuk]

Thank you for the thorough report, and apologies for the breakage.

No worries. Happens :)

[twm]

I have a fix ready over at #107. It is currently waiting for review.

If I install the package from CI (please disregard the version number):

# python -m pip install incremental-24.7.1-py3-none-any.whl --force-reinstall --no-deps

(I have to use --no-deps or it tries to upgrade setuptools and fails.

Then virtualenv seems to work:

root@aceedf37b248:/opt/airflow# python -m virtualenv venv
created virtual environment CPython3.8.19.final.0-64 in 353ms
  creator CPython3Posix(dest=/opt/airflow/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==24.1, setuptools==70.1.0, wheel==0.43.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

[potiuk]

Confirmed. Downloaded the package, installed it in the old image and it works for me as well. I also prepared a Draft PR in Airlfow to test the new version once it is out apache/airflow#41075

[potiuk]

Running tests in apache/airflow#41075, we exclued both .0 and .1 but it would be great to yank .1 as well I think.

[twm]

I didn't want to yank 24.7.1 because that would disrupt the Twisted security release currently in progress (it contained a relevant bugfix). In my judgement the danger of breakage is much lower than with 24.7.0, and as 24.7.2 is out anyone who hasn't pinned a specific Incremental will get the fixed version.

I want to be cautious about yanking: the yank of 24.7.0 actually broke Twisted's build because it had a >=24.7.0 lower bound. I felt that shipping 24.7.2 was the lowest-risk option.

I'm open to being convinced otherwise, but those are my thoughts.

[potiuk]

For us no problem - we already excluded it in apache/airflow#41075 so that should be no issue for us