Repository navigation
[Regression] Can't upgrade from 0.33.1 to 0.35.0 because of change to how sdists are processed #2152
Description
🐞 bug report
Affected Rule
pip.parse, I guess?
Is this a regression?
Yes. Things were working in 0.33.1 and failed in 0.35.0.
A git bisect shows that 6f9082f (#2091 "fix: use downloaded archive in sdist") is the cause.
I confirmed that reverting the change resolves the issue and we're able to update to 0.35.0. Given that the PR was originally intended to fix a different issue, a simple revert is not the likely solution.
Description
A simple rules_python version bump should not require any changes to python dependencies (requirements.txt lockfile). However, I've noticed that I do have to bump some deps, namely these so far:
- lazy-object-proxy 1.7.1 --> 1.10.0
- google-crc32c 1.3.0 --> 1.5.0
- kiwisolver 1.4.2 --> 1.4.5
These are packages that did not have wheels for our python version (3.11) and then subsequently added wheels in later releases. So they would download sdists instead.
🔬 Minimal Reproduction
It should be enough to:
- Be on python 3.11, use bzlmod, use experimental bazel downloader
experimental_index_url - make a requirements.txt lockfile that uses
lazy-object-proxy(as it has no deps) 1.7.1 - try building a python manifest file
bazel run //:gazelle_python_manifest.update - Update
lazy-object-proxyto 1.8.0 (they added a 3.11 wheel) - try updating the manifest file again.
And you'll see the error.
🔥 Exception or Error
$ bazel run //:gazelle_python_manifest.update
INFO: Repository rules_python~~pip~pypi_311_lazy_object_proxy_sdist_d609c75b instantiated at:
<builtin>: in <toplevel>
Repository rule whl_library defined at:
/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/pypi/whl_library.bzl:451:30: in <toplevel>
INFO: repository @@rules_python~~pip~pypi_311_lazy_object_proxy_sdist_d609c75b' used the following cache hits instead of downloading the corresponding file.
* Hash 'd609c75b986def706743cdebe5e47553f4a5a1da9c5ff66d76013ef396b5a8a4' for https://files.pythonhosted.org/packages/75/93/3fc1cc28f71dd10b87a53b9d809602d7730e84cc4705a062def286232a9c/lazy-object-proxy-1.7.1.tar.gz
If the definition of 'repository @@rules_python~~pip~pypi_311_lazy_object_proxy_sdist_d609c75b' was updated, verify that the hashes were also updated.
ERROR: An error occurred during the fetch of repository 'rules_python~~pip~pypi_311_lazy_object_proxy_sdist_d609c75b':
Traceback (most recent call last):
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/pypi/whl_library.bzl", line 243, column 35, in _whl_library_impl
repo_utils.execute_checked(
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/repo_utils.bzl", line 212, column 29, in _execute_checked
return _execute_internal(fail_on_error = True, *args, **kwargs)
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/repo_utils.bzl", line 143, column 20, in _execute_internal
logger.fail((
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/repo_utils.bzl", line 85, column 39, in lambda
fail = lambda message_cb: _log(-1, "FAIL", message_cb, fail),
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/repo_utils.bzl", line 75, column 16, in _log
printer("\nrules_python:{} {}:".format(
Error in fail:
rules_python:whl_library(@@rules_python~~pip~pypi_311_lazy_object_proxy_sdist_d609c75b) FAIL: repo.execute: whl_library.BuildWheelFromSource(rules_python~~pip~pypi_311_lazy_object_proxy_sdist_d609c75b, lazy-object-proxy==1.7.1): end: failure:
command: /usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~python~python_3_11_host/python -m python.private.pypi.whl_installer.wheel_installer --requirement lazy-object-proxy==1.7.1 --isolated --extra_pip_args "{\"arg\":[\"--no-index\",\"--find-links\",\".\"]}" --pip_data_exclude "{\"arg\":[]}" --environment "{\"arg\":{}}"
return code: 1
working dir: <default: /usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~pip~pypi_311_lazy_object_proxy_sdist_d609c75b>
timeout: 600
environment:
PYTHONPATH="/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~:/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~internal_deps~pypi__build:/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~internal_deps~pypi__click:/usr/local/google/home/dthor/.c
ache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~internal_deps~pypi__colorama:/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~internal_deps~pypi__importlib_metadata:/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~internal_deps~pypi__installer:/usr/local/google/home/dthor
/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~internal_deps~pypi__more_itertools:/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~internal_deps~pypi__packaging:/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~internal_deps~pypi__pep517:/usr/local/google/home/dthor/.c
ache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~internal_deps~pypi__pip:/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~internal_deps~pypi__pip_tools:/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~internal_deps~pypi__pyproject_hooks:/usr/local/google/home/dthor/.cache/
bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~internal_deps~pypi__setuptools:/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~internal_deps~pypi__tomli:/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~internal_deps~pypi__wheel:/usr/local/google/home/dthor/.cache/bazel/_bazel
_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~internal_deps~pypi__zipp"
CPPFLAGS="-isystem /usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~python~python_3_11_host/include/python3.11"
===== stdout start =====
Looking in links: .
Processing ./lazy-object-proxy-1.7.1.tar.gz (from -r /tmp/tmp1_24gfue (line 1))
File was already downloaded /usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~pip~pypi_311_lazy_object_proxy_sdist_d609c75b/lazy-object-proxy-1.7.1.tar.gz
Installing build dependencies: started
Installing build dependencies: finished with status 'error'
===== stdout end =====
===== stderr start =====
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [3 lines of output]
Looking in links: .
ERROR: Could not find a version that satisfies the requirement setuptools>=30.3.0 (from versions: none)
ERROR: No matching distribution found for setuptools>=30.3.0
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/pypi/whl_installer/wheel_installer.py", line 205, in <module>
main()
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/pypi/whl_installer/wheel_installer.py", line 190, in main
subprocess.run(pip_args, check=True, env=env)
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~python~python_3_11_x86_64-unknown-linux-gnu/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~~python~python_3_11_host/python', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '--no-index', '--find-links', '.', '-r', '/tmp/tmp1_24gfue']' returned non-zero exit status 1.
===== stderr end =====
🌍 Your Environment
Operating System:
gLinux (Debian Bookworm-based)
Output of bazel version:
7.2.0
Rules_python version:
0.33.1
Anything else relevant?
Activity
dougthor42 commentedon Aug 23, 2024
As a local workaround we're just applying this patch.
#2090 does apply to us and we'd like it to be fixed, but it's not causing any major performance hits or preventing us from working, so for now we're fine with un-fixing it with the above patch.
aignas commentedon Aug 24, 2024
This is fixed by #2126.
fix(whl_library): remove `--no-index` and add `--no-build-isolation` …
dougthor42 commentedon Aug 24, 2024
Confirmed that b99bb61 works 👍. Thanks! And sorry for the noise 🙃