Activity

1. linzhp

   changed the title [-]pip_parse builds wheels with different cases[/-] [+]pip_parse builds wheels with different names[/+] on Mar 6, 2025
2. aignas commented on Mar 6, 2025

   

   aignas

   on Mar 6, 2025

   Last edited by aignas

   Collaborator

   More actions

   #2410 should include this, however no one is working on this yet.

   I am inclined to close as duplicate, but will leave open if you want to submit fixes to the repo rules.

   EDIT: Some extra thoughts on this:

   • pip is vendoring all of the deps with itself, so what gets loaded should be deterministic?
   • If it is not, I wonder if the PYTHON path that gets constructed in the pypi_repo_utils is non-deterministic.
   • Maybe we should remove pip from hermetic toolchain, which get rid of those vendored extra deps. Maybe having a completely clean interpreter help with determinism here, but it is somewhat a spray and pray strategy...
   • are there any blocking issues that would hold you from testing if the bug is reproducible with the latest rules_python version?
3. linzhp commented on Mar 6, 2025

   

   linzhp

   on Mar 6, 2025

   ContributorAuthor

   More actions

   pip is vendoring all of the deps with itself, so what gets loaded should be deterministic?

   From the documentation, it's isolated but I don't think it's deterministic. Pip reads build-system.requires key in the pyproject.toml to see if it requires specific version of setuptools. In the example, it specifies "setuptools ~= 58.0". So pip will get the latest version the range of 58.0.x. In the case of PyPika, it doesn't specify setuptools version, so pip will get the latest version available in PyPI. The "latest" mean different version if we build at different times.

   are there any blocking issues that would hold you from testing if the bug is reproducible with the latest rules_python version?

   I just verify it in rules_python 1.2.0. I can see that pip still creates pypika-0.48.9-py2.py3-none-any.whl, which indicates that it's using setuptools 75.8.1 or later, while the setuptools that comes with rules_python is 70.0.0: https://github.com/bazelbuild/rules_python/blob/a816962e509311c23230730b4b28f9d52a229949/python/private/pypi/deps.bzl#L79
4. aignas

   added

   type: pippip/pypi integrationpip/pypi integration

   on Mar 10, 2025
5. chrisirhc commented on Mar 14, 2025

   

   chrisirhc

   on Mar 14, 2025

   Last edited by chrisirhc

   Contributor

   More actions

   Looked into this a bit, and I'm guessing the behavior can sort of be explained by https://discuss.python.org/t/pip-build-isolation-without-installing-build-dependencies/52145 . I was able to reproduce it in https://github.com/chrisirhc/rules_python-repro/tree/chua/pypika-setuptools , and you can see the behavior in the logs by using the command in readme.

   To "fix" this behavior and make pypika installation make use of the pypi deps, you'd need to add a --no-build-isolation to extra_pip_args. This stops pip from creating a new empty virtual environment for the sdist build, and lets the build use the setuptools that's available to it.

   It's not ideal to remove build isolation. I guess what you'd want/expect is:

   • have the build isolation of a virtual environment
   • the virtual environment to sort of "inherit" the available dependencies (e.g. setuptools from rules_python/python/private/pypi/deps.bzl)

   I don't think that mode exists right now, at least based on the above discussion link.

   Another workaround to get stable reproducible builds would be to avoid sdist — build a wheel and upload it into an internal repo and point the requirements to a direct uri of that wheel.
6. groodt commented on Mar 14, 2025

   

   groodt

   on Mar 14, 2025

   Last edited by groodt

   Contributor

   More actions

   This doesn't even seem like a bug? It seems like a bug fix by setuptools. Now that setuptools is following PEP 491 for wheel names, it's expected behaviour.

   pip is an implementation detail of rules_python and it is a build frontend, not a build backend. sdist declare their own build backend configuration in their pyproject.toml

   e.g

   [build-system]
   requires = ["setuptools"]
   build-backend = "setuptools.build_meta"
   

   So Im struggling to understand the issue or impact here. If you want to avoid sdist head-aches, don't use sdist. Otherwise, this is behaving as expected and intended by the package authors tbh.

   I'd like to understand:

   • The impact of this issue. It seems very minor to me. If you have any sdist in your dependencies, all bets are off anyway
   • Why is this a bug with rules_python and not with PyPika

   I think this is a very minor bug, so I will chat with the other maintainers, but probably suggest we close it in a week as "won't fix".
7. linzhp commented on Mar 14, 2025

   

   linzhp

   on Mar 14, 2025

   ContributorAuthor

   More actions

   Why is this a bug with rules_python and not with PyPika

   Without any changes to the repo, rules_python can pick up any version of setuptools and build a wheel with a different sha256. This violates the hermeticity, which should be provided by Bazel/rules_python.

   The impact of this issue.

   In a large repo that depends on the sha256 of all targets to determine what to build and test on CI, and which service to redeploy BazelCon 2019, this led to indeterminism.
8. groodt commented on Mar 14, 2025

   

   groodt

   on Mar 14, 2025

   Last edited by groodt

   Contributor

   More actions

   There are no guarantees of determinism for any sdist. It goes beyond setuptools. Any sdist build can (and often is) non-deterministic. This is true inside or outside bazel. This is true even if no dependencies change.

   The support for wheels can be considered deterministic because the download and unpack steps can be made to be deterministic.

   But sdist builds can't be guaranteed to be deterministic because they involve arbitrary code execution. And the builds are also not managed by bazel.

   So if you require full determinism for third-party python dependencies, you can achieve this by sourcing the third-party dependencies as wheel-only in the dependency closure.

   So we should close this as duplicate of #2410 because it's just another of the known issues with sdist. It's not a regression.

   👍React with 👍3Reacted by Richard Levasseur, Zhongpeng Lin and Ignas Anikevicius
9. aignas

   closed this as a duplicateof Support building from sdist in a build action #2410 on Mar 14, 2025