[pypi] Support pulling dependencies using direct url specifiers without using an index

🐞 bug report

Affected Rule

Module extension:

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(...)

Is this a regression?

Not sure.

Description

The repository rule for pip dependencies doesn't use repository_ctx.download(). This breaks usages of --experimental_downloader_config for example to use a transparent dependency mirror.

🔬 Minimal Reproduction

https://github.com/bazelbuild/rules_python/blob/b6fc2a080d4884510dea02cc77b4e0b8fd3a0ccd/python/private/pypi/whl_library.bzl#L245-L256

https://github.com/bazelbuild/rules_python/blob/b6fc2a080d4884510dea02cc77b4e0b8fd3a0ccd/python/private/pypi/whl_installer/wheel_installer.py#L168-L190

🔥 Exception or Error

Stack trace prints to the code linked above.

🌍 Your Environment

Operating System:

Linux

Output of bazel version:

7.0.2

Rules_python version:

0.37.2

Anything else relevant?

Please see the documentation here: https://rules-python.readthedocs.io/en/latest/pypi-dependencies.html#bazel-downloader-and-multi-platform-wheel-hub-repository.

This currently has a limitation that it needs hashes for all of the files in the requirements.txt. The stabilization of this feature is tracked by #260.

@aignas

@aignas How can I use experimental_index_url in combination with dependencies without an index like the following:

proto-breaking-change-detector @ https://github.com/googleapis/proto-breaking-change-detector/archive/refs/tags/v2.4.0.tar.gz
tensorflow_graphics @ git+https://github.com/tensorflow/graphics@ca63cf788978db0bde2db4ce488e102c455da2e7

The support for direct references has not been added yet, I'll reopen this issue and rename it to scope it to supporting it, but I am not sure when I can work on it.

proto-breaking-change-detector @ https://github.com/googleapis/proto-breaking-change-detector/archive/refs/tags/v2.4.0.tar.gz could be easy enough. Outline of the changes:

Update the https://github.com/bazelbuild/rules_python/blob/main/python/private/pypi/index_sources.bzl file to support extracting URLs from the requirements file
Change the code that calls the index to not call it for requirement lines that refer to URLs
Pipe the url and the filename to the whl_library respective attributes.

For the second one it may be harder - I am not sure how to handle git repos yet, however, with bazel 7.4 the extension can pass a labels within the extension more easily, so one could look into how to do that.

For now the workaround would be to build the wheels from sdists outside bazel and host them in some PyPI compatible artifact mirror and then use experimental_index_url flag to enable the usage of the bazel-downloader.

feat(pypi): support direct urls for wheels in bazel downloader (#2655)

feat(pypi/parse_requirements): get dists by version when no hash prov…

fix(pypi): correctly fallback to pip for git direct URLs

fix(pypi): correctly fallback to pip for git direct URLs (#2732)

@mering

As @mering suggests the git_repository archive is downloading things by shelling out to the git executable source. I am not entirely sure that this is something that we can do easily because duplicating calling git should be out of scope of rules_python.

However, we should support git_repository when pulling sources for sdist building, which we want to officially support via build actions (#2410). Ideally we could just use the http_archive or git_repository to get the sources.

I'd like to mark this as done and add a note to the sdist building ticket for git sources support. Does that sound good to you, @mering?

@aignas

@aignas thanks for fixing the direct download case 1. When we applied this to our code base, we noted that it still tries to reach out to an index-like URL. Could this be skipped in that case?

I am fine with continuing the conversation about the git repository in the other issue.

I tried this with 1.5.0 and it seems like this issue still isn't fixed. Worse, it seems like our workaround is no longer working.

We have the following in our requirements.txt file (generated via pip-compile):

proto-breaking-change-detector @ https://github.com/googleapis/proto-breaking-change-detector/archive/refs/tags/v2.6.0.tar.gz \
    --hash=sha256:e99f9a83d097b5fa774ee45206778b8a543e38766d4e698e21d65e1ca1302100
    # via -r requirements.in

In pip.parse() in MODULE.bazel, we set experimental_index_url = "https://pypi.org/simple",.

Then we get the following error:

ERROR: Could not find a version that satisfies the requirement proto-breaking-change-detector (from versions: none)
ERROR: No matching distribution found for proto-breaking-change-detector

Even our previous workaround doesn't work any more:

        experimental_index_url_overrides = {
            "proto-breaking-change-detector": "https://github.com/googleapis",
        },

Does simpleapi_skip work for you? https://rules-python.readthedocs.io/en/latest/api/rules_python/python/extensions/pip.html#pip.parse.simpleapi_skip

Something like:

    simpleapi_skip = ["proto-breaking-change-detector"]

could work.

Does simpleapi_skip work for you? https://rules-python.readthedocs.io/en/latest/api/rules_python/python/extensions/pip.html#pip.parse.simpleapi_skip

Something like:
    simpleapi_skip = ["proto-breaking-change-detector"]
could work.

Thanks for the suggestion. I added this to our pip.parse() invocation but unfortunately, the same error occurrs.

Full traceback is

===== stderr start =====
ERROR: Could not find a version that satisfies the requirement proto-breaking-change-detector (from versions: none)
ERROR: No matching distribution found for proto-breaking-change-detector
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/runner/.cache/bazel/_bazel_runner/80ee21f5abd5872fd13fbc92e73e36c7/external/rules_python+/python/private/pypi/whl_installer/wheel_installer.py", line 183, in <module>
    main()
  File "/home/runner/.cache/bazel/_bazel_runner/80ee21f5abd5872fd13fbc92e73e36c7/external/rules_python+/python/private/pypi/whl_installer/wheel_installer.py", line 168, in main
    subprocess.run(pip_args, check=True, env=env)
  File "/home/runner/.cache/bazel/_bazel_runner/80ee21f5abd5872fd13fbc92e73e36c7/external/rules_python++python+python_3_11_x86_64-unknown-linux-gnu/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/runner/.cache/bazel/_bazel_runner/80ee21f5abd5872fd13fbc92e73e36c7/external/rules_python++python+python_3_11_host/python', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '--extra-index-url', 'https://us-central1-python.pkg.dev/intrinsic-mirror/intrinsic-public-python/simple', '--extra-index-url', 'https://download.cadexchanger.com/python', '--find-links', '.', '-r', '/tmp/tmpdd421hof']' returned non-zero exit status 1.
===== stderr end =====

Ah it seems that the bug/issue is in whl_library.bzl code somewhere. Let's reopen it.

I am not sure how/if it is possible to fix it. Does pip install or uv pip install work without contacting the index?

I now get the same error, but we do not use experimental_index_url.

It seems like the general use of direct URLs to a non-wheel package is just broken, is that correct?

fix(pypi): support properly installing sdists via pypi without index

@mering

@mering, @thomasbaekkegaard, could you please test #3115？

@aignas

@aignas Thanks for looking into this. I can confirm that your PR fixes the previous issue and the dependency is now downloaded correctly again.

fix(pypi): support properly installing sdists via pypi without index

fix(pypi): support properly installing sdists via pypi without index (#…

Navigation Menu

Repository navigation

[pypi] Support pulling dependencies using direct url specifiers without using an index #2363

🐞 bug report

Affected Rule

Is this a regression?

Description

🔬 Minimal Reproduction

🔥 Exception or Error

🌍 Your Environment

9 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[pypi] Support pulling dependencies using direct url specifiers without using an index #2363

Description

🐞 bug report

Affected Rule

Is this a regression?

Description

🔬 Minimal Reproduction

🔥 Exception or Error

🌍 Your Environment

Activity

aignas commented on Nov 3, 2024

mering commented on Nov 4, 2024

aignas commented on Nov 14, 2024

aignas commented on Apr 8, 2025

mering commented on Apr 8, 2025

9 remaining items

mering commented on Jul 4, 2025

aignas commented on Jul 6, 2025

mering commented on Jul 7, 2025

aignas commented on Jul 7, 2025

thomasbaekkegaard commented on Jul 11, 2025

aignas commented on Jul 22, 2025

mering commented on Jul 28, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions