download_and_extract: fail over to other URLs if a TLS certificate is expired

[jayconrod]

Description of the bug:

I saw this error when trying to work around #28101. One of the build errors I saw was rules_foreign_cc downloading make-4.4.1.tar.gz. The download is initiated by a module extension that instantiates http_archive.

The bug here is that this repository rule actually has two download URLs, but Bazel doesn't appear to have tried the second one:

        maybe(
            http_archive,
            name = "gnumake_src",
            build_file_content = _ALL_CONTENT,
            sha256 = "dd16fb1d67bfab79a72f5e8390735c49e3e8e70b4945a15ab1f81ddb78658fb3",
            strip_prefix = "make-4.4.1",
            urls = [
                "https://mirror.bazel.build/ftpmirror.gnu.org/gnu/make/make-4.4.1.tar.gz",
                "http://ftpmirror.gnu.org/gnu/make/make-4.4.1.tar.gz",
            ],
        )

Here's the full error message:

ERROR: /Users/jay/Library/Caches/bazel/_bazel_jay/e8de7d109452b60e8be8fd52c5e3b64d/external/bazel_tools/tools/build_defs/repo/http.bzl:156:45: An error occurred during the fetch of repository 'rules_foreign_cc++tools+gnumake_src':
   Traceback (most recent call last):
        File "/Users/jay/Library/Caches/bazel/_bazel_jay/e8de7d109452b60e8be8fd52c5e3b64d/external/bazel_tools/tools/build_defs/repo/http.bzl", line 156, column 45, in _http_archive_impl
                download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://mirror.bazel.build/ftpmirror.gnu.org/gnu/make/make-4.4.1.tar.gz] to /Users/jay/Library/Caches/bazel/_bazel_jay/e8de7d109452b60e8be8fd52c5e3b64d/external/rules_foreign_cc++tools+gnumake_src/temp6156153349777229410/make-4.4.1.tar.gz: (certificate_expired) PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
ERROR: no such package '@@rules_foreign_cc++tools+gnumake_src//': java.io.IOException: Error downloading [https://mirror.bazel.build/ftpmirror.gnu.org/gnu/make/make-4.4.1.tar.gz] to /Users/jay/Library/Caches/bazel/_bazel_jay/e8de7d109452b60e8be8fd52c5e3b64d/external/rules_foreign_cc++tools+gnumake_src/temp6156153349777229410/make-4.4.1.tar.gz: (certificate_expired) PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
ERROR: /Users/jay/Library/Caches/bazel/_bazel_jay/e8de7d109452b60e8be8fd52c5e3b64d/external/rules_foreign_cc+/toolchains/private/BUILD.bazel:23:10: @@rules_foreign_cc+//toolchains/private:make_tool depends on @@rules_foreign_cc++tools+gnumake_src//:all_srcs in repository @@rules_foreign_cc++tools+gnumake_src which failed to fetch. no such package '@@rules_foreign_cc++tools+gnumake_src//': java.io.IOException: Error downloading [https://mirror.bazel.build/ftpmirror.gnu.org/gnu/make/make-4.4.1.tar.gz] to /Users/jay/Library/Caches/bazel/_bazel_jay/e8de7d109452b60e8be8fd52c5e3b64d/external/rules_foreign_cc++tools+gnumake_src/temp6156153349777229410/make-4.4.1.tar.gz: (certificate_expired) PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed

Which category does this issue belong to?

Core

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

#28101 is no longer happening, so this is somewhat speculative.

1. Set up a local HTTPS server with an expired certificate.
2. Add an entry to /etc/hosts matching the certificate's domain pointing to the local server's IP.
3. Add the certificate to the system root CA store so it's valid, other than being expired.
4. Copy a file from a public server with a valid certificate, serve from the local server.
5. Get Bazel to download the file with http_archive, using both URLs (expired local server first).

Bazel should fail to download the file from the local server, but should fall back and succeed when downloading from the public server.

Which operating system are you running Bazel on?

macOS

What is the output of bazel info release?

10.0.0-pre.20251208.3

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[Wyverald]

The logic to download multiple URLs is here: https://cs.opensource.google/bazel/bazel/+/master:src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/HttpDownloader.java;drc=92d426e8ecdb6e947aa47907a114693b1a06cffd;l=76

From a cursory glance, the code should be falling back to subsequent URLs if a previous URL has an SSL cert expiry. Some more digging is required.

[Wyverald]

Actually, given this line, all URLs passed to rctx.download_and_extract should be present in the exception message. That doesn't seem to be the case in your pasted output. Weird...

[elskow]

i think i found the root cause @Wyverald,

so the issue isn't with the fallback logic in HttpDownloader.download(), that part is actually fine. The problem is happening one layer deeper in HttpConnector.java.

when a connection fails, HttpConnector has a retry loop that retries the same url up to 8 times with exponential backoff. This makes sense for transient errors like 500s, timeouts, or connection refused. But ssl certificate errors are also going through this same retry path.

so what happens is:

1. first url has expired cert.
2. HttpConnector tries it, fails, waits, retries.
3. repeat 8 times (cert is still expired obviously).
4. finally gives up and throws the exception.
5. exception propagates up but at this point its too late, the error message already only contains the first url.