Requests 2.32.0 Not supported URL scheme http+docker

[joshzcold]

Newest version of requests 2.32.0 has an incompatibility with python lib docker

INTERNALERROR> Traceback (most recent call last):

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/requests/adapters.py", line 532, in send

INTERNALERROR>     conn = self._get_connection(request, verify, proxies=proxies, cert=cert)

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/requests/adapters.py", line 400, in _get_connection

INTERNALERROR>     conn = self.poolmanager.connection_from_host(

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/urllib3/poolmanager.py", line 304, in connection_from_host

INTERNALERROR>     return self.connection_from_context(request_context)

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/urllib3/poolmanager.py", line 326, in connection_from_context

INTERNALERROR>     raise URLSchemeUnknown(scheme)

INTERNALERROR> urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme http+docker

INTERNALERROR> 

INTERNALERROR> During handling of the above exception, another exception occurred:

INTERNALERROR> 

INTERNALERROR> Traceback (most recent call last):

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/docker/api/client.py", line 213, in _retrieve_server_version

INTERNALERROR>     return self.version(api_version=False)["ApiVersion"]

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/docker/api/daemon.py", line 181, in version

INTERNALERROR>     return self._result(self._get(url), json=True)

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/docker/utils/decorators.py", line 44, in inner

INTERNALERROR>     return f(self, *args, **kwargs)

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/docker/api/client.py", line 236, in _get

INTERNALERROR>     return self.get(url, **self._set_request_timeout(kwargs))

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/requests/sessions.py", line 602, in get

INTERNALERROR>     return self.request("GET", url, **kwargs)

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/requests/sessions.py", line 589, in request

INTERNALERROR>     resp = self.send(prep, **send_kwargs)

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/requests/sessions.py", line 703, in send

INTERNALERROR>     r = adapter.send(request, **kwargs)

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/requests/adapters.py", line 534, in send

INTERNALERROR>     raise InvalidURL(e, request=request)

INTERNALERROR> requests.exceptions.InvalidURL: Not supported URL scheme http+docker

Expected Result

Normal initalization of docker client

Actual Result

Stack trace posted above

Reproduction Steps

mkvirtualenv debug_issue
pip install docker
pip install 'requests>=2.32.0'
python

import docker
docker.from_env()

System Information

$ python -m requests.help

{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "3.3.2"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.7"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.10.12"
  },
  "platform": {
    "release": "6.5.0-28-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.31.0"
  },
  "system_ssl": {
    "version": "30000020"
  },
  "urllib3": {
    "version": "2.2.1"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}

[ShawnHardwick]

I believe this was introduce with this line change which started enforcing the URL scheme:
c0813a2#diff-a4598bf8444e0feca489e60704c809c0dbfc47b90c4364216c931b497b3312d7R512

The issue is also being tracked over in docker-py repo:
docker/docker-py#3256

[nateprewitt]

Hi @joshzcold, could you clarify more about the expected functionality with a http+docker scheme vs calling the URI with just http. Is docker-py using a custom Requests adapter or was this just being passed as generated to Requests?

This appears to be a byproduct of fixing CVE-2024-35195. urllib3 is expecting the scheme to be either http or https here in the context construction.

[nateprewitt]

It looks like there's a proposed patch for docker here. Their sub-adapter seems to reasonably have been relying on implementing get_connection. We'll use this issue to track the change in #6655 regarding get_connection and use the docker issue linked about the docker adapter.

[felixfontein]

@nateprewitt do you think the fix in docker/docker-py#3257 is good enough for now, or do you have a suggestion for a better way to work around this problem, at least for now? I guess it would be great if Docker SDK for Python wouldn't need to depend on internal behavior of requests, but changing that in a proper way likely needs a larger rewrite / rearchitecturing of that part of Docker SDK for Python...

[joshzcold]

Since it looks like docker-py needs apply the fix on their end I believe we can close this PR.

[nateprewitt]

We wrote the #6655 fix trying to avoid breaking users that were relying on the existing behavior of get_connection in their custom Adapters but missed the other side where they implemented a custom get_connection but were still using the default send.

I don't think what docker-py was doing is unreasonable, but unfortunately I don't see a clear way to mitigate this in Requests while maintaining the security fix. docker/docker-py#3257 seems like the least invasive option at the moment.

I have a hunch docker-py may not be the only package that hits this issue so I'd like to track these kinds of problems to evaluate if we need to take further action on our side. I'll leave this open for now to gather any other cases.

[mainland]

requests-unixsocket, which implements the http+unix scheme, is also broken by this change.

[sigmavirus24]

I would guess requests-file or whichever fork of that is still maintained would be affected if anyone uses it

[felixfontein]

requests-unixsocket, which implements the http+unix scheme, is also broken by this change.

It implements it very similarly to Docker SDK for Python, i.e. it also overwrites get_connection(). Adding a _get_connection() similar to the one in docker/docker-py#3257 should fix it as well.