ssl.SSLZeroReturnError exception on startup with builtin ssl backend python 3.8 or above

[toppk]

❓ I'm submitting a ...

• 🐞 bug report
• 🐣 feature request
• ❓ question about the decisions made in the repository

🐞 Describe the bug. What is the current behavior?
I get an exception on startup. Everything seems to be working okay.
I have attached the exception, and the code used you will need to generate an ssl cert/key pair to test.

❓ What is the motivation / use case for changing the behavior?

💡 To Reproduce

python test.py

this is the exception so people don't have to download the attachment. the code is a basic CherryPy startup that I grabbed from #346

[17/Sep/2022:01:17:23] ENGINE Started monitor thread 'Autoreloader'.
[17/Sep/2022:01:17:23] ENGINE Serving on https://0.0.0.0:4400
[17/Sep/2022:01:17:23] ENGINE Bus STARTED
[17/Sep/2022:01:17:23] ENGINE Error in HTTPServer.serve
Traceback (most recent call last):
  File "/home/toppk/.local/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve
    self._connections.run(self.expiration_interval)
  File "/home/toppk/.local/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run
    self._run(expiration_interval)
  File "/home/toppk/.local/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run
    new_conn = self._from_server_socket(self.server.socket)
  File "/home/toppk/.local/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket
    s, ssl_env = self.server.ssl_adapter.wrap(s)
  File "/home/toppk/.local/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap
    s = self.context.wrap_socket(
  File "/usr/lib64/python3.9/ssl.py", line 501, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib64/python3.9/ssl.py", line 1041, in _create
    self.do_handshake()
  File "/usr/lib64/python3.9/ssl.py", line 1310, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129)

exception.txt
test.py.txt

💡 Expected behavior
startup without exception

📋 Details

📋 Environment

• Cheroot version: 8.6.0
• CherryPy version: 18.6.0
• Python version: 3.9.14
• OS: Fedora 37 beta
• Browser: all

📋 Additional context
I've tried this in python3.11rc2 and python3.9, both have the issue. I do not see any similar issue if I switch to
pyopenssl backend.

[toppk]

one thing is important to note, is that there is no tcp connection made when the exception occurs, this exception is spontaneous.

[webknjaz]

This strongly reminds me of cherrypy/cherrypy#1618.

[webknjaz]

one thing is important to note, is that there is no tcp connection made when the exception occurs, this exception is spontaneous.

Well, that might be CherryPy's Checker kicking in. Have you tried disabling it?

[toppk]

i disabled the checker, no difference. I tested on ubuntu 22.04 with python3.10 and the same issue.

[webknjaz]

Oh, wait, I remember there was some kind of a TLS probe added in relatively recent releases. Maybe that's what's failing...

FWIW It'd be useful to have a pure-Cheroot repro for this.

[toppk]

update: tried straight cheroot and it didn't have any issues, added logging and I can definetly see the peer address so it may be a tls probe (which I haven't found in the code just yet).

Another major update, is that this issue goes away when using python<3.8, no changes in cherrypy/cheroot versions, just switching between python 3.8 and python 3.7 flips this exception on and off, and they both seem to be hitting the same codepath (connection.run())

[toppk]

Okay, I monkey patched socket, and printed a stacktrace on connect and found the code. it is portend, specifically in cherrypy/process/servers.py

# wait for port to be occupied
with _safe_wait(*self.bound_addr):
        portend.occupied(*self.bound_addr, timeout=Timeouts.occupied)

So everything is happening as expected. There are two questions.

1. why does cheroot under python 3.7 hid these zero-bytes-sent connections and only see them under python 3.8, and
2. do we want to hide such exceptions in similar cases in the future?

I can confirm that just telnet'ing to the port and closing the connection will generate this exception under python>=3.8 and not log anything otherwise.

I personally can understand leaving this issue alone, but I think it's bad as is, especially since we forcibly generate this corner case on startup. I will try to understand how python 3.7 is able to silence these 0byte tcp connections.

[toppk]

adding the test code w/ monkeypatch, in case people need to debug something like this in the future.

import cherrypy
import sys

import socket

print("python version:", sys.version)
print("cherrpy version:", cherrypy.__version__)


class RootServer:
    @cherrypy.expose
    def index(self, **keywords):
        return "it works!"


oldSocket = socket.socket


class MonkeySocket2(oldSocket):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def connect(self, *args, **kwargs):
        import traceback
        traceback.print_stack()
        print("in connect")
        return super().connect(*args, **kwargs)

socket.socket = MonkeySocket2

if __name__ == "__main__":
    server_config = {
        #'server.socket_host': '127.0.0.1',
        "server.socket_host": "0.0.0.0",
        "server.socket_port": 4400,
        "server.ssl_module": "builtin",
        #'server.ssl_module':'pyopenssl',
        "server.ssl_certificate": "local/cert.pem",
        "server.ssl_private_key": "local/privkey.pem",
        "engine.autoreload.on": False,
        "checker.on": False,
    }

    cherrypy.config.update(server_config)
    cherrypy.quickstart(RootServer())

[toppk]

I found the change in cpython that caused the issue, and submitted a pr to cheroot

[toppk]

This strongly reminds me of cherrypy/cherrypy#1618.

as it turns out, the cpython patch was made in response to cherrypy/cherrypy#1618 , which ended up manifesting the same exception a different way. This was a confusing bug to track down.

[The-Compiler]

I started seeing this as well after a recent Archlinux system update. I'm not entirely sure that the CPython commit linked in #518 is really to blame (after all, it also talks about ssl.SSLEOFError and not SSLZeroReturnError!).

The more likely culprit seems to be upgraded openssl (1.1.1.q-1 -> 3.0.7-2) and not any change in CPython.