hackney_conn processes leak: completed no_reuse (Connection: close) HTTP/1.1 connections park in `connected` forever (sync body-completion path never closes non-reusable connections)

[lucianoengel]

Update (root cause corrected). My first hypothesis below — that the leak is caused by ?IDLE_TIMEOUT defaulting to infinity — turned out to be incomplete. Giving ?IDLE_TIMEOUT a finite value does not fix the leak: the idle reaper does {next_state, closed, Data}, but closed(enter) for a non-pooled connection (pool_pid = undefined) simply {keep_state, ...} with no stop and no grace timer (only pooled connections arm closed_grace_expired -> {stop, normal}). So a finite timeout just moves the zombie from connected to closed — the socket closes, but the gen_statem process is never terminated and keeps accumulating.

The actual root cause is in the synchronous body-completion path. receiving({call, From}, body, ...) and receiving({call, From}, stream_body, ...) unconditionally return {next_state, connected, ...} after the body is read — they never consult no_reuse or should_close_connection/1. So every completed no_reuse (or Connection: close) connection parks back in connected forever, pinned alive by its owner link → one leaked hackney_conn process per request. The async streaming path already handles this correctly in finish_async_streaming/1 (it stops non-reusable / non-pooled connections); the sync path (what HTTPoison uses) does not. See the corrected "Root cause" and "Suggested fix" sections below.

Summary

Under sustained load against an HTTP/1.1 endpoint whose connections are flagged no_reuse (e.g. Connection: close), hackney_conn gen_statem processes accumulate without bound and are never terminated. Each completed connection parks in the connected state as a child of hackney_conn_sup and nothing ever reaps it. Process memory grows until the node OOMs.

Observed on 4.4.5; the relevant code paths are unchanged in 4.5.2, so this is not fixed by upgrading.

Live evidence (production node — plain-HTTP JSON-RPC client, a few req/s)

• 2539 hackney_conn processes and climbing ~1.5/s (was 1714 a few minutes earlier), while only 34 OS ports exist. These are orphaned FSMs, not live connections (so it is not an fd/socket leak).
• Every one of them:
  • :sys.get_state/2 → connected
  • hackney_conn:checkin_info/1 →

    #{protocol => http1, upgraded_ssl => false, no_reuse => true,
      ready => false, should_close => false, pool_ssl => false}

• Their only live link is hackney_conn_sup. The calling owner is a long-lived process that never dies, so no owner-DOWN ever triggers cleanup.

Root cause (in src/hackney_conn.erl)

The synchronous request path never closes a non-reusable connection when the request completes. After the body is fully read:

receiving({call, From}, body, Data) ->
    case read_full_body(Data, <<>>) of
        ...
        {ok, Body, NewData} ->
            %% Socket still valid - return to connected state
            {next_state, connected, NewData, [{reply, From, {ok, Body}}]};   %% <-- unconditional
        ...

stream_body does the same on {done, NewData}. Neither consults no_reuse nor should_close_connection/1, so a no_reuse/Connection: close connection returns to connected and stays there. Because its owner is long-lived and its socket is idle (or already gone with no {tcp_closed, _} delivered), there is no path to termination.

Contrast the async path, which gets it right — finish_async_streaming/1:

case should_close_connection(Data) of
    true -> ... {stop, normal, ...};
    false when PoolPid =:= undefined -> ... {stop, normal, ...};
    false -> {next_state, connected, reset_async(Data)}
end

Two secondary factors make it impossible for any fallback to save the parked process:

1. -define(IDLE_TIMEOUT, infinity). and idle_timeout is never populated from app-env or a request option, so the connected(state_timeout, idle_timeout) -> {next_state, closed} reaper is never armed.
2. Even if it were, closed(enter, ...) for a non-pooled connection (pool_pid = undefined) does {keep_state, ...} — it closes the socket but never stops the process. Only pooled connections arm closed_grace_expired -> {stop, normal}.

This looks adjacent to #886 (HTTP/1.1 active-once stranding) and #888 (keepalive pooling correctness), but those addressed reused/pooled connections; the completed non-reused sync case is still uncovered.

Reproduction

Issue hackney:request/5 (e.g. via httpoison) repeatedly against a plain-HTTP/1.1 server that responds with Connection: close, at a modest rate, with a long-lived caller. Then count connection processes:

length([P || P <- erlang:processes(),
             {dictionary, D} <- [erlang:process_info(P, dictionary)],
             match =:= (case lists:keyfind('$initial_call', 1, D) of
                          {_, {hackney_conn, _, _}} -> match; _ -> nomatch end)]).

It grows monotonically while open sockets (length(erlang:ports())) stay flat.

Suggested fix

Make the sync completion path mirror finish_async_streaming/1 — stop the process instead of parking it when the connection must not be reused:

finish_sync_request(From, Reply, #conn_data{transport = Transport, socket = Socket,
                                            no_reuse = NoReuse} = Data) ->
    case NoReuse orelse should_close_connection(Data) of
        true ->
            case Socket of
                undefined -> ok;
                _ -> Transport:close(Socket)
            end,
            {stop_and_reply, normal, [{reply, From, Reply}],
             Data#conn_data{socket = undefined}};
        false ->
            {next_state, connected, Data, [{reply, From, Reply}]}
    end.

…called from both receiving({call,From}, body, ...) {ok, Body, NewData} and receiving({call,From}, stream_body, ...) {done, NewData}. Reusable keep-alive connections still return to connected.

Optionally, as defense-in-depth, also make closed(enter, ...) for pool_pid = undefined arm a grace timer + {stop, normal} (mirroring the pooled path) so that connections reaching closed via server-close / tcp_error also terminate.

Happy to send a PR.

[lucianoengel]

Correction / follow-up. I initially attributed this to ?IDLE_TIMEOUT defaulting to infinity and tried a finite default (120s) on a canary — it did not fix the leak. Reaching closed via the idle reaper does not terminate a non-pooled connection: closed(enter) for pool_pid = undefined just {keep_state, ...} (no stop, no grace timer — only pooled conns arm closed_grace_expired -> {stop, normal}). So a finite timeout only converts a connected zombie into a closed zombie; the process still leaks.

The real cause is that the synchronous body-completion path (receiving({call,From}, body/stream_body, ...)) returns {next_state, connected, ...} unconditionally, without checking no_reuse/should_close_connection/1 — unlike the async path (finish_async_streaming/1), which correctly stops non-reusable/non-pooled connections. I've corrected the issue body accordingly and will send a PR that mirrors the async logic in the sync path. Verified on our vendored 4.4.5; the sync path is identical in 4.5.2.

[lucianoengel]

Fix validated on a production canary. Deployed a build with the sync-path fix (finish_sync_request/3 stopping non-reusable connections, mirroring finish_async_streaming/1) to one pod and sampled the hackney_conn process count every 15s:

conns=0  ports=32
conns=4  ports=36
conns=6  ports=38
conns=0  ports=32
conns=0  ports=32
conns=0  ports=32
conns=5  ports=37
conns=5  ports=37
conns=0  ports=32
conns=0  ports=32
conns=1  ports=33
conns=2  ports=34

Before the fix, on the same node type, this count climbed monotonically (1714 → 2539, ~1.5/s) with only ~34 live OS ports — thousands of socketless zombie FSMs stuck in connected.

After the fix:

• The count oscillates near zero and returns to 0 between request bursts — non-reusable connections are reaped the instant each request completes, so nothing accumulates.
• hackney_conn process count now tracks live OS ports 1:1 (ports = 32 baseline + conns), i.e. every surviving process has a real socket and is a genuine in-flight request, not an orphan.
• Spot-checking the survivors' states returned %{connected: 1} (a single in-flight request), versus %{connected: <thousands>} before.

I'll open a PR shortly.