Skip to content

fix: discard a connection whose reply was never read - #545

Merged
cconstab merged 1 commit into
trunkfrom
fix/discard-connection-after-abandoned-read
Jul 30, 2026
Merged

fix: discard a connection whose reply was never read#545
cconstab merged 1 commit into
trunkfrom
fix/discard-connection-after-abandoned-read

Conversation

@cconstab

Copy link
Copy Markdown
Member

The problem

If a command is sent and its reply is not fully read — a read timeout, or any error raised while reading — that reply stays queued on the socket. Reusing the connection hands it to the next command, and every command after that is one reply behind. A lookup can then return a completely different record's value, with no error raised anywhere.

How it showed up

In a long-running deployment, publishing to one recipient began failing every cycle with:

Failed to decrypt shared_key.<recipient>@<sender>
  - Ciphertext length must be equal to key size.

The stored shared_key record was perfectly good. The lookup was returning the previous command's reply, which is not an RSA ciphertext. Restarting the process cleared it — the fault was in the connection, not the data.

Reproduced directly against a live atServer: after an abandoned read, llookup:publickey@bravo returned the earlier shared_key reply.

The fix

execute_command disconnects when a reply cannot be read, so the next use starts from a clean stream (disconnect() clears _connected, so connect() rebuilds — #523).

Deliberately minimal:

  • the successful path is untouched — the new handling only wraps the read;
  • commands sent with read_the_response=False (monitor, noop) are unaffected;
  • it composes with the existing retry_on_exception branch, which calls connect() and will now get a fresh socket.

This matters more as callers adopt read timeouts (which are otherwise the right way to stop a silently dead peer blocking forever): without this, a timeout trades a hang for a connection that quietly returns wrong answers.

Tests

test/abandoned_read_test.py, network-free:

Case Expected
read raises (timeout) exception propagates and the connection is left unusable
read succeeds response parsed, connection still usable
read_the_response=False unaffected, connection still usable

Related

Supersedes #544, which I closed: I had misattributed this incident to a damaged shared_key record. Since the desynchronised case reports the identical error, replacing the key there would have rotated a perfectly good shared key for both parties — fixing the cause is both safer and sufficient.

If a command is sent and its reply is not fully read — a read timeout, or any error
raised while reading — that reply stays queued on the socket. Reusing the connection
hands it to the NEXT command, and every command afterwards is one reply behind, so a
lookup can return a completely different record's value with no error anywhere.

Observed in a long-running deployment: a shared-key lookup returned the previous
command's reply, which is not an RSA ciphertext, and reported

    Failed to decrypt shared_key.<recipient>@<sender> - Ciphertext length must be equal
    to key size.

The stored record was perfectly good; restarting the process cleared it, because the
fault was in the connection rather than the data. Reproduced directly: after an
abandoned read, 'llookup:publickey@bravo' returned the earlier shared_key reply.

execute_command now disconnects when a reply cannot be read, so the next use starts
from a clean stream. The successful path is untouched, and commands sent with
read_the_response=False (monitor, noop) are unaffected.

Adds network-free tests: an abandoned read leaves the connection unusable, a successful
command keeps it, and a command that reads no reply is unaffected.
@cconstab
cconstab merged commit c05bb24 into trunk Jul 30, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants