Relay foundation (#2436): fix before wiring the relay #2471

#2436 adds the relay crypto, mux and offer foundation unwired: nothing in the app uses it yet. Its review left three findings that must be fixed, each with a failing test first, before anything connects to the relay:

• HIGH: a carrier frame is decoded before its size is checked. An oversized frame is buffered and decoded in full before it is rejected. Check the length prefix against the limit first and reject without decoding.
• HIGH: the handshake queue is unbounded. Messages that arrive before the handshake completes are queued without a limit. Cap the queue and fail the connection past the cap.
• MEDIUM: the relay identity file is written 0644. Write it 0600, and fix the mode of an existing file on load.

Found in the merge pass's review of #2436 (vendored paseo-e2ee code, SHA-256 checked).