[balajinvda]

What does this PR do?

Adds an opt-in CRLF heartbeat to the global aiohttp client so that long-running, non-streaming model calls survive network intermediaries that drop idle TCP connections. Default off; no behaviour change unless global_aiohttp_crlf_heartbeat_seconds is set.

Why

Some hosted-inference endpoints sit behind intermediaries that close any TCP connection carrying no payload for a fixed period and that ignore TCP keepalive probes. AWS Global Accelerator is the common example: its idle timeout is fixed at 340 s and not configurable (AWS docs). A judge or reward-model call that takes longer than that to produce its first response byte fails with ServerDisconnectedError at ~340 s on every retry. Gym already enables TCP keepalive on its sockets; that does not help here because the intermediary does not count keepalive probes.

HTTP/2 clients handle this with PING frames. aiohttp is HTTP/1.1-only, and Gym intentionally standardises on aiohttp (CLAUDE.md). The HTTP/1.1 equivalent is to write a bare CRLF on the connection while the request is in flight:

• The bytes travel as encrypted TLS records, so the intermediary sees payload and resets its idle timer.
• RFC 9112 §2.2 requires servers to ignore empty lines received before a request-line, so the origin discards them once the pending response is sent and the connection stays reusable. RFC 5626 §4.4.1 standardises the same "CRLF keep-alive technique" for SIP over TCP; nginx, HAProxy and llhttp all skip leading CR/LF.

How

• nemo_gym/aiohttp_heartbeat.py: HeartbeatTCPConnector(TCPConnector). Exactly one background task per connector, not per connection. Every heartbeat seconds it makes a single pass over the connector's acquired set (requests in flight, including the wait before response headers) and calls transport.write(b"\r\n") on each, yielding to the event loop after every batch_size writes (default 128) so a large pass cannot stall other coroutines. No per-connection tasks, timers or callbacks; idle pooled connections are untouched. heartbeat=0 makes it identical to TCPConnector.
• nemo_gym/server_utils.py: new config field global_aiohttp_crlf_heartbeat_seconds: float = 0; set_global_aiohttp_client builds HeartbeatTCPConnector(heartbeat=cfg....) instead of TCPConnector. Everything else (socket factory, limits, request(), retries, tracing) is unchanged.
• tests/unit_tests/test_aiohttp_heartbeat.py: local slow HTTP/1.1 server that counts and skips stray CRLFs. Asserts heartbeats are written while a response is pending, the server ignores them, the body is intact, the same connection serves the next request, nothing is written while idle, and the default is off.

Enable with:

global_aiohttp_crlf_heartbeat_seconds: 60

Evidence

Measured against a generic echo endpoint behind AWS Global Accelerator with a 400 s server-side delay, aiohttp 3.14.x, ClientTimeout() with no limits (Gym's configuration):

Client	Result
aiohttp HTTP/1.1, unchanged	ServerDisconnectedError at 340 s
aiohttp HTTP/1.1 + HeartbeatTCPConnector(heartbeat=60)	200 OK at ~401 s
Gym request() from this branch, global_aiohttp_crlf_heartbeat_seconds: 0 (default)	ServerDisconnectedError at 340.1 s
Gym request() from this branch, global_aiohttp_crlf_heartbeat_seconds: 60	200 OK at 401.2 s, 6 heartbeats sent
raw HTTP/1.1 socket + CRLF every 60 s (mechanism check)	200 OK at 401 s

Connection reuse was verified after single CRLFs and after a burst of ten CRLFs on the same connection (next request 200). A bare space instead of CRLF breaks the next request (400), so the connector only ever writes CRLF.

Evidence: instrumented traces

Raw HTTP/1.1 over TLS to an NVCF echo function behind a generic AWS Global Accelerator endpoint (anycast IP redacted), server-side delay 400 s, nvcf-poll-seconds: 1200. After each heartbeat the client reads the kernel's TCP_INFO.bytes_sent for the socket: each CRLF leaves the host as one 31-byte TLS record. Times are UTC.

Heartbeat every 60 s: response arrives at 401 s

00:59:58 connected peer=<Global Accelerator anycast IP> tls=TLSv1.2 alpn=http/1.1 heartbeat=every 60s server_delay=400s
00:59:58 request sent: tcp bytes_sent=1002
01:00:58 t+  60s heartbeat #1: CRLF written; tcp bytes_sent 1002->1033 (+31)
01:01:59 t+ 121s heartbeat #2: CRLF written; tcp bytes_sent 1033->1064 (+31)
01:02:59 t+ 181s heartbeat #3: CRLF written; tcp bytes_sent 1064->1095 (+31)
01:03:59 t+ 241s heartbeat #4: CRLF written; tcp bytes_sent 1095->1126 (+31)
01:05:00 t+ 302s heartbeat #5: CRLF written; tcp bytes_sent 1126->1157 (+31)
01:06:00 t+ 362s heartbeat #6: CRLF written; tcp bytes_sent 1157->1188 (+31)
01:06:39 t+ 401s RESULT: HTTP/1.1 200 OK nvcf-reqid: 0776ae15-20fa-48f7-8a45-910b322bc7f3 body=b'"proof-hb60"' heartbeats=6

Heartbeat off: the accelerator closes the connection at 340 s

01:06:39 connected peer=<Global Accelerator anycast IP> tls=TLSv1.2 alpn=http/1.1 heartbeat=OFF server_delay=400s
01:06:39 request sent: tcp bytes_sent=1001
01:12:19 t+ 340s RESULT: connection closed by peer (EOF), no response. heartbeats=0

Same endpoint, same delay, same client; the only difference is the heartbeat.

Evidence: does the CRLF ever corrupt a response or the next request?

A concern raised during evaluation was that a CRLF written while a response is still streaming could be read by an intermediary as a pipelined request and cut the body, or make the next request on that connection fail with HTTP 400. Tested end to end against a second production gateway path (a Go net/http origin behind an AWS ALB and Global Accelerator; Go's server itself does answer 400 to a bare CRLF ahead of a request-line, so this path is the adversarial case), aiohttp 3.14 + this connector, one pooled connection so every request follows the previous request's CRLFs, heartbeat every 0.2 s (300x the recommended cadence, so CRLFs land mid-body on every request), non-streaming chat completions from a 235B model:

Arm	Requests	Body size	CRLFs written per request	Complete 200 + valid JSON	Cut bodies / stray 400s / disconnects
heartbeat 0.2 s	10	25–31 KB	201–249	10 / 10	0
heartbeat 0.2 s, top-5 logprobs	8	2.1–2.7 MB	218–319	8 / 8	0
control, heartbeat off	6	24–32 KB	0	6 / 6	0
control, heartbeat off, logprobs	4	2.1–2.4 MB	0	4 / 4	0

4,313 CRLFs landed mid-body or ahead of the next request-line on reused connections; every chunked body arrived complete and the following request on the same connection was always accepted. The load balancer discards the empty line before it reaches the origin, as RFC 9112 §2.2 intends.

Known side effect on AWS ALB (found in the load balancer's own access logs, not visible to the client). The ALB does not discard a CRLF that arrives while a request is pending; it holds it as the first byte of the next request. A real request that follows is accepted (1,947 connection reuses in the 600-concurrent run all returned 200). If the client instead closes that connection while idle, the ALB logs the dangling CRLF as an incomplete request: elb_status_code 400, no target, 0 bytes. In the 600-concurrent heartbeat run (2,420 requests, 0 client-side errors) the ALB logged 616 such entries, all when connections went idle at the end of the run and aiohttp's keep-alive cleanup closed them. Nobody receives these 400s; they show up only in the load balancer's 4xx metrics. Operators of an ALB-fronted endpoint should expect roughly one per heartbeat-carrying connection that is closed idle.

Benchmark: event-loop cost of the heartbeat pass at scale

Addresses the review question about the cost of the pass under load (the earlier "negligible at 16k" wording was not measured; this is).

Setup. Client and server in separate pods on separate nodes of a Kubernetes dev cluster, TLS 1.3 over the pod network. The server accepts HTTP/1.1 requests and never answers, so every connection stays acquired (worst case for the heartbeat). The client opens N concurrent POSTs through the connector and then, for 130 s, an independent coroutine sleeps 10 ms in a loop and records how late it wakes (event-loop lag). Heartbeat interval 60 s (the production value), so two full passes fall in the window. Three variants: heartbeat off (baseline), the unbatched single synchronous pass as first pushed in this PR (since replaced), and the batched pass now in the PR (batch_size=128). Passes that ran while connections were still being established are excluded from the pass-time figures. Numbers are from one run each; RSS is the client process's peak resident set after all connections are up.

In-flight TLS connections	Variant	Process CPU (share of one core)	Loop lag p99	Loop lag max	Steady-state pass time	Client RSS
16,000	heartbeat off	0.4 %	0.14 ms	0.7 ms	–	4.9 GiB
16,000	unbatched pass (as in commit 1, since replaced)	0.8 %	0.15 ms	257 ms	262 ms (blocking)	4.9 GiB
16,000	batched (this PR)	0.9 %	0.14 ms	8.1 ms	320 ms (cooperative)	4.9 GiB
50,000	heartbeat off	0.4 %	0.15 ms	1.2 ms	–	15.2 GiB
50,000	unbatched pass (as in commit 1, since replaced)	1.7 %	0.14 ms	879 ms	763 ms (blocking)	15.2 GiB
50,000	batched (this PR)	1.8 %	0.17 ms	7.8 ms	962 ms (cooperative)	15.2 GiB
100,000	heartbeat off	0.4 %	0.14 ms	0.3 ms	–	30.4 GiB
100,000	unbatched pass (as in commit 1, since replaced)	3.5 %	0.14 ms	2,095 ms	2,064 ms (blocking)	30.4 GiB
100,000	batched (this PR)	3.4 %	6.07 ms	7.7 ms	1,975 ms (cooperative)	30.4 GiB

Reading it.

• CPU cost is the same for both variants and small: about 3 percentage points of one core at 100k in-flight connections with a 60 s interval (≈ 20 µs of TLS write per connection per pass).
• The unbatched pass blocks the event loop for its whole duration, which grows linearly with concurrency: a quarter second at 16k, over two seconds at 100k. The batched pass caps the stall at one batch, ≈ 8 ms, at every scale tested; the pass itself takes the same total time but other coroutines keep running through it. At 100k the p99 rises to 6 ms because a 2 s pass of 780 slices covers ~3 % of the observation window.
• Memory: the heartbeat adds no measurable memory. RSS is identical across off/original/batched at every scale (differences of a few MiB). The ~300 KiB per idle TLS connection is aiohttp plus OpenSSL state and is present with the heartbeat off. The connector holds one task and one counter per connector and no per-connection state.
• Where it broke: only the test harness, on memory. Client and server together exceed 40 GiB at 100k, which is why they were split into two pods; the heartbeat itself ran to 100k without error.

Machine. Client pod on a c5.9xlarge node (Intel Xeon Platinum 8124M @ 3.00 GHz, 36 logical CPUs; pod limited to 16 CPUs / 52 GiB), Ubuntu 24.04, kernel 6.14, Kubernetes 1.34. Python 3.13.15, aiohttp 3.14.3, OpenSSL 3.5.7. Loop lag sampled by a 10 ms ticker on the same event loop as the connector, as it would be in a Gym server process.

Benchmark scripts (bench_beat.py, tls_hold_server.py) are available on request.

Notes for reviewers

• Uses three aiohttp internals: BaseConnector._create_connection, BaseConnector._acquired, ResponseHandler.transport. Verified on 3.14.x; the unit test guards them.
• Cost: see the benchmark section above. One pass per interval, ~16–20 µs of TLS write per in-flight connection; the batching keeps the worst-case event-loop stall to one batch (~8 ms measured) instead of the whole pass.

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally (tests/unit_tests/test_aiohttp_heartbeat.py; passes together with test_server_utils.py).
• Pre-commit checks pass locally (pre-commit run --files ...: end-of-file, trailing whitespace, ruff, ruff-format all Passed).
• All commits have DCO sign-off (git commit -s).

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[kajalj22]

Initial review:

Feature offered is a good alternative way to ensure model calls survive network intermediaries.

Once per interval, when _beat_once() actually runs, it executes as one uninterrupted step: nothing else on the event loop (other requests, other responses) can make progress until that single call returns. Cost is O(N) in the number of in-flight connections, including a TLS-encrypted write per connection.

Changes requested:

The PR asserts this is "negligible at 16k concurrency" but that's not benchmarked, the evidence sections test connection survival against the 340s timeout, not _beat_once()'s cost under load - please correct me if I'm mistaken here. Also, our target concurrency is much higher.

Suggest:

benchmark _beat_once()'s actual duration at realistic concurrency before merging.

[balajinvda]

Thanks @kajalj22, both points were worth chasing, and you were right on the second.

On "one task per acquired connection": the connector runs exactly one background task per connector, regardless of connection count. Each interval that task makes one pass over the connector's acquired set and writes b"\r\n" on each in-flight connection. No per-connection tasks, timers or callbacks are created. I have made that explicit in the module and class docstrings (commit 2) since the code did not say so.

On "negligible at 16k, not benchmarked": correct, that sentence was an estimate, not a measurement, and measuring showed the cost was not negligible in the way that matters. The pass was a single synchronous loop, so at 16,000 in-flight TLS connections it blocked the event loop for ~260 ms; at 100,000, for ~2.1 s. CPU was small (3.5 % of one core at 100k with a 60 s interval), but a multi-second loop stall once a minute is not acceptable in a process serving other requests.

Change (commit 3): _beat_once is now async and yields to the event loop after every batch_size writes (default 128). Same total work, but the worst-case loop stall is one batch. Measured max loop lag with the batched pass: 8.1 ms at 16k, 7.8 ms at 50k, 7.7 ms at 100k in-flight connections, versus 257 ms / 879 ms / 2,095 ms unbatched. CPU is unchanged and memory is identical to the heartbeat-off baseline at every scale. A unit test asserts the pass yields between batches.

Full method, machine details and the table (off / unbatched / batched at 16k, 50k, 100k, with CPU, loop lag p99 and max, pass time and RSS) are in the "Benchmark" section of the PR description; the unbatched rows are kept there as the before measurement. Client and server were separate pods on separate c5.9xlarge nodes, TLS 1.3, aiohttp 3.14.3, Python 3.13.15, with a 10 ms ticker on the same event loop measuring lag.

Caveat: this measures the heartbeat pass in isolation with idle in-flight connections, not under Gym's real request mix. Happy to run it against a Gym workload if you can point me at a representative setup.

[kajalj22]

Thanks for the benchmarking and identifying the cost of the changes @balajinvda

This is great progress! Especially the yields on the batch size - this amortizes the cost well and will limit the pressure on the event loop.

Now we can benchmark the solution with an actual training run to see the numbers when the event loop is under pressure

[kajalj22]

PR looks good - pending the benchmark data from a training run