Repository navigation
Conversation
The default range was 10001-20000 and the head server was pinned to 11000. Both sit inside the OS ephemeral range, which starts at 32768 on stock Linux and at 9000 on some DGX and GB200 nodes. The kernel can assign a port in that range to an outgoing connection between the time we probe it and the time a server binds it, and startup then fails with EADDRINUSE. The new default stays under 9000 and leaves room for the services Gym is usually colocated with: training frameworks below it, and sandboxes and inference engines above it, commonly at 6000-6999 and 7000-8999. The head server moves too. It is the only port Gym pins rather than probes for, so leaving it at 11000 would have kept the one port every client must reach inside the ephemeral range. It now defaults to the top of the band and is still seeded into disallowed_ports. Also: - Hoist the range into DEFAULT_PORT_RANGE_LOW/HIGH instead of inline literals, and drop the duplicate DEFAULT_HEAD_SERVER_PORT in server_utils, which is now re-exported from global_config so both import sites keep working. - Fail fast in _find_open_port_using_range when the whole range is disallowed. Its rejection loop is unbounded; at 10000 ports that was unreachable, but at 1000 a narrowed or exhausted range would hang with no diagnostic. Callers that hardcoded a port in 10001-20000, or that reached the head server at 11000, need updating. Deployments that set port_range_low/port_range_high explicitly are unaffected. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Contributor
|
🌿 Preview your docs: https://nvidia-preview-ananthsub-port-range-defaults.docs.buildwithfern.com/nemo/gym Here are the markdown pages you've updated: |
github-actions
Bot
added
the
sla:review-overdue
This was referenced Sep 8, 2026
svcnvidia-nemo-ci
added
area:core
This was referenced Oct 22, 2026
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The default port range was 10001-20000 and the head server was pinned to 11000. On some clusters' nodes, the ephemeral range can sit as low as 9000 for the OS ephemeral range. A port drawn from there can be handed to an outgoing connection between our bind probe and the server's real bind, so startup fails with EADDRINUSE.
for instance, NeMo RL already reserves 5000-5999 for Gym and passes down the range here: https://github.com/NVIDIA-NeMo/RL/blob/aa5c74edec9657f3aca3c570255970e24fa6768d/ray.sub#L163
Adopting the same defaults here means a standalone Gym run lands in the same band. The head server port needed to move too: it is the only port Gym pins rather than probes for, so leaving it at 11000 would have kept the one port every client must reach potentially inside the ephemeral zone. It now defaults to the top of the band and is still seeded into disallowed_ports.
Also:
DEFAULT_PORT_RANGE_LOW/HIGHand drop the duplicateDEFAULT_HEAD_SERVER_PORTin server_utils_find_open_port_using_rangewhen the whole range is disallowed instead of infinite looping