Repository navigation
feat(orchestration): route Gym's Lens telemetry through the job collector - #3619
Conversation
|
🌿 Preview your docs: https://nvidia-preview-iguehring-orchestrator-lens-telemetry.docs.buildwithfern.com/nemo/gym Here are the markdown pages you've updated: |
2ce46de to
7afa366
Compare
iguehring-nvidia
added
the
area:orchestration
e53d170 to
de933b7
Compare
e845e61 to
da5a708
Compare
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
de933b7 to
1747dc4
Compare
7724f74 to
4720610
Compare
yaoyu-33
added
complexity:medium
4720610 to
dc5426e
Compare
oyilmaz-nvidia
left a comment
oyilmaz-nvidia
left a comment
There was a problem hiding this comment.
So, this PR does not start the OTel collector itself, it only wires Gym's servers to export telemetry to a collector already started.
In the clusters, most likely we won't be able to send data outside of the cluster. So, it's not convenient to start Grapaha and related system, and then submit the task.
I guess we should either start grapaha, etc with the script or save the data to a local disk that can be imported by the grapaha.
| `slurm_job_id`, `benchmark`, `cluster` and `model`, plus `service.name` set to the routing identity | ||
| and `service.name.override` to the producer's display name. | ||
| and `service.name.override` to the producer's display name: each Gym server keeps its own name | ||
| (`nemo-gym/<server>`) and the scraped engine carries `component`. |
There was a problem hiding this comment.
docs claim service.name.override equals just component, but the collector config actually sets it to component/name (e.g. gym-vllm/policy). Users filtering dashboards per the docs will get empty results.
There was a problem hiding this comment.
Thx! Raising a new revision.
| repo=gi.repo if gi else None, | ||
| ref=gi.ref if gi else None, | ||
| prepare_cmd=prepare_cmd, | ||
| extras=(GYM_TELEMETRY_EXTRA,) if observed else (), |
There was a problem hiding this comment.
when otel.enabled: true (the default) but driver.gym_install isn't set, the telemetry extra is requested but render_driver_entrypoint only emits the install line inside its if repo and ref: branch, so nemo-lens never gets installed. Telemetry silently no-ops with no visible error.
There was a problem hiding this comment.
Thx! Raising a new revision.
github-actions
Bot
added
sla:author-overdue
▎ So, this PR does not start the OTel collector itself, it only wires Gym's servers to export telemetry to a collector already started. Correct. For runs through the Gym orchestrator the collector is started per job by #3610, merged last week, as the first srun step of the sbatch script. We need that collector regardless of Gym's own telemetry, because vLLM and the GPU exporters only expose Prometheus endpoints and something on the node has to scrape them and stamp the run identity. This PR wires Gym's servers into that same collector. They could talk to a node-local collector directly, but going through ours means one place stamps run_id, benchmark, cluster and user on Gym rows and engine rows alike, one place fans out to the destinations. From the collector there are two destinations, both configuration:
I have verified both options in a POC. ▎ I guess we should either start Grafana etc with the script or save the data to a local disk that can be imported by the Grafana. In both options the Grafana already exists and is operated by another team, so it does not need to be started per job. That has the benefit that all runs will appear in the same dashboard. Local disk is already covered since #3610: the collector writes everything as OTLP JSON to /otel/, and this PR adds Gym's spans, metrics and logs to those files. A collector with the file receiver replays them into any backend, including the docker stack from #2935, so submit-now-look-later on a laptop works without any live endpoint. |
…ctor The driver installs Gym with the telemetry extra and receives the environment that switches nemo-lens on and points it at the job collector on localhost; every Gym server process then pushes spans, metrics and logs into the pipeline the collector PR (#3610) set up. The collector keeps each producer's own service.name as the display identity, derives latency and call-count series from spans with the span_metrics connector, and runs for every job whether or not it has a model service to scrape. Logs use the collector's gRPC port because nemo-lens builds its log exporter over gRPC regardless of the protocol setting. Config: otel.gym_span_groups (default "default,verify") and otel.gym_logs (default on). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Ingo Guehring <iguehring@nvidia.com>
…ervice The collector names the scrape job <component>/<service>, so the engine's service.name.override is gym-vllm/policy, not the bare component the page said. A renderer test pins the shape. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Ingo Guehring <iguehring@nvidia.com>
…alled The telemetry extra is installed with the driver's gym_install checkout. Without one, Lens was switched on in a driver that could not import it and the run exported engine and node metrics only, with no error anywhere. The submit now fails with a message naming both ways out, and the new otel.gym_telemetry switch keeps the collector for engine and node metrics while skipping Lens entirely. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Ingo Guehring <iguehring@nvidia.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Ingo Guehring <iguehring@nvidia.com>
dc5426e to
89af61a
Compare
github-actions
Bot
removed
the
sla:author-overdue
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/36666409661 |
…RPC port The driver environment pointed OTEL_EXPORTER_OTLP_LOGS_ENDPOINT at the collector's gRPC port because the nemo-lens of #3619 built its log exporter over gRPC whatever the protocol said. Lens picks that exporter by import order, not by protocol, so which one runs depends on the environment; in the Slurm jobs it is now the OTLP/HTTP exporter, which the gRPC port answers with a connection reset. Every Gym log batch fails with "Transient error ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))" while traces and metrics flow (seen on oci-hsg job 7755363). Logs now follow the shared endpoint and protocol like the other two signals. NVIDIA-NeMo/Lens#80 makes Lens honour the protocol for logs, so the choice is deterministic once that pin lands; until then this matches what the jobs actually run. The two prefill/decode fixtures lose the override line. Validation: pytest tests/unit_tests/test_orchestration_otel.py tests/unit_tests/test_slurm_script.py tests/unit_tests/test_slurm_executor.py tests/unit_tests/test_orchestration_api.py -- 367 passed, 1 failed (test_build_vllm_ray_serve_command_single_node_model_with_space_survives_quoting shells out and needs pip on the login-shell PATH; fails identically on main here). pre-commit on the changed files passes. Cluster check in progress: oci-hsg job 7755613 runs the same scicode config with driver.env routing logs to the HTTP port. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Ingo Guehring <iguehring@nvidia.com>
…RPC port The driver environment pointed OTEL_EXPORTER_OTLP_LOGS_ENDPOINT at the collector's gRPC port because the nemo-lens of #3619 built its log exporter over gRPC whatever the protocol said. Lens picks that exporter by import order, not by protocol, and in one Slurm job the processes disagree: the driver gets the gRPC log exporter, the Gym servers get the OTLP/HTTP one. With the gRPC port configured, the servers' log batches fail with "Transient error ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))", no server log reaches the collector, and the driver spends its shutdown in the HTTP exporter's retry loop until the job hits its wall time (oci-hsg job 7755363: rollouts done at 10:11, killed at 11:00 after 477 retries). Logs now follow the shared endpoint and protocol like the other two signals, which is correct once nemo-lens selects the log exporter by protocol (NVIDIA-NeMo/Lens#80). This change must land together with that pin bump: on the current pin the driver's gRPC log exporter would aim at the HTTP port instead (oci-hsg job 7755613 showed exactly that when the port was overridden by hand). The two prefill/decode fixtures lose the override line. Validation: pytest tests/unit_tests/test_orchestration_otel.py tests/unit_tests/test_slurm_script.py tests/unit_tests/test_slurm_executor.py tests/unit_tests/test_orchestration_api.py -- 367 passed, 1 failed (test_build_vllm_ray_serve_command_single_node_model_with_space_survives_quoting shells out and needs pip on the login-shell PATH; fails identically on main here). pre-commit on the changed files passes. Cluster validation of the combined fix pending the Lens pin. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Ingo Guehring <iguehring@nvidia.com>
`_build_log_exporter` picked the gRPC log exporter whenever that package imported and fell back to HTTP otherwise, ignoring OTEL_EXPORTER_OTLP_PROTOCOL and OTEL_EXPORTER_OTLP_LOGS_PROTOCOL. Spans and metrics already resolve the protocol first and fall back on ImportError only. A process configured for http/protobuf could therefore end up with a gRPC log exporter aimed at an HTTP endpoint, or, once a caller worked around that with a per-signal gRPC endpoint, with an HTTP exporter aimed at the gRPC port. Either way every log record is dropped after a connection reset, while traces and metrics flow. Seen in NeMo Gym's Slurm jobs: the driver sets http/protobuf with the collector's HTTP endpoint plus OTEL_EXPORTER_OTLP_LOGS_ENDPOINT on the gRPC port (the workaround from NVIDIA-NeMo/Gym#3619); the HTTP log exporter then logs "Transient error ... Connection reset by peer" for each batch. Logs now follow the same rule as the other two signals: the signal-specific protocol wins, then the general one, default grpc, with the existing import fallback. Three tests mirror the span-exporter protocol tests. Signed-off-by: Ingo Guehring <iguehring@nvidia.com>
…80) * fix: select the OTLP log exporter by protocol like spans and metrics `_build_log_exporter` picked the gRPC log exporter whenever that package imported and fell back to HTTP otherwise, ignoring OTEL_EXPORTER_OTLP_PROTOCOL and OTEL_EXPORTER_OTLP_LOGS_PROTOCOL. Spans and metrics already resolve the protocol first and fall back on ImportError only. A process configured for http/protobuf could therefore end up with a gRPC log exporter aimed at an HTTP endpoint, or, once a caller worked around that with a per-signal gRPC endpoint, with an HTTP exporter aimed at the gRPC port. Either way every log record is dropped after a connection reset, while traces and metrics flow. Seen in NeMo Gym's Slurm jobs: the driver sets http/protobuf with the collector's HTTP endpoint plus OTEL_EXPORTER_OTLP_LOGS_ENDPOINT on the gRPC port (the workaround from NVIDIA-NeMo/Gym#3619); the HTTP log exporter then logs "Transient error ... Connection reset by peer" for each batch. Logs now follow the same rule as the other two signals: the signal-specific protocol wins, then the general one, default grpc, with the existing import fallback. Three tests mirror the span-exporter protocol tests. Signed-off-by: Ingo Guehring <iguehring@nvidia.com> * test: cover the OTLP log exporter import fallbacks Exercise the HTTP-to-gRPC and gRPC-to-HTTP fallbacks and the error when neither exporter package is installed. Signed-off-by: Ingo Guehring <iguehring@nvidia.com> --------- Signed-off-by: Ingo Guehring <iguehring@nvidia.com>
Standalone against
mainsince #3610 (the collector) merged as f2f446a. Revision of 2026-09-30: rebased onto main (28c6035), the feature squashed into one commit, plus the two review fixes as their own commits (display-name doc, submit-time check for the telemetry extra) and a comment trim.What
Gym's own servers become producers for the collector that #3610 runs beside every job:
telemetryextra and started with the environment that switches nemo-lens on and points it at the collector on localhost:NEMO_GYM_OTEL_ENABLED,NEMO_GYM_OTEL_RUN_ID(the gym job id),NEMO_GYM_OTEL_SPAN_GROUPS,NEMO_GYM_OTEL_LOGS_ENABLED,OTEL_EXPORTER_OTLP_ENDPOINT/_PROTOCOL. Every server the driver starts inherits it; an explicitdriver.envvalue wins.otel.gym_telemetry(default on) switches Gym's own nemo-lens telemetry; it needsdriver.gym_install, where thetelemetryextra is installed, and the submit fails with a message naming both ways out when that is missing. Off keeps the collector for engine and node metrics only.otel.gym_span_groups(defaultdefault,verify) selects Gym's span groups for the job;otel.gym_logs(default on) ships Gym's Python logging as OTel logs. Because nemo-lens builds its log exporter over gRPC regardless of the protocol setting, logs get the collector's gRPC port throughOTEL_EXPORTER_OTLP_LOGS_ENDPOINT.service.nameasservice.name.overridebefore stamping the routing identity (transform/identity), accepts OTLP metrics, and derives latency histograms and call counters from spans with thespan_metricsconnector, keyed by span name, producer and sandbox provider, so sandbox start/exec latency and error rates need no metric of their own.type: apipolicy or a CPU-only judge run produces rollout and verify telemetry all the same. The Prometheus receiver is rendered only when there are scrape jobs.Why
#3610 gives engine and node metrics with a per-run identity. This PR adds the application layer: how long rollouts and verifications take, one trace per rollout across agent, model and resources servers, request latency per route, sandbox operations, and trace-correlated logs, all under the same
run_id/userlabels on the same boards.No issue filed; follow-up to #3610 as planned there.
Validation
pytest tests/unit_tests/test_orchestration_otel.py tests/unit_tests/test_slurm_script.py tests/unit_tests/test_slurm_executor.py tests/unit_tests/test_orchestration_api.py tests/unit_tests/test_fern_docs_links.py: 363 passed (including the docs-link tests). The executor bookkeeping tests, which build model-less configs, now setotel.enabled: false, since the collector no longer skips such jobs.pre-commit run --files ...on the changed files: pass.Rollout evidence
Slurm cluster, vLLM policy,
gym eval submit, nemo-lens in the driver venv:gym.rollout.duration_ms,gym.verify.*(with theverifygroup) and FastAPI request histograms per Gym server in the metrics backend; one trace per job visible in the traces backend withrun_idon every span.traces_span_metrics_*series forgym.sandbox.start/gym.sandbox.execwith the provider label, including error status on failed creates.Compatibility
Adds fields with defaults; no change for jobs with a model service beyond the extra telemetry. Jobs without a model service now get a collector too (they got none in #3610).
otel.gym_span_groups/gym_logstune or turn off Gym-side telemetry;otel.enabled: falseturns the whole thing off.