Repository navigation
Conversation
With telemetry enabled and traces disabled, setup_telemetry still stored the span-group spec, so every enabled group passed the gate in managed_span and trace_fn. Each gated site then started, attached and ended a span on a no-op tracer. That is a per-call cost for spans no exporter will receive, paid in every metrics-only process of every consuming library. Setup now pins the empty set when traces are disabled, as it already does when telemetry is disabled, so gated sites return before touching OpenTelemetry. Pinning drops the spec, so a library that registers its groups after setup does not enable one either. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Contributor
|
🌿 Preview your docs: https://nvidia-preview-pr-87-4cb79c447970.docs.buildwithfern.com/nemo/lens |
ahmadki
approved these changes
Oct 21, 2026
Member
|
/ok to test 5e29619 |
7 tasks done
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
With traces disabled,
setup_telemetrynow enables no span group, somanaged_spanandtrace_fnsites return before touching OpenTelemetry instead of building spans that no exporter will receive.Before this change, a process with
enabled=Trueandtraces_enabled=Falsestill stored the span-group spec. Every enabled group passed the gate inmanaged_spanandtrace_fn, so each gated site started a span on the no-op tracer, attached it to the current context, set its attributes, and ended it. That is a per-call cost on the hot path of every metrics-only process, in every consuming library, for spans that are discarded.setup_telemetrynow pins the empty set whentraces_enabledis false, the same way it already does when telemetry is disabled. Pinning drops the stored spec, so a library that registers its groups after setup cannot enable one either. With traces enabled, nothing changes.Issues
No issue. Found while measuring metrics-only overhead in NeMo Gym, where the same gap made Gym skip span-only work itself (NVIDIA-NeMo/Gym#4481).
A consumer that checks
is_span_group_enabledbefore recording a metric would stop recording that metric in metrics-only runs once this lands. NeMo Gym did that for its verification, rollout, and sandbox metrics, and NVIDIA-NeMo/Gym#4612 moves those metrics onto the metrics-exporter check. NVIDIA-NeMo/Gym#4612 should merge before Gym bumps its nemo-lens pin past this change.Usage
No API change. A metrics-only setup now behaves like this:
Before your PR is "Ready for review"
Pre checks:
git commit -s) per the DCO.tests/for any behavior change.pytestandpre-commit run --all-fileslocally; both pass.docs/andREADME.mdwhere behavior, configuration, or public API changed._fallbacks.pyfiles in Megatron-LM / RL / Gym.Additional information
_fallbacks.pyfiles need no edit.tests/test_handle.pycovers the new behavior: with traces disabled no group is enabled andmanaged_spanyieldsNone, and a library that registers after setup stays disabled. Both tests fail without the change.docs/user-guide/configuration.mdxdocuments the effect oftraces_enabledon span groups, anddocs/user-guide/troubleshooting.mdxnotes thatenabled_span_groups()is empty while traces are disabled.README.mddoes not describetraces_enabled.