Skip to content

fix: enable no span group when traces are disabled - #87

Open
ananthsub wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
ananthsub:ananthsub/no-span-groups-without-traces
Open

ananthsub wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
ananthsub:ananthsub/no-span-groups-without-traces

Conversation

@ananthsub

Copy link
Copy Markdown

What does this PR do?

With traces disabled, setup_telemetry now enables no span group, so managed_span and trace_fn sites return before touching OpenTelemetry instead of building spans that no exporter will receive.

Before this change, a process with enabled=True and traces_enabled=False still stored the span-group spec. Every enabled group passed the gate in managed_span and trace_fn, so each gated site started a span on the no-op tracer, attached it to the current context, set its attributes, and ended it. That is a per-call cost on the hot path of every metrics-only process, in every consuming library, for spans that are discarded.

setup_telemetry now pins the empty set when traces_enabled is false, the same way it already does when telemetry is disabled. Pinning drops the stored spec, so a library that registers its groups after setup cannot enable one either. With traces enabled, nothing changes.

Issues

No issue. Found while measuring metrics-only overhead in NeMo Gym, where the same gap made Gym skip span-only work itself (NVIDIA-NeMo/Gym#4481).

A consumer that checks is_span_group_enabled before recording a metric would stop recording that metric in metrics-only runs once this lands. NeMo Gym did that for its verification, rollout, and sandbox metrics, and NVIDIA-NeMo/Gym#4612 moves those metrics onto the metrics-exporter check. NVIDIA-NeMo/Gym#4612 should merge before Gym bumps its nemo-lens pin past this change.

Usage

No API change. A metrics-only setup now behaves like this:

from nemo.lens import NemoLensConfig, enabled_span_groups, managed_span, setup_telemetry

setup_telemetry(NemoLensConfig(enabled=True, traces_enabled=False, span_groups="all", exporter="console"))
assert enabled_span_groups() == frozenset()
with managed_span("step", "train.step") as span:
    assert span is None  # the body runs; no span is created

Before your PR is "Ready for review"

Pre checks:

  • I read the Contributor guidelines.
  • All commits are signed off (git commit -s) per the DCO.
  • I added or updated tests under tests/ for any behavior change.
  • I ran pytest and pre-commit run --all-files locally; both pass.
  • I updated docs/ and README.md where behavior, configuration, or public API changed.
  • If this changes a lens public API symbol, I updated the cross-repo _fallbacks.py files in Megatron-LM / RL / Gym.

Additional information

  • No public symbol or signature changes, so the consumer _fallbacks.py files need no edit.
  • tests/test_handle.py covers the new behavior: with traces disabled no group is enabled and managed_span yields None, and a library that registers after setup stays disabled. Both tests fail without the change.
  • docs/user-guide/configuration.mdx documents the effect of traces_enabled on span groups, and docs/user-guide/troubleshooting.mdx notes that enabled_span_groups() is empty while traces are disabled. README.md does not describe traces_enabled.
  • Validated against NeMo Gym: Gym's unit suite on fix(telemetry): record metrics whenever metrics export, independent of span groups Gym#4612 runs the same against this branch as against the current pin.

With telemetry enabled and traces disabled, setup_telemetry still stored the span-group spec, so
every enabled group passed the gate in managed_span and trace_fn. Each gated site then started,
attached and ended a span on a no-op tracer. That is a per-call cost for spans no exporter will
receive, paid in every metrics-only process of every consuming library.

Setup now pins the empty set when traces are disabled, as it already does when telemetry is
disabled, so gated sites return before touching OpenTelemetry. Pinning drops the spec, so a
library that registers its groups after setup does not enable one either.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

@ahmadki

ahmadki commented Oct 21, 2026

Copy link
Copy Markdown
Member

/ok to test 5e29619

This branch was successfully deployed

1 active deployment
public — 4cb79c44 Deployed Oct 21, 2026 by copy-pr-bot[bot] via release / finalize / notify #174
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants