Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 5 additions & 6 deletions nemo_gym/telemetry/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,13 +68,12 @@ Gym serves at 16k+ concurrency, so per-request sites are hot paths. Attribute di
f-strings go *inside* the gate too, never in the arguments of a call above it.
`tests/unit_tests/telemetry/test_overhead.py` guards the disabled-path cost.

**2. `_fallbacks.py` is one of four synchronised copies.**
**2. `_fallbacks.py` stands in for nemo-lens when it is absent.**

`kb/knowledge/conventions/fallback-sync.md`: nemo-lens's no-op surface exists in
`nemo/lens/fallbacks.py` plus one `_fallbacks.py` per consumer. When a signature changes
in lens, change it here in the same PR — a drift only breaks the configuration where lens
is *absent*, which is the one nobody runs by accident.
`tests/unit_tests/telemetry/test_fallbacks.py` compares the two parameter-for-parameter.
Its no-op branch provides only the primitives Gym calls, and must accept every call Gym
makes. A mismatch breaks only the configuration where lens is *absent*, which is the one
nobody runs by accident, so `tests/unit_tests/telemetry/test_fallbacks.py` exercises it
with lens blocked.

Note this file differs from NeMo-RL's: it resolves to the **real** `nemo.lens.helpers`
implementations when lens is installed, so a call site needs one import rather than its
Expand Down
7 changes: 2 additions & 5 deletions nemo_gym/telemetry/_fallbacks.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,8 @@
Gym's call sites to a single import and removes the chance of a site binding the no-op
while lens is present.

The no-op branch below is one of the four places named in
``kb/knowledge/conventions/fallback-sync.md``. It mirrors ``nemo/lens/fallbacks.py`` at
commit ``b0f977d4``; when a signature changes there, change it here in the same PR.
``tests/unit_tests/telemetry/test_fallbacks.py`` asserts the two agree parameter-for-
parameter whenever lens is importable.
The no-op branch below provides only what Gym calls, and must accept every call Gym
makes. ``tests/unit_tests/telemetry/test_fallbacks.py`` exercises it with lens blocked.
"""

try:
Expand Down
7 changes: 3 additions & 4 deletions nemo_gym/telemetry/spans.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,9 @@
called the model server, so Jaeger/Tempo/Honeycomb service maps lose the edge and
client-vs-server latency attribution stops working.

Gym is pinned to nemo-lens ``b0f977d4``, so rather than change lens this module creates
that one span directly against the OTel API. Adding a ``kind`` parameter to
``managed_span`` is the proper fix and is raised as a decision in the PR body; when it
lands, delete this module and pass ``kind=`` instead.
Rather than wait for a lens change, this module creates that one span directly against the OTel API.
The proper fix is a ``kind`` parameter on ``managed_span``.
Once the pinned nemo-lens has one, delete this module and pass ``kind=`` instead.

Everything else is deliberately identical to ``managed_span``: context attach/detach,
exception recording, and an unconditional ``end()`` in ``finally``.
Expand Down
4 changes: 2 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -383,7 +383,7 @@ telemetry = [
# Pinned to a Lens commit via [tool.uv.sources] until 0.2.0 ships to PyPI; then drop that
# entry and use a version range. NeMo-RL pins this same commit; the two must agree,
# since an RL job installs Gym into its environment and resolves one nemo-lens.
# Updated: Thu Oct 1, 2026 with nemo-lens @ b0f977d4
# Updated: Thu Oct 8, 2026 with nemo-lens v0.3.0 @ 7bcc03ce
# License: Apache 2.0 https://github.com/NVIDIA-NeMo/Lens/blob/main/LICENSE
"nemo-lens[sdk]",

Expand Down Expand Up @@ -535,7 +535,7 @@ exclude-dependencies = [
# venvs that `gym env test` / `gym env start` build, which inherit it through the editable
# nemo-gym install exactly as override-dependencies does.
[tool.uv.sources]
nemo-lens = { git = "https://github.com/NVIDIA-NeMo/Lens.git", rev = "b0f977d414b2f89938604a0b7eaa78ee08bc8700" }
nemo-lens = { git = "https://github.com/NVIDIA-NeMo/Lens.git", rev = "7bcc03ce3565d2be6858b819b049bcc4c3ec05b4" }

[tool.setuptools.package-data]
# The built-in asset trees (resources_servers, responses_api_agents, responses_api_models,
Expand Down
106 changes: 8 additions & 98 deletions tests/unit_tests/telemetry/test_fallbacks.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,54 +12,18 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""The nemo-lens-absent path, and the shim-parity rule that guards it.
"""The nemo-lens-absent path: Gym's no-op stand-ins, and binding the real helpers otherwise.

``kb/knowledge/conventions/fallback-sync.md``: Lens's no-op surface exists in four places
— ``nemo/lens/fallbacks.py`` plus one ``_fallbacks.py`` per consumer. A signature that
drifts breaks consumers only in the configuration where lens is *absent*, which is the
configuration nobody runs by accident. These tests make that configuration routine.
Installs without the ``telemetry`` extra have no nemo-lens, so every instrumentation site
runs against the stand-ins in ``nemo_gym/telemetry/_fallbacks.py``. That configuration is
the one nobody runs by accident; these tests make it routine.
"""

import inspect

import pytest

from tests.unit_tests.telemetry.conftest import import_without_lens


#: Every name Gym's shim is required to provide. Adding a name to
#: ``nemo/lens/fallbacks.py`` without adding it here fails ``test_shim_covers_every_lens_fallback``.
EXPECTED_NAMES = frozenset(
{"trace_fn", "managed_span", "span_cm", "is_span_group_enabled", "safe_set_span_attributes"}
)

#: Lens fallbacks Gym deliberately does not mirror. ``SpanRegistry`` is only touched by
#: ``nemo_gym.telemetry.span_groups``, which skips registration when lens is absent.
NOT_MIRRORED = frozenset({"SpanRegistry"})


def signature_map(module, names):
"""Map name -> parameter list for *names* on *module*."""
return {name: list(inspect.signature(getattr(module, name)).parameters.values()) for name in names}


def compare_signatures(reference, candidate, names):
"""Return a list of human-readable drift descriptions, empty when the two agree.

Factored out so ``test_comparison_detects_drift`` can prove this helper actually
catches a mismatch — without that, a parity test that never sees drift asserts
nothing.
"""
problems = []
ref_sigs = signature_map(reference, names)
cand_sigs = signature_map(candidate, names)
for name in sorted(names):
ref, cand = ref_sigs[name], cand_sigs[name]
if [(p.name, p.kind, p.default) for p in ref] != [(p.name, p.kind, p.default) for p in cand]:
problems.append(f"{name}: lens has {ref}, Gym shim has {cand}")
return problems


@pytest.fixture(scope="module")
def gym_shim_without_lens():
"""Gym's ``_fallbacks`` module imported with nemo-lens unavailable (the no-op branch)."""
Expand All @@ -72,72 +36,18 @@ def test_telemetry_package_imports_without_lens():
assert module.NEMO_LENS_AVAILABLE is False


def test_shim_provides_every_expected_name(gym_shim_without_lens):
for name in sorted(EXPECTED_NAMES):
assert hasattr(gym_shim_without_lens, name), f"Gym's no-lens shim is missing {name}"


def test_shim_covers_every_lens_fallback(gym_shim_without_lens):
"""Gym's shim must cover every public no-op nemo-lens declares — no more, no less.

Fails when lens grows a fallback Gym has not mirrored, which is the drift direction a
per-name check would miss.
"""
lens_fallbacks = pytest.importorskip("nemo.lens.fallbacks")
lens_names = {
name
for name, obj in vars(lens_fallbacks).items()
if not name.startswith("_") and inspect.isfunction(obj) and obj.__module__ == lens_fallbacks.__name__
}
# contextmanager-wrapped functions report the decorated module; pick them up by name.
lens_names |= {
name for name in vars(lens_fallbacks) if not name.startswith("_") and callable(getattr(lens_fallbacks, name))
} - {"contextmanager"}
lens_names -= NOT_MIRRORED

assert lens_names == EXPECTED_NAMES, (
"nemo.lens.fallbacks changed its public surface. Update nemo_gym/telemetry/_fallbacks.py "
f"and EXPECTED_NAMES in the same PR. lens={sorted(lens_names)} expected={sorted(EXPECTED_NAMES)}"
)


def test_shim_signatures_match_lens(gym_shim_without_lens):
"""Gym's no-op shim must match nemo.lens.fallbacks parameter-for-parameter."""
lens_fallbacks = pytest.importorskip("nemo.lens.fallbacks")
problems = compare_signatures(lens_fallbacks, gym_shim_without_lens, EXPECTED_NAMES)
assert not problems, "Gym's _fallbacks.py has drifted from nemo.lens.fallbacks:\n" + "\n".join(problems)


def test_comparison_detects_drift(gym_shim_without_lens):
"""The parity check must actually fail on a mismatch.

Without this, ``test_shim_signatures_match_lens`` would pass just as happily against a
comparison that never reports anything — the failure mode
``kb/knowledge/conventions/test-strength.md`` exists for.
"""
lens_fallbacks = pytest.importorskip("nemo.lens.fallbacks")

class DriftedShim:
# Renamed parameter (group -> span_group) and a dropped keyword.
@staticmethod
def is_span_group_enabled(span_group):
return False

problems = compare_signatures(lens_fallbacks, DriftedShim, ["is_span_group_enabled"])
assert problems, "compare_signatures failed to notice a renamed parameter"
assert "is_span_group_enabled" in problems[0]


def test_no_op_primitives_are_inert(gym_shim_without_lens):
"""Every shim behaves as a no-op, not merely as an importable name."""
shim = gym_shim_without_lens

assert shim.is_span_group_enabled("server") is False
assert shim.is_span_group_enabled("anything-at-all") is False

with shim.managed_span("server", "gym.test") as span:
# Called the way Gym's call sites call them: dotted attribute names as keywords.
attributes = {"nemo.gym.sandbox.provider": "test"}
with shim.managed_span("server", "gym.test", **attributes) as span:
assert span is None
with shim.span_cm("gym.test") as span:
with shim.span_cm("gym.test", **attributes) as span:
assert span is None

calls = []
Expand Down
6 changes: 3 additions & 3 deletions uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading