feat(telemetry)!: rl.iteration on every span and the shared nv.dl.* attributes

docs/observability/configuration.md

@@ -86,16 +86,31 @@ Each process sets stable-for-the-run values on its OTel `Resource` — `init_tel
 | Attribute | Source |
 |---|---|
 | `nemo.run.id` | the run id resolved above (`NEMO_RL_OTEL_RUN_ID` / `SLURM_JOB_ID` / generated). Emitted by lens from the `run_id` config field — see [Run identification](#run-identification) |
-| `nv.dl.campaign.stage` | always the literal `RL`. Set by every NeMo-RL process, driver and worker, so a backend collecting several stages of a model's lifecycle (pretrain → SFT → RL) can select this stage without matching on service names |
+| `nv.dl.campaign.stage` | the lifecycle stage this run occupies: `SFT` for `sft` and `sft_v2`, `RM` for `rm`, `RL` for everything else including DPO. Set by every NeMo-RL process, driver and worker, so a backend collecting several stages of a model's lifecycle (pretrain → SFT → RL) can select one stage without matching on service names |
 | `rl.algorithm` | the `algorithm="<algo>"` passed to `init_telemetry_driver` |
 | `rl.model` | `policy.model_name` |
 | `nemo.precision` | `policy.precision` |
-| `dl.tensor_parallel.size` | `policy.megatron_cfg` / `automodel_cfg` TP size |
-| `dl.pipeline_parallel.size` | `policy.megatron_cfg` PP size |
+| `nv.dl.provider.name` | always the literal `nemo-rl`: which training framework produced the telemetry, for a backend collecting several. Separate from the service name, which a user may rename |
+| `nv.dl.run.uuid` | lens's `derive_nv_dl_run_uuid` over the run id — a uuid5, not the raw id, which is reported separately as `nemo.run.id`. A backend joining RL runs to Megatron ones groups on the derived value, so the raw id under this key would match nothing |
+| `nv.dl.topology.size.tp` | TP size, read from whichever backend `policy.megatron_cfg.enabled` selects. Every config carries both blocks populated and only the live one is read, so a disabled `megatron_cfg` never contributes |
+| `nv.dl.topology.size.pp` | PP size, Megatron runs only — Automodel has no pipeline dimension |
+| `nv.dl.training.config.global_batch_size` | `policy.train_global_batch_size` |
+| `nv.dl.training.config.micro_batch_size` | `policy.train_micro_batch_size` |
+| `nv.dl.training.config.sequence_length` | `policy.max_total_sequence_length` |
+| `nv.dl.training.config.optimizer` | on Megatron `policy.megatron_cfg.optimizer.optimizer` (`adam`), otherwise `policy.optimizer.name` (`torch.optim.AdamW`) — the backends both spell it and store it differently, and only the enabled one is read |
+| `nv.dl.training.config.recompute_granularity` | Megatron runs only: `policy.megatron_cfg.recompute_granularity`, and only when `activation_checkpointing` is on. Megatron ignores the granularity otherwise, so reporting it would describe recompute that is not happening |
+| `nv.dl.training.target.train_iters` | `<algo>.max_num_steps`, when positive. This is the configured cap, not a prediction: many configs set a placeholder (`grpo_math_1B` uses `1000000` with `max_num_epochs: 1`) and stop earlier on epochs. Looked up by algorithm name, so an entry point that keeps its settings under a differently named section reports no target at all — `sft_v2` reads `master_config.sft`, and omits this. `rm` spells "train for one epoch" as `-1`, which is omitted rather than reported as a target |
+| `nv.dl.software.torch`, `.cuda`, `.nccl` | versions as loaded in the process that opened the span, not as pinned in the lockfile — they are per process, so filter on `rl.worker_group` to read one backend's versions. CUDA and NCCL are omitted on a CPU-only build. SGLang, TRT-LLM and Dynamo workers do not report versions yet |
+| `nv.dl.software.transformer_engine` | the installed distribution's version, read from package metadata rather than by importing it — importing transformer-engine loads its PyTorch extension in every process that sets up telemetry. Omitted when it is not installed, which is every process except Megatron and Automodel workers: its absence on a driver, controller or vLLM span does not mean the run lacked TE |
 | `nv.dl.rank`, `nv.dl.world_size` | this process's rank and group size (`RANK` / `WORLD_SIZE`, or `0` / `1` for the driver and singleton actors) |
+| `nv.dl.local_rank` | worker processes only: `LOCAL_RANK`, which `RayWorkerGroup` exports beside `RANK` as the worker's bundle index in its placement group. Node-local only when each node has its own placement group — a generation backend spanning nodes requests a single unified one (SGLang always; vLLM, TRT-LLM and Megatron when their parallelism crosses nodes) and the index then runs across nodes. Not a physical device index either, since Ray gives each worker its own `CUDA_VISIBLE_DEVICES` |
 | `rl.worker_group` | worker processes only: the worker group's `name_prefix` (`lm_policy`, `vllm_policy`, ...), from `NRL_WORKER_GROUP`. The `NemoGym` actor reports the literal `nemo_gym` — it is built from the environment registry rather than by `RayWorkerGroup`, so nothing sets the env var for it |
 
-Attribute construction is best-effort: a missing config key simply omits that attribute; it never raises. Plus auto-detected host / GPU / SLURM / Kubernetes attributes from lens's resource detection.
+The config-derived rows are driver-only: a worker is handed the propagated environment, not the `MasterConfig`. Attribute construction is best-effort: a missing config key simply omits that attribute; it never raises. Plus auto-detected host / GPU / SLURM / Kubernetes attributes from lens's resource detection.
+
+Shared `nv.dl.*` keys are taken from `nemo.lens.semconv` rather than written as strings, so a typo is an `ImportError` instead of silent data loss. Note the two spellings of the parallelism sizes: `nv.dl.topology.size.*` is the resource-scoped one and the `dl.tensor_parallel.*` family is the span and metric spelling of the same quantity. That split is a convention this repo keeps, not something lens checks — `normalize_resource_attributes` runs only over the attributes lens derives from `OTEL_RESOURCE_ATTRIBUTES`, and the `resource_attributes` RL passes in are merged as they are. Either spelling would arrive intact, so the wrong one is a silently misnamed series rather than a failure.
+
+Lens also declares `nv.dl.topology.size.dp`, which NeMo-RL does not report. RL configures no data-parallel size, so it could only be derived by dividing the cluster's GPU count by the model-parallel sizes — and that assumes the policy owns every GPU in the cluster. A reward model with its own `resources` block, distillation teachers, expert parallelism and non-colocated generation each leave the policy fewer, and the division comes out too high rather than failing. Read the worker's device mesh if you need the real number.
 
 ## Typical configurations
 

docs/observability/extending.md

@@ -161,14 +161,18 @@ For a hot path where you want minimal cost when the group is disabled:
 
 ```python
 with umbrella_span(RLSpanGroup.U_ROLLOUT, "rl.grpo.generation",
-                   **{"rl.iteration": iteration}) as span:
+                   **{"rl.num_prompt_groups": n_groups}) as span:
     result = collect()
     if span is not None:
         span.set_attribute("rl.num_generations_per_prompt", n)
 ```
 
 Both yield `None` when the group is disabled; the body still runs, so guard attribute-setting with `if span is not None`.
 
+Do not pass `rl.iteration` here. The training loops set it once per step with
+`iteration_scope`, so any span opened inside a step already carries it — see
+[`rl.iteration` is set once per step](span-groups.md#rliteration-is-set-once-per-step).
+
 ### Always-on block — `span_cm`
 
 `span_cm` always creates a span when telemetry is active (no group gate) — for cold, top-level paths only:
@@ -186,10 +190,14 @@ if telemetry is not None:
 |---|---|---|
 | Span name | `rl.<algorithm>.<phase>`, matching the block's `Timer` key (the two umbrella spans excepted — see [span groups](span-groups.md#per-algorithm-span-names)) | `rl.grpo.generation` |
 | Span tag | `rl.<attr>` categorical | `rl.iteration`, `rl.backend` |
-| Resource attribute | `rl.<attr>` / shared `dl.<attr>` | `rl.model`, `dl.tensor_parallel.size` |
+| Resource attribute | `rl.<attr>` for RL's own, shared `nv.dl.<attr>` for anything a distributed training job has | `rl.model`, `nv.dl.topology.size.tp` |
 | Metric name | `rl.<subsystem>.<metric>` (application scope) | `rl.efficiency.seconds` |
 
-Metric names use the **application scope** (`rl.*`) — never `dl.*`. NeMo-RL names its own attributes, in `nemo_rl/telemetry/`, rather than importing them from `nemo.lens.semconv`: they describe RL's properties, so RL is where a rename has to happen. The lens constants are for the few attributes lens itself sets, such as `nv.dl.rank`.
+**Metric names use the application scope (`rl.*`) — never `dl.*`.** This is not a style preference: lens registers a `dl.*` series for the trainer that owns the step, so an RL job publishing `dl.grad_norm` would be writing into a co-resident Megatron trainer's series. `rl.grad_norm` and `rl.learning_rate` are deliberate spelling matches for exactly that reason, so query the two by their own names rather than by swapping the prefix. Lens's own semconv file says the same: RL's metric series are declared through `register_metric_group` in this tree and are deliberately not defined there.
+
+**Attributes split by who owns the name.** RL names every `rl.*` attribute in `nemo_rl/telemetry/`, so a rename lands where the producers are. That holds even for the few `rl.*` keys lens also declares in its semconv (`rl.algorithm`, `rl.reward`, `rl.generation.backend`, `rl.num_rollouts`): lens declares those and reads none of them, so importing the constant would buy a dependency with no contract behind it while RL remains the only producer. Import from `nemo.lens.semconv` only for names lens actually acts on — the shared `nv.dl.*` resource surface — where a typo should fail as an `ImportError` rather than become silent data loss.
+
+Mind the scope when picking a shared name: `nv.dl.topology.size.tp` is the resource-scoped parallelism size, while `dl.tensor_parallel.size` is the span and metric spelling of the same quantity.
 
 ## Choosing a span group
 

docs/observability/index.md

@@ -60,7 +60,9 @@ Keeping the settings in the config file is what makes a run's telemetry reproduc
 
 ## What gets instrumented
 
-Each algorithm's `examples/run_<algo>.py` calls `init_telemetry_driver(config, algorithm="<algo>")` **before** `init_ray()` (so the resolved `NEMO_RL_OTEL_*` settings are snapshotted into the Ray `runtime_env` and inherited by every worker) and `shutdown_telemetry()` from a `finally` block wrapping the whole run, so buffered spans are flushed even when the run fails.
+The instrumented entry points — those in the table below, plus `examples/run_sft_v2.py` — call `init_telemetry_driver(config, algorithm="<algo>")` **before** `init_ray()` (so the resolved `NEMO_RL_OTEL_*` settings are snapshotted into the Ray `runtime_env` and inherited by every worker) and `shutdown_telemetry()` from a `finally` block wrapping the whole run, so buffered spans are flushed even when the run fails.
+
+Not every entry point is wired up: `run_eval.py`, `run_grpo_sliding_puzzle.py`, `run_vlm_grpo.py`, `run_vlm_mpo.py`, `run_vlm_sft.py` and `run_xtoken_off_policy_distillation.py` emit nothing today. A run launched through one of those produces no spans and no metrics, whatever `telemetry.enabled` says.
 
 | Algorithm | Entry point | Representative spans |
 |---|---|---|
@@ -80,7 +82,7 @@ Each span belongs to a **span group** that controls whether it is emitted at run
 ## What gets exported
 
 - **Traces**: any OTLP-compatible backend (Jaeger, Grafana Tempo, an OpenTelemetry Collector, ...) via OTLP.
-- **Metrics**: the `rl.efficiency.*` async accounting teed from the driver's metrics logger, the training scalars (`rl.reward.mean`, `rl.policy.loss`, …), `rl.setup.duration` for the startup phases, the `rl.vllm.*` engine deltas, and the driver-side `gen_ai.*` generation series — see [Metrics](metrics.md).
+- **Metrics**: the `rl.efficiency.*` async accounting teed from the driver's metrics logger, the training scalars (`rl.reward.mean`, `rl.policy.loss`, …), `rl.setup.duration` for the startup phases, the `rl.vllm.*` engine deltas, the driver-side `gen_ai.*` generation series, and `rl.host.memory.*` per node — see [Metrics](metrics.md).
 - **Logs** (optional): via the OTel log bridge when `telemetry.logs_enabled` is true — correlates Python `logging` records with the active span's trace ID.
 
 **Every** process that enables telemetry exports, each labelled with `nv.dl.rank` / `nv.dl.world_size`. Narrowing a large fleet down is a collector-side filter rather than a config setting — see [Configuration — Which ranks export](configuration.md#which-ranks-export).