feat(telemetry): task id, repeat index, reward and health verdict as span attributes of a rollout #4988

Problem

The rollout id on every span is <task index>-<repeat index> (for example 45-0), built from _ng_task_index and _ng_rollout_index on the collector row. The task name (task_id, for example terminal-bench/foodstuff-beta-activity) is on the same row and in rollouts.jsonl, but on no span, metric or log. The reward is likewise only in rollouts.jsonl, MLflow and the bounded counter gym.rollout.rewarded_total (range label, no rollout id); the trajectory health verdict only in rollout_verdicts.jsonl and the counters gym.rollout.health_verdict_total / gym.rollout.health_flagged_total (verdict and check labels, no rollout id). The rollout anatomy and drilldown boards (FEP-1817) can therefore only show numeric ids and no outcome, and comparing repeats of one task or reward against engine time means leaving Grafana.

Change

Four per-rollout facts on spans:

• On gym.rollout.dispatch (_dispatch_span in nemo_gym/rollout_collection.py, the row is in hand): the task id (task_id, falling back to task_name, omitted when the row has neither) and the repeat index.
• On gym.verify (environment server, the verify call returns the reward): the numeric reward. Unscored rollouts (infrastructure failures, no top-level reward) get no attribute. Alternative: on the dispatch span, which then has to stay open until the response body is read (today it closes when the response object arrives, two lines before get_response_json).
• The health verdict in a child span gym.rollout.health of the dispatch span, with the verdict and the flagged check names as attributes. The verdict is computed on the driver in the result loop (_record_rollout_health), after the dispatch span has closed, so the dispatch span's context is kept when it closes and the health span is started under it afterwards (a child may start after its parent ended); it then sits in the rollout's trace and carries the rollout id. Values are bounded (healthy / unhealthy / unobserved, fixed set of check names). Do not widen the dispatch span to cover the result handling: its duration is the rollout's wall clock on the boards.

Spans only, not metrics: the reward stays a bounded range on gym.rollout.rewarded_total; task id and reward have per-rollout cardinality. The rollout id stays the join key, since the engine spans carry only that.

Why spans

A span attribute describes the operation the span represents: the reward is the outcome of the verify operation (as http.response.status_code or gen_ai.response.finish_reasons are outcomes), task id and repeat index are the identity of the dispatch (as http.route). The boards join signals by rollout id in one store, so per-rollout facts have to be in that store; rollouts.jsonl and MLflow cannot be joined with engine spans. A log record per rollout would also work but is a second place for the same fact and unusable next to traces in Tempo.

Attribute names follow Gym's OTel conventions owned by the Rollout Observability project (#3182); proposed nemo.gym.task.id, nemo.gym.rollout.repeat, nemo.gym.rollout.reward, and on the health span the existing nemo.gym.health.verdict plus nemo.gym.health.checks, to be reviewed from that side. The reward is a model outcome in telemetry (a number, not content); say so in the PR.

Boards afterwards

Join on the rollout id: the rollout dropdown shows foodstuff-beta-activity (repeat 1), tables get task, reward, health and flags columns (one maxIf / groupArrayIf each) and can group by task to compare repeats or sort by verdict to pick the rollouts worth drilling into, the anatomy board can plot reward against engine time. SQL falls back to the index when the attribute is absent (older runs, datasets without a readable task name).

Done when

• The dispatch span of a tb4 run carries the task id and repeat index, its verify span the reward, and a gym.rollout.health span in the same trace the verdict and flagged checks (check in ClickHouse: SpanAttributes by SpanName).
• The anatomy and drilldown boards show task names, reward and health verdict when present (EFB generator change).