Skip to content

feat(video): add SA-V tracks verifier - #3504

Open
ehosseiniasl wants to merge 2 commits into
mainfrom
ehsan/sav-tracks-video-verifier-final
Open

feat(video): add SA-V tracks verifier#3504
ehosseiniasl wants to merge 2 commits into
mainfrom
ehsan/sav-tracks-video-verifier-final

Conversation

@ehosseiniasl

@ehosseiniasl ehosseiniasl commented Sep 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Adds the SA-V tracks-schema resource server used by the combined SA-V + CapRL video GRPO teacher run.

  • scores point, box, and text-referred tracking rows
  • supports COCO RLE mask membership, box IoU, visibility/absence credit, and one-to-one ID assignment
  • includes the training config, format documentation, example data, and focused tests

This is a video-only commit, rebased onto current Gym main.

Validation

  • resources_servers/sav_tracks/tests/test_app.py: 25 passed
  • git diff --check: clean

Authored by Pulkit Kumar; branch prepared by @ehosseiniasl for the final Super 3.5 video training handoff.

Current-main and full-stack validation — 2026-09-30

  • Repository main is now da3c2044; this PR head 25e2c505 merges cleanly with it. The non-destructive synthetic merge produced tree da2b72f9ea42c8087200a421e38d5609b66c55dc, git diff --check passed, and the complete resources_servers/sav_tracks implementation and tests remain present.
  • This exact Gym head was used by NeMo-RL full-stack vLLM 0.29 video validation. Sync job 7557507 completed 96 consecutive optimizer/refit/wake cycles and exited cleanly at its planned wall-time boundary with Slurm exit 0:0: https://wandb.ai/adlr/Nemotron-omni-RL-debug/runs/d2f77jk1
  • Async job 7560433 completed five consecutive 128-group / 2,048-sequence optimizer-and-refit cycles with mixed CapRL and SA-V rewards, clearing the prior update-2 allocator OOM: https://wandb.ai/adlr/Nemotron-omni-RL-debug/runs/o2ew7b6e

@copy-pr-bot

copy-pr-bot Bot commented Sep 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@ehosseiniasl ehosseiniasl mentioned this pull request Sep 17, 2026
Closed
@ehosseiniasl
ehosseiniasl force-pushed the ehsan/sav-tracks-video-verifier-final branch from 6c84245 to bc0b3a9 Compare September 17, 2026 22:11
@yaoyu-33 yaoyu-33 added area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers complexity:medium Single-domain change with interacting parts or a moderate review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer labels Sep 17, 2026
This was referenced Sep 17, 2026
Merged
Closed
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Sep 18, 2026
@github-actions github-actions Bot mentioned this pull request Sep 18, 2026
Open
@ehosseiniasl
ehosseiniasl force-pushed the ehsan/sav-tracks-video-verifier-final branch 2 times, most recently from e63dadc to 25e2c50 Compare September 28, 2026 20:48
@ehosseiniasl ehosseiniasl mentioned this pull request Sep 29, 2026
Draft
@ehosseiniasl
ehosseiniasl force-pushed the ehsan/sav-tracks-video-verifier-final branch from 25e2c50 to 91e4535 Compare October 1, 2026 00:37
ehosseiniasl and others added 2 commits September 30, 2026 22:38
@ehosseiniasl Pulkit Kumar
feat(video): add SA-V tracks verifier
9e0e81e 
Add the SA-V tracks-schema verifier used by combined SA-V and CapRL video GRPO training, including point, box, and text-referred tracking rewards, config, example data, documentation, and focused tests.

Co-authored-by: Pulkit Kumar <pulkitk@nvidia.com>
Signed-off-by: ehosseiniasl <ehsan.hosseiniasl@gmail.com>
@ehosseiniasl
fix(video): declare SA-V environment servers
a78d805 
Signed-off-by: ehosseiniasl <ehsan.hosseiniasl@gmail.com>
@ehosseiniasl
ehosseiniasl force-pushed the ehsan/sav-tracks-video-verifier-final branch from e33ddd8 to a78d805 Compare October 1, 2026 05:42
@ehosseiniasl ehosseiniasl mentioned this pull request Oct 4, 2026
Merged
khazic pushed a commit to khazic/Gym-lao that referenced this pull request Oct 15, 2026
@ehosseiniasl Pulkit Kumar
feat(video): add the SA-V tracking verifier (NVIDIA-NeMo#3999)
d9f17a2 
## Summary
- Add the SA-V tracking verifier and its server configuration.
- Declare the SA-V environment servers used by video RL.
- Add portable five-row example data, schema, metrics, and checked
rollout artifacts.

## Why
This is the Gym-side verifier foundation for mixed SA-V tracking and
CapRL video training in NeMo-RL. No separate issue is needed; this is
part of the Super 3.5 video RL upstreaming stack.

## Validation
- Runtime evidence: the [healthy 16-node async `r18` W&B
run](https://wandb.ai/adlr/Nemotron-omni-RL-debug/runs/oci-hsg-cs-001_n3post_main_1d76d243b_vllm029_super35_video_step120_async16n_trainmb45056_expandable_uncapped_r18_20261003)
shows repeated GRPO cycles, stable TMPE, and increasing reward on the
equivalent video implementation.
- `gym dataset collate ... +mode=example_validation`: 5 examples;
generated `data/example_metrics.json`.
- Focused validation: 25 verifier tests and all 5 example rows against
`task_data.py` passed.
- Ruff format and lint passed for the changed Python files.
- Both Gym stack ranges pass `git diff --check`.

## Rollouts
`data/example_rollouts.jsonl` contains one checked rollout for each of
the five example rows. Representative task checks:

- `pt`:
`<answer>{"tracks":[{"point":1,"frame":1,"xy":[500,500],"visible":true}]}</answer>`
— `format_ok=true`, reward `1.0`, visible-frame mask membership `true`.
- `box`:
`<answer>{"tracks":[{"box":1,"frame":1,"bbox":[250,250,750,750],"visible":true}]}</answer>`
— `format_ok=true`, reward `1.0`, frame IoU `1.0`.
- `ref`:
`<answer>{"tracks":[{"id":7,"frame":1,"xy":[500,500]}]}</answer>` —
`format_ok=true`, reward `1.0`, predicted id `7` assigned to the GT
object and mask membership `true`.

The artifact also contains visible/invisible `pt` coverage and an
all-absent `ref` negative case, preserving the verifier's current
unmatched-ID behavior.

## Compatibility
Adds a new verifier/configuration and does not change unrelated
environments. Example media is embedded as a small inline JPEG, so no
employee or cluster-local paths are required.

## Stack
Gym PR 1 of 2. Consolidates and supersedes the implementation scope of
NVIDIA-NeMo#3504 after review history is preserved.

## Consolidated stack links

### Gym
1. [Gym NVIDIA-NeMo#3999 — SA-V tracking
verifier](NVIDIA-NeMo#3999)
2. [Gym NVIDIA-NeMo#4000 — R3 routing replay and rollout
identity](NVIDIA-NeMo#4000)

### NeMo-RL
1. [NeMo-RL NVIDIA-NeMo#4472 — Super video data and model
loading](NVIDIA-NeMo/RL#4472)
2. [NeMo-RL NVIDIA-NeMo#4473 — RL/MOPD video frame
contract](NVIDIA-NeMo/RL#4473)
3. [NeMo-RL NVIDIA-NeMo#4474 — sync/async Gym routing
runtime](NVIDIA-NeMo/RL#4474)
4. [NeMo-RL NVIDIA-NeMo#4475 — SA-V pin and CI
fixtures](NVIDIA-NeMo/RL#4475)
5. [NeMo-RL NVIDIA-NeMo#4476 — final dependency
pins](NVIDIA-NeMo/RL#4476)

### Child dependency refs
- [MCore integration
ref](https://github.com/ehosseiniasl/Megatron-LM/tree/ehsan/video-r3-mtp-gradient-integration-20261004),
adapting [MCore #7278](NVIDIA/Megatron-LM#7278)
and [MCore #7279](NVIDIA/Megatron-LM#7279) to
the pinned base.
- [MBridge integration
ref](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/ehsan/video-r3-mcore-pin-integration-20261004).

---------

Signed-off-by: ehosseiniasl <ehsan.hosseiniasl@gmail.com>
Co-authored-by: Pulkit Kumar <pulkitk@nvidia.com>
This was referenced Oct 22, 2026
Open
Open

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers complexity:medium Single-domain change with interacting parts or a moderate review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer sla:triage-overdue Review assignment is over the one-business-day SLA

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ehosseiniasl @yaoyu-33