Skip to content

epic: routing-aware evals #2088

Closed

Description

@cwing-nvidia

Context

Teams using Switchyard (or similar weak/strong routing) need to systematically measure how routing affects benchmark scores, failures, or cost. Gym can proxy to Switchyard today, but routing policy isn't a first-class eval variable: no comparable runs, no attribution, no guided workflow.

Gym already does this for other knobs (e.g. skills). Routing should work the same way.

Note: These changes may have cross-repo impact: Switchyard + Gym

User outcomes

  • Run the same benchmark under different routing conditions and compare scores + token/cost signals fairly.
  • Identify whether routing helps or hurts on tasks they care about
  • Reproduce routing experiments like any other Gym eval.

Scope Notes

  • Enable routing conditions as experimental variables in Gym evals, with outputs that support comparison and diagnosis.
  • In scope: eval workflow, provenance/comparability, analysis hooks, docs + ecosystem mention, Switchyard alignment.
  • Out of scope: building routers in Gym, replacing Switchyard, RL training paths, switchyard launch interactive workflows, production serving.

Acceptance Criteria

  • Documented workflow comparing ≥2 routing conditions on the same benchmark/harness
  • Routing condition identifiable in eval outputs across runs
  • Docs: when/why Gym + Switchyard
  • One end-to-end example published

Activity

  1. moved this from Dev Todo to Details Todo in NeMo Gym 0.6.0 - Sept 2on Jul 30, 2026
  2. moved this from Details Todo to Dev Todo in NeMo Gym 0.6.0 - Sept 2on Jul 30, 2026
  3. moved this from Dev WIP to Review Todo in NeMo Gym 0.6.0 - Sept 2on Aug 25, 2026
  4. moved this from Review Todo to Dev WIP in NeMo Gym 0.6.0 - Sept 2on Aug 26, 2026
  5. moved this from Dev WIP to QA Todo in NeMo Gym 0.6.0 - Sept 2on Aug 26, 2026
  6. moved this from QA Todo to QA WIP in NeMo Gym 0.6.0 - Sept 2on Aug 26, 2026
  7. phtran8 commented on Aug 26, 2026

    @phtran8

    Hi @cwing-nvidia , please help share SWITCHYARD_API_KEY for QA testing.

  8. moved this from QA WIP to Done in NeMo Gym 0.6.0 - Sept 2on Sep 2, 2026
  9. closed this as completedby moving to Done in NeMo Gym 0.6.0 - Sept 2on Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions