feat: Switchyard OpenCode integration by drifold · Pull Request #2080 · NVIDIA-NeMo/Gym

drifold · 2026-07-20T12:50:23Z

Stacked on #2026 (base: feature/switchyard-openhands-integration). Adds upstream opencode as a second harness for the Switchyard token-capture integration — same zero-fork principle as the OpenHands path — plus per-run Switchyard instance spawning for token injection.

What

Runs opencode headless (opencode run) with an opencode.json provider pointed at Switchyard — no fork, no harness modification.
Captures the session tree through opencode's own correlation headers: a distinct X-Session-Id per (sub)agent session, x-parent-session-id linking subagents to the root.
Reconstructs the root session as the main rollout and subagent sessions as token-annotated subagent_trajectories.
Per-run Switchyard instances: switchyard_spawn_routing_profile spawns one dedicated token-capture Switchyard instance per agent run on a free port — stateful token injection (feat: token-level capture for RL training (Switchyard ↔ NeMo Gym) Switchyard#63) requires all of a session's calls to reach one process — and reaps it after trace retrieval in run().

Zero-fork architecture

The opencode.json provider id is deliberately not opencode-prefixed — opencode only emits its native correlation headers for non-opencode providers; those headers are the whole capture contract.
Gym mints no session ids here: opencode's own ids om Switchyard's GET /v1/sessions (no harness logs
read).
Title-generator side-calls (same session id, different conversation) are filtered before reconstruction.

Activation and safety

swe_agents:
  responses_api_agents:
    swe_agents:
      agent_framework: opencode
      opencode_source: opencode        # now the default; use "nv-opencode" for the fork
      switchyard_base_url: http://<host>:<port>
      # OR spawn mode (mutually exclusive with switchyard_base_url)
      # switchyard_spawn_routing_profile: /abs/path/

condition	behavior
`opencode_source: nv-opencode`	fork path (bench entrypoint)
upstream (`opencode_source: opencode`, now default+ subagents reconstruct
retrieval/reconstruction failure	rollout kept, `switchyard_trace_error` set
`switchyard_spawn_routing_profile` absent	behavichyard_base_url` or no Switchyard)
spawn mode enabled	per-run instance, records under `persistent_dir`, instance reaped after retrieval

Changes

responses_api_agents/swe_agents/app.py — opencovider JSON), the opencode run command, _list_switchyard_sessions/_reconstruct_sessionssk_id steering via system-prompt instructions (works
around an opencode quirk where it invents task_ids).
responses_api_agents/swe_agents/app.py — switchyard_spawn_routing_profile / switchyard_spawn_host config (mutually exclusive
with switchyard_base_url, validated fail-fast), specycle, agent-script build deferred until theinstance URL is known.
responses_api_agents/swe_agents/app.py — openco to opencode` (upstream); config comment/docs updated
accordingly.
responses_api_agents/swe_agents/configs/ — splitnch_opencode.yaml/swebench_opencode_training.yaml now target **upstream** opencode (opencode_source: gs move to swebench_nv_opencode.yaml /
swebench_nv_opencode_training.yaml (`opencode_soured). Root README table regenerated.
nemo_gym/switchyard_trace.py — canonicalize toolstory-extension comparison only (opencodere-serializes compact vs. pretty-printed JSON across turns); reconstructed items keep the model's original arguments.
Tests: 8 new test classes across test_app.py and`, plus 7 spawn-lifecycle tests (config conflicts,session minting + script rebuild, spawn/teardown with mocked subprocess, readiness-failure reaping).

Dependencies

Requires the Switchyard side on NVIDIA-NeMo/Switchya

7b33c2c (native X-Session-Id capture), be0b2c5r the opencode capture path.
7f20b2a, 940b767, 8307fd9 (token-continuity ion use-case; spawn mode itself works withcapture-only).

Testing

232 passed across the swe_agents and `switchyard
Cluster E2E (vLLM 0.24.0, Qwen3): root + subagent ode headers, both reconstruct with exact token
fidelity, 0% masked; with token injection enabled, buous across multi-turn reasoning + tool-calling
conversations (0 fallbacks), and an OpenHands-pattertiguity verification.

Changes from the live description are only: the openthe first two behavior-table rows (fork now keyed onnv-opencode; upstream noted as default), and two addault flip + config split). Everything else — including the - bullets, headings, dependency SHAs, and the vLs unchanged.

Adds opencode_source='opencode': runs upstream opencode headless with an opencode.json provider pointed at Switchyard (no fork), captures the session tree via opencode's native correlation headers (X-Session-Id / x-parent-session-id), and reconstructs root and subagent rollouts from the retrieved sessions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: drifold <drifold@nvidia.com>

…mparison OpenCode re-serializes tool-call arguments differently across turns (compact vs. pretty-printed JSON), which failed the message-history extension check even though the tokens were contiguous. Canonicalize arguments for the comparison only; reconstructed training items keep the model's original arguments. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: drifold <drifold@nvidia.com>

…system-prompt instructions Signed-off-by: drifold <drifold@nvidia.com>

copy-pr-bot · 2026-07-20T12:50:27Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Adds switchyard_spawn_routing_profile: when set, each agent run spawns one dedicated token-capture Switchyard instance on a free port (stateful token injection requires all of a session's calls to reach one process), scopes records under the run's persistent_dir, and reaps the instance after trace retrieval in run(). Mutually exclusive with switchyard_base_url; behavior is unchanged when absent. The agent script is built after the instance URL is known, since it embeds the Switchyard routing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: drifold <drifold@nvidia.com>

…nfigs as nv-opencode Signed-off-by: drifold <drifold@nvidia.com>

…ed context length Fetch the served model's max_model_len from Switchyard's /v1/models (surfaced there for token-capture routes) before building the opencode agent command, and declare limit.context/limit.output on the generated provider entry. Without limits, opencode requests a flat 32000 max_tokens and never compacts (limit.context defaults to 0); with them it budgets like any registry-known model. Output is reserved as min(32000, context // 4) so the compaction threshold (context - output) keeps most of the window for history. Fetch failure degrades to today's limit-less behavior with a warning. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: drifold <drifold@nvidia.com>

…ends Distribute per-rollout Switchyard spawns across all vLLM generation servers using a round-robin counter (_switchyard_spawn_count % N). Previously all spawns used policy_base_url[0], leaving servers 1-15 idle while server 0 was saturated, causing 3600s timeouts for all concurrent rollouts. Signed-off-by: drifold <drifold@nvidia.com>

Three fixes needed when running 1000+ concurrent rollouts on a single node: 1. Attribute fix: _spawn_switchyard used self.config.ng_global_config_dict_str (SWEBenchWrapperConfig, request-level) instead of params.ng_global_config_dict_str (SWEBenchWrapperInstanceConfig, which carries the server-level global config). This caused AttributeError -> 500 on every /run call. 2. SWITCHYARD_POLICY_MODEL: setting SWITCHYARD_VLLM_BASE_URL via spawn_env caused the wrapper to skip its NEMO_GYM_CONFIG_DICT lookup entirely, leaving SWITCHYARD_POLICY_MODEL unset. Fix: wrapper now derives the model name independently, regardless of whether the URL was pre-set by Gym. 3. Concurrency: 1024 simultaneous Switchyard spawns on one node caused ClientOSError (port exhaustion) and ServerDisconnectedError (uvicorn connection saturation, each /run held open ~168s during spawn+wait). Fix: asyncio.Semaphore(64) limits concurrent spawns; TCP socket probe replaces HTTP /v1/models polling (pure kernel-level, no aiohttp overhead). Signed-off-by: drifold <drifold@nvidia.com>

… of discarding all of it reconstruct_switchyard_rollout validates each captured record as it rebuilds a rollout, and raised on the first record that failed — discarding every record already validated. A discarded session yields a rollout with no generation tokens, which makes NeMo-RL reject and retry the sample's entire 16-rollout prompt group. Breaks are not rare: context compaction legitimately stops later records from extending earlier ones (~4% of sessions, measured over 300 production traces), and at that rate group-level rejection destroyed 97% of groups — a 4-hour run produced zero training steps. Now a mid-session failure returns the already-validated prefix — a complete, token-exact rollout up to the break — with `partial_reason` set. Callers mask the sample: real tokens in the batch, zero loss contribution, and the other 15 samples train normally. This was always the intended semantics ("emit what is trustworthy, mask the sample" — the compaction degrade path already worked this way); raise-and-discard was the inconsistency. Deliberate boundaries: a failure on the FIRST record still raises (nothing is certified, so there is nothing honest to return), and allow_partial=False gives fail-fast for analysis and the validation unit tests. Partial is the default because the training pipeline is the primary caller. Signed-off-by: drifold <drifold@nvidia.com>

…n the rollout's own node; drop shared mode Distribution. Previously the Gym server node hosted every per-rollout Switchyard (~1024 x 94MB at recipe concurrency) and later every trace retrieval — hundreds of Lustre record reads plus CPU-bound token reconstruction per rollout, executed on the server's single event loop. At scale that starved response delivery outright: 2,600 rollouts finished server-side while ~200 responses reached the trainer in 2h and the replay buffer never left zero. Both now live in the rollout's Ray task (scheduling_strategy=SPREAD): the proxy spawns on the agent's node, and after teardown — when the record set is complete by construction — the task reads and reconstructs its own trace and returns it in the task result. The server keeps only in-memory work: applying the trace to the response, masking, and the round-robin backend cursor (carried to the task via switchyard_backend_url). Crossing the server->Ray-worker boundary surfaced three environment assumptions, each fixed here and each caught at full scale first: - the gym venv's bin/ is not on a Ray worker's PATH: resolve the switchyard CLI next to the running interpreter; - NEMO_GYM_CONFIG_DICT does not inherit: set it at the task entry point (the venv's switchyard wrapper derives SWITCHYARD_POLICY_MODEL from it, and get_global_config_dict() otherwise falls through to parsing Hydra CLI args); - proxy readiness now waits 240s, not 60: under Lustre load the CLI's imports alone stall past 60s (measured p90=46s with the distribution truncated at the old timeout), and every timeout kill triggered a 16-rollout group retry — an amplification loop. Dead-on-arrival proxies still fail fast via the process-exit check. Retrieval consumes the longest-valid-prefix semantics from nemo_gym: partial or degraded traces (context compaction) emit real tokens plus mask_sample=True, so a sample excludes itself from the loss without aborting its prompt group. Shared mode (an externally managed proxy at switchyard_base_url, records fetched over HTTP) is removed: a dead proxy strands its tokens behind an unreachable endpoint — retried without bound — which is the failure mode disk retrieval exists to eliminate, and per-rollout spawn is the only configuration injection supports anyway. The instance-level field survives as switchyard_spawned_base_url, internal plumbing from the task's spawn to the harness command builder. Signed-off-by: drifold <drifold@nvidia.com>

Runs the per-rollout Switchyard lifecycle inside a REAL Ray task with the production decorator options — deliberately unmocked. The existing spawn tests mock create_subprocess_exec, which verifies argv but cannot fail on what broke at scale three times: state the Gym server process has that a Ray worker does not (PATH, environment, sys.path). One task round-trip covers CLI resolution without PATH (with a negative control that reports the test inconclusive rather than vacuously green when the worker has the CLI on PATH), proxy readiness (bound + accepting), params crossing the boundary exactly as production sends them (model_dump -> model_validate), task-side trace reconstruction from seeded records, payload survival through the Ray object store, and clean teardown. Skips gracefully where ray or the CLI is unavailable. ~33s wall when fully exercised; reviewers may prefer an integration marker for per-commit CI. Signed-off-by: drifold <drifold@nvidia.com>

…ports its env The spawned proxy's config previously lived outside the repo: a routing profile with hardcoded parsers, plus a wrapper script installed over the venv's switchyard entrypoint to derive the policy endpoint/model and put the serving vLLM on PYTHONPATH. Rebuilding the venv silently deleted the wrapper and broke training. - switchyard_profile.yaml: shipped, deployment-independent profile. The endpoint, model, and parsers resolve from ${...} env vars; sampling params stay numeric literals (env interpolation is string-only). The default/policy_model routes are aliases of one route: dispatch is an exact match on the request's model id, and upstream opencode POSTs "default" while the OpenHands fork POSTs the model server's name. - _spawn_switchyard_local now exports everything the profile references: endpoint/model derived from the run's config, parsers and PYTHONPATH from three new config fields (switchyard_tool_parser, switchyard_reasoning_parser, switchyard_parser_pythonpath). Signed-off-by: drifold <drifold@nvidia.com>

Serves Gym model calls through a Switchyard routing proxy, so any benchmark Gym already supports can be evaluated against a router without touching harness code: gym eval run --benchmark <name> --model-type switchyard_model Switchyard is integrated at the model layer rather than the agent layer. The existing Switchyard work in Gym (#2026 token capture, #2080 OpenCode) wires one harness at a time under responses_api_agents/swe_agents; putting the router behind a Responses API model instead makes it available to every benchmark at once, which is what evaluating the router itself requires. Two modes. Attach (default) points at a proxy run outside Gym, which keeps proxy lifecycle and image pinning with the eval harness so a run can be compared against a scaled-evals run of the same Switchyard commit. Launch starts the proxy from a routing profile and shuts it down at exit, for a single-command run. Gym's rollout-attempt id is forwarded as Switchyard's opaque session id, so proxy-side routing decisions and costs join back to the rollout that produced them. The correlation prefix is stripped by Gym's capture middleware before handlers run, so a thin outer middleware republishes the id for the request. The nemo-switchyard dependency is scoped to this package rather than Gym's core dependencies: only runs that route through Switchyard pay for it, and attach mode needs no dependency at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>

…ion provenance (#2141) Part of #2088. This PR now also absorbs #2600 (routing-condition provenance) so the whole feature reviews in one place. Adds `responses_api_models/switchyard_model/`, a model server that routes Gym model calls through a Switchyard proxy, so any benchmark Gym supports can run against a router without harness changes — unlike the per-harness integrations in #2026/#2080, which are right for RL trace capture but not for evaluating the router itself. Gym either hosts the router in-process from a native Switchyard TOML deployment (`deployment=routes.toml`; a bad deployment fails at startup, nothing outlives the process, and multi-worker configs are rejected since each worker would host its own proxy) or attaches to an externally run proxy via `switchyard_base_url`, which does support multiple workers. A caller-supplied `model` is always overridden with the configured route so an agent can't bypass the router, and Gym forwards its rollout-attempt id as the Switchyard session id (`x-switchyard-session-id`) so routing decisions and costs join back to the rollout that produced them. That id only arrives when model-call capture or token capture is enabled, so the server validates the requirement at startup: a warning under the default config, a refusal to start when forwarding was requested explicitly but no capture is on. From #2600: set `condition_dir` and the server records the routing condition the run served under — `switchyard-condition.json` at startup (route, mode, proxy URL, deployment SHA-256 plus an archive with inline `api_key` and all `extra_headers` values redacted, and the proxy's identity: the local wheel version when hosting, a caller-supplied `proxy_provenance` mapping when attaching) and `switchyard-stats.json` at shutdown (per-target requests, errors, tokens, and latency plus classifier-side usage from `/v1/stats`, fetched through Gym's shared async client as the configured caller and hard-capped inside the shutdown window). The snapshot labels its scope, since a hosted proxy's counters cover exactly the run while an attached proxy's aggregate everything it has served. Both writes are best-effort and cannot take down serving. This makes two routed runs comparable and a hosted run reproducible from the archived deployment; the docs page gains a "Compare routing conditions" workflow that aligns rollout rows by task and rollout index, reports rollouts missing a counterpart, and includes the classifier tokens the selected model's usage omits. The only dependency is `nemo-switchyard==0.2.0`, pinned exactly in this server's own pyproject.toml (Gym core never imports Switchyard); 0.2.0 is the floor because it is the first release whose Responses translation passes `NeMoGymResponse` validation (NVIDIA-NeMo/Switchyard#144), and the ceiling because upstream has already removed the Python serve stack this PR originally drove. 47 tests pass with 100% coverage on app.py, including an integration suite that drives the real 0.2.0 wheel with no mocks (chat/responses round trips, route override, stats, provenance files written across a real shutdown), and a live routed rollout ran against this revision with reward 1.0 (results in [this comment](#2141 (comment))). Two reviewer notes. `ModelCallRecord.model_ref` names the Gym model server, not the routed target, so per-call attribution must be read from captured payloads — promoting it to a first-class record field is the follow-up flagged for #2088 and needs a core-schema design pass. And `harbor_agent` overwrites `response["model"]` with the policy model name, so Harbor-run benchmarks report the route name for every call; flagging in case that overwrite is unintentional. --------- Signed-off-by: Lin Jia <linj@nvidia.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

drifold and others added 3 commits July 20, 2026 15:00

fix(swe_agents): steer upstream opencode to omit invalid task_id via …

20125ee

…system-prompt instructions Signed-off-by: drifold <drifold@nvidia.com>

nemo-automation-bot Bot added the community-request Issue reported or requested by someone from the community label Jul 20, 2026

github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Jul 21, 2026

github-actions Bot mentioned this pull request Jul 21, 2026

[Tracker] Pull request handoff SLA #2067

Open

svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Jul 22, 2026

drifold mentioned this pull request Jul 22, 2026

feat: token-level capture for RL training (Switchyard ↔ NeMo Gym) NVIDIA-NeMo/Switchyard#63

Merged

11 tasks done

drifold and others added 2 commits July 23, 2026 09:35

feat(swe_agents): default opencode harness to upstream, split fork co…

a01d40e

…nfigs as nv-opencode Signed-off-by: drifold <drifold@nvidia.com>

linj-glitch mentioned this pull request Jul 25, 2026

feat: add switchyard_model server for router-backed evals with condition provenance #2141

Merged

drifold added 2 commits July 29, 2026 01:23

drifold force-pushed the switchyard-opencode-integration branch from ec12517 to 8afdd31 Compare July 29, 2026 08:23

drifold added 2 commits August 2, 2026 00:45

drifold requested a review from a team as a code owner August 3, 2026 09:11

drifold force-pushed the switchyard-opencode-integration branch from 510bf3f to afd76b8 Compare August 3, 2026 09:30

This was referenced Sep 8, 2026

[Tracker] Pull request handoff SLA #3158

Open

[Tracker] Pull request handoff SLA #3402

Open

svcnvidia-nemo-ci added area:agent Agent harnesses and Responses API agent behavior feature New capabilities, enhancements, or enablement work labels Oct 20, 2026

This was referenced Oct 22, 2026

[Tracker] Pull request handoff SLA #4925

Open

[Tracker] Pull request handoff SLA #4926

Open

Navigation Menu

Repository navigation

feat: Switchyard OpenCode integration - #2080

feat: Switchyard OpenCode integration#2080
drifold wants to merge 12 commits into
NVIDIA-NeMo:feature/switchyard-openhands-integrationfrom
drifold:switchyard-opencode-integration

drifold commented Jul 20, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jul 20, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

drifold commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Zero-fork architecture

Activation and safety

Changes

Dependencies

Testing

Uh oh!

copy-pr-bot Bot commented Jul 20, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

drifold commented Jul 20, 2026 •

edited

Loading