feat(harness): rollout collection + openenv collect CLI by sergiopaniego · Pull Request #560 · huggingface/OpenEnv

sergiopaniego · 2026-04-17T16:10:33Z

Summary

Adds a collect submodule under openenv.core.harness and an openenv collect CLI that turn any deployed OpenEnv environment + any LLMClient into a TRL-ready JSONL dataset pushed to the Hub, using the session-based runtime from #471 as foundation. First env driver: OpenSpielSessionFactory for tic_tac_toe.

Stacked on #471 — base branch is feature/harness-interface. This PR should merge after #471.

Type of Change

(Refactoring is a pure rename harness.py → harness/__init__.py to make room for the submodule; no behaviour change.)

Alignment Checklist

Before submitting, verify:

I have read .claude/docs/PRINCIPLES.md and this PR aligns with our principles
I have checked .claude/docs/INVARIANTS.md and no invariants are violated
I have run /pre-submit-pr (or bash .claude/hooks/lint.sh and tests) and addressed all issues

RFC Status

Not required (bug fix, docs, minor refactoring)
RFC exists: #___
RFC needed (will create before merge)

Test Plan

66 new tests; the existing 1009 pass unchanged.

Reproduce locally:

# Full suite relevant to this PR
PYTHONPATH=src:envs uv run pytest \
    tests/core/test_harness_runtime.py \
    tests/core/test_harness_collect.py \
    tests/envs/test_openspiel_harness.py \
    tests/envs/test_openspiel_collect.py \
    tests/test_cli/test_collect.py -v

# Lint / format
uv run usort check src/openenv/core/harness/ src/openenv/cli/commands/collect.py envs/openspiel_env/harness.py tests/
uv run ruff check src/openenv/core/harness/ src/openenv/cli/commands/collect.py envs/openspiel_env/harness.py tests/
uv run ruff format src/ tests/ --check

End-to-end smoke test (no Docker, no API key) — verifies the pipeline produces real JSONL:

PYTHONPATH=src:envs uv run openenv collect openspiel:tic_tac_toe \
    --base-url <deployed-openspiel-space-with-OPENSPIEL_GAME=tic_tac_toe> \
    --output-dir /tmp/ttt-smoke \
    -n 5 --provider scripted
cat /tmp/ttt-smoke/results.jsonl | head -1 | python -m json.tool

Against a real LLM teacher + push to Hub:

OPENAI_API_KEY=... PYTHONPATH=src:envs uv run openenv collect openspiel:tic_tac_toe \
    --base-url <deployed-openspiel-space> \
    --output-dir /tmp/ttt-sft-v1 \
    -n 20 --provider openai --model gpt-5-mini \
    --push-to-hub <user>/ttt-sft-v1

Validated manually against a live HF Space running openspiel_env with OPENSPIEL_GAME=tic_tac_toe; dataset visible in the HF Dataset Viewer thanks to the auto-generated README front-matter.

Claude Code Review

Automated: lint/format pass on all files added/changed in this PR. check-debug.sh is clean in our src/ scope (pre-existing hits in unrelated files).
Principle conflicts: none identified. "Rewards in environment", client-server separation, Gymnasium signatures, dual API boundary, MCP-as-standard, and one-env-one-trajectory all upheld.
RFC conflicts — 3 flags, none blocking, all documented in "Out of scope":
1. Filter primitive (should_keep) bypasses RFC 004 Rubric abstraction. v1 uses a bare Callable[[EpisodeRecord], bool]; a future rubric= arg would subsume it.
2. Building on experimental surface of RFC 005 / Add harness session runtime and BrowserGym adapter for training and evaluation #471. PR gated on Add harness session runtime and BrowserGym adapter for training and evaluation #471 merging; rebase if the harness API shifts.
3. CLI env dispatch is a hardcoded registry. Adding envs (TextArena, Connect4) requires editing cli/commands/collect.py. Follow-up candidate for an entry-points mechanism.

Preserves the public API (`from openenv.core.harness import X` keeps working) while making room for a ``collect`` submodule that layers synthetic-dataset generation on top of the runtime primitives from RFC 005 / PR huggingface#471. Relative imports in the module are adjusted from ``.client_types`` → ``..client_types``. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Introduces ``openenv.core.harness.collect``, a thin layer on top of the harness runtime from RFC 005 / PR huggingface#471 for generating synthetic datasets from deployed environments: - ``EpisodeRecord`` — serializable view of one rollout + its verification. Uses ``_resolve_env_reward`` so any mismatch between a tool-result reward and ``verify.env_reward`` raises, preserving the "rewards in env" invariant end-to-end. - ``RolloutSerializer`` — append-only JSONL writer with a ``metadata.json`` sidecar. ``collected_episode_ids()`` enables resume. - ``CollectRunner.run()`` — orchestrates N episodes: session.create → harness.run_white_box → verify → optional rubric filter → serialize. Returns a ``CollectResult`` with aggregate stats. - ``build_model_step(llm_client)`` — adapts any ``LLMClient`` (OpenAI, Anthropic, and any OpenAI-compatible endpoint such as vLLM, TGI, Ollama, HF Inference, Together, Groq, Fireworks) into a ``ModelStep`` for the white-box harness. - ``push_to_hf_hub(output_dir, repo_id)`` — uploads ``results.jsonl``, ``metadata.json``, and an auto-generated dataset card to the Hub. The card's YAML front-matter tells the HF Dataset Viewer to treat the JSONL as ``split=train`` instead of trying to merge it with the metadata sidecar. 37 new tests (EpisodeRecord, RolloutSerializer, CollectRunner, build_model_step, push_to_hf_hub, build_dataset_readme). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

``envs/openspiel_env/harness.py`` exposes an ``OpenSpielSessionFactory`` that wraps ``OpenSpielEnv`` (via ``StepEnvSessionAdapter``) and lets a harness drive any OpenSpiel game — tic_tac_toe initially — through a single ``play_move(action_id)`` MCP-style tool. - Initial prompt renders legal actions plus a human-readable board for TTT so the LLM can reason about positions without needing to decode the 27-float info_state tensor. - ``render_tic_tac_toe_board`` decodes the OpenSpiel TTT info_state (empty/X/O planes) into a 3x3 grid. Empty cells show their action_id so the prompt doubles as an action legend. - Follows the pattern established by ``envs/browsergym_env/harness.py`` in PR huggingface#471 — no changes to the underlying env, client, or protocol. Tests cover the board rendering, tool dispatch, reset-kwargs forwarding, and an end-to-end collect run against a scripted client exercising the full ``MCPHarnessAdapter`` → ``CollectRunner`` → JSONL pipeline. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Wraps ``CollectRunner`` + ``OpenSpielSessionFactory`` into a single command so a deployed OpenEnv environment can produce a dataset with one call: openenv collect openspiel:tic_tac_toe \\ --base-url https://user-space.hf.space \\ --output-dir /tmp/ttt-sft-v1 \\ -n 200 --provider openai --model gpt-5-mini \\ --push-to-hub user/ttt-sft-v1 Flags in short: - ``--provider scripted | openai | anthropic`` — teacher selection. Scripted picks the first legal action and requires no API key, making ``openenv collect`` smoke-testable out of the box. - ``--llm-endpoint / --llm-port`` — point at any OpenAI-compatible endpoint (vLLM, TGI, Ollama, HF Inference, Together, Groq, ...). - ``--push-to-hub REPO`` — upload the directory as a dataset after collect; ``--private``/``--commit-message`` available. - ``--resume / --no-resume`` — ``CollectRunner`` skips ``episode_ids`` already serialized on disk. - ``--keep-losses`` — by default filters rollouts with reward < 0 so the output is SFT-ready. Env dispatch is via ``"family:variant"`` strings (e.g. ``openspiel:tic_tac_toe``). Unknown families raise a typer ``BadParameter`` with the supported set. 7 new CLI tests mocking ``CollectRunner`` + ``OpenSpielEnv`` so the dispatch logic (scripted vs hosted provider, push-to-hub, filter defaults, bad ``--env``) is exercised without network. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

- ``src/openenv/core/harness/README.md`` documents the collect module: quick-start (CLI + programmatic), output schema, and design notes covering the "thin envs" / "rewards in env" / "provider-agnostic teacher" choices. - ``examples/ttt_collect_demo.py`` — scripted teacher against either a built-in fake OpenSpiel client or a real deployed server (``--base-url``). Runs with zero setup for pipeline smoke-testing. - ``examples/ttt_collect_with_llm.py`` — provider-agnostic example that picks between hosted OpenAI/Anthropic (via ``--provider``) and any OpenAI-compatible self-hosted endpoint (via ``--llm-endpoint``), using the same collect pipeline unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

greptile-apps · 2026-04-17T16:10:37Z

PR title or description contains an excluded keyword.

Darktex

Note: This is an automated review by Claude Code, not a human review.

Alignment Review Report

Automated Checks

Lint: PASS on PR files (lint failures are pre-existing in envs/carla_env/, not introduced here)
Debug code: CLEAN in PR scope

Tier 1: Fixes Required

src/openenv/core/harness/collect.py (build_model_step): asyncio.run() is called inside a synchronous wrapper. If the caller is already inside a running event loop (async test runner, Jupyter, or an async orchestrator), this raises RuntimeError: This event loop is already running. Use asyncio.get_event_loop().run_until_complete() with a loop-detection guard, or require callers to be sync-context-only and document it explicitly. This is the path that all LLM teacher traffic goes through — it will silently fail in common notebook and async-server contexts.
src/openenv/core/harness/collect.py (CollectResult.episode_ids): The field is populated with planned_ids — the full list of all planned episode IDs including skipped and dropped ones, not just collected ones. The name episode_ids implies "what was collected". Either rename to planned_episode_ids or filter to only collected IDs. Currently result.episode_ids returns stale IDs from prior runs when resume=True, which is surprising.

Tier 2: Alignment Discussion

ALIGNMENT FLAG: HarnessAdapter name collision with RFC 005 WIP

Principle at stake: RFC 005 defines HarnessAdapter as an async ABC for wrapping external harness processes (OpenClaw, Claude Code). This PR introduces a different HarnessAdapter in openenv.core.harness — a synchronous protocol for white-box/black-box rollouts. Both will live under openenv.core once #471 and this PR merge. The names are identical, the semantics are entirely different.
The concern: Imports like from openenv.core.harness import HarnessAdapter will silently shadow whichever one loads second. If RFC 005's implementation lands before this is resolved, the public API surface breaks for one of them. The PR does not acknowledge this conflict.
Suggested reviewer: @Darktex

ALIGNMENT FLAG: CLI env registry is hardcoded, bypasses the self-registration pattern implied by RFC 003

Principle at stake: RFC 003 (MCP as universal standard) and the "be hands-on" principle assume environments are self-describing. The _build_session_factory dispatch in collect.py is a hand-written if env_name != "openspiel" block. Adding TextArena or Connect4 requires editing a core CLI file.
The concern: This is a load-bearing architectural pattern that will get copied for every new env. The right fix (Python entry points or a registry decorator) is not expensive to add now but becomes painful to retrofit. The PR authors flag this as a follow-up with no issue number.
Suggested reviewer: @Darktex

Automated review by Claude Code | Learn more

burtenshaw · 2026-04-18T18:45:48Z

Addressed the three review findings on the PR branch.

--llm-endpoint now forces the LLM teacher path even when --provider is left at its default, and the CLI metadata/output label that case as openai-compatible.
CollectRunner.run() now consumes task slots for skipped episode ids during resume so task stays aligned with episode_id.
build_model_step() now uses run_async_safely(...) instead of raw asyncio.run(...), so the programmatic API works under an active event loop.

Added regression tests for all three cases. Verified locally with:
PYTHONPATH=src:envs uv run pytest tests/core/test_harness_collect.py tests/envs/test_openspiel_collect.py tests/envs/test_openspiel_harness.py tests/test_cli/test_collect.py -q

All 57 targeted tests passed.

burtenshaw · 2026-04-18T18:48:32Z

Merging into the harness branch.

* Add harness session runtime for training and evaluation * Handle MCP tool errors and lazy-load BrowserGym exports * Refine experimental harness APIs and reward forwarding * docs: mention environment_factory in harness tutorial * test: cover harness review regressions * feat(harness): rollout collection + openenv collect CLI (#560) * refactor(harness): move harness.py into harness/ package Preserves the public API (`from openenv.core.harness import X` keeps working) while making room for a ``collect`` submodule that layers synthetic-dataset generation on top of the runtime primitives from RFC 005 / PR #471. Relative imports in the module are adjusted from ``.client_types`` → ``..client_types``. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(harness): add rollout collection module Introduces ``openenv.core.harness.collect``, a thin layer on top of the harness runtime from RFC 005 / PR #471 for generating synthetic datasets from deployed environments: - ``EpisodeRecord`` — serializable view of one rollout + its verification. Uses ``_resolve_env_reward`` so any mismatch between a tool-result reward and ``verify.env_reward`` raises, preserving the "rewards in env" invariant end-to-end. - ``RolloutSerializer`` — append-only JSONL writer with a ``metadata.json`` sidecar. ``collected_episode_ids()`` enables resume. - ``CollectRunner.run()`` — orchestrates N episodes: session.create → harness.run_white_box → verify → optional rubric filter → serialize. Returns a ``CollectResult`` with aggregate stats. - ``build_model_step(llm_client)`` — adapts any ``LLMClient`` (OpenAI, Anthropic, and any OpenAI-compatible endpoint such as vLLM, TGI, Ollama, HF Inference, Together, Groq, Fireworks) into a ``ModelStep`` for the white-box harness. - ``push_to_hf_hub(output_dir, repo_id)`` — uploads ``results.jsonl``, ``metadata.json``, and an auto-generated dataset card to the Hub. The card's YAML front-matter tells the HF Dataset Viewer to treat the JSONL as ``split=train`` instead of trying to merge it with the metadata sidecar. 37 new tests (EpisodeRecord, RolloutSerializer, CollectRunner, build_model_step, push_to_hf_hub, build_dataset_readme). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(envs/openspiel): add TTT harness session factory ``envs/openspiel_env/harness.py`` exposes an ``OpenSpielSessionFactory`` that wraps ``OpenSpielEnv`` (via ``StepEnvSessionAdapter``) and lets a harness drive any OpenSpiel game — tic_tac_toe initially — through a single ``play_move(action_id)`` MCP-style tool. - Initial prompt renders legal actions plus a human-readable board for TTT so the LLM can reason about positions without needing to decode the 27-float info_state tensor. - ``render_tic_tac_toe_board`` decodes the OpenSpiel TTT info_state (empty/X/O planes) into a 3x3 grid. Empty cells show their action_id so the prompt doubles as an action legend. - Follows the pattern established by ``envs/browsergym_env/harness.py`` in PR #471 — no changes to the underlying env, client, or protocol. Tests cover the board rendering, tool dispatch, reset-kwargs forwarding, and an end-to-end collect run against a scripted client exercising the full ``MCPHarnessAdapter`` → ``CollectRunner`` → JSONL pipeline. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): add openenv collect command Wraps ``CollectRunner`` + ``OpenSpielSessionFactory`` into a single command so a deployed OpenEnv environment can produce a dataset with one call: openenv collect openspiel:tic_tac_toe \\ --base-url https://user-space.hf.space \\ --output-dir /tmp/ttt-sft-v1 \\ -n 200 --provider openai --model gpt-5-mini \\ --push-to-hub user/ttt-sft-v1 Flags in short: - ``--provider scripted | openai | anthropic`` — teacher selection. Scripted picks the first legal action and requires no API key, making ``openenv collect`` smoke-testable out of the box. - ``--llm-endpoint / --llm-port`` — point at any OpenAI-compatible endpoint (vLLM, TGI, Ollama, HF Inference, Together, Groq, ...). - ``--push-to-hub REPO`` — upload the directory as a dataset after collect; ``--private``/``--commit-message`` available. - ``--resume / --no-resume`` — ``CollectRunner`` skips ``episode_ids`` already serialized on disk. - ``--keep-losses`` — by default filters rollouts with reward < 0 so the output is SFT-ready. Env dispatch is via ``"family:variant"`` strings (e.g. ``openspiel:tic_tac_toe``). Unknown families raise a typer ``BadParameter`` with the supported set. 7 new CLI tests mocking ``CollectRunner`` + ``OpenSpielEnv`` so the dispatch logic (scripted vs hosted provider, push-to-hub, filter defaults, bad ``--env``) is exercised without network. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(harness): add README and runnable examples - ``src/openenv/core/harness/README.md`` documents the collect module: quick-start (CLI + programmatic), output schema, and design notes covering the "thin envs" / "rewards in env" / "provider-agnostic teacher" choices. - ``examples/ttt_collect_demo.py`` — scripted teacher against either a built-in fake OpenSpiel client or a real deployed server (``--base-url``). Runs with zero setup for pipeline smoke-testing. - ``examples/ttt_collect_with_llm.py`` — provider-agnostic example that picks between hosted OpenAI/Anthropic (via ``--provider``) and any OpenAI-compatible self-hosted endpoint (via ``--llm-endpoint``), using the same collect pipeline unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: address collect feedback --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: burtenshaw <ben.burtenshaw@gmail.com> * fix: address browsergym harness review comments * fix: format collect files * envs/opencode_env: OpenCode coding-agent harness primitive (#603) * feat(opencode_env): OpenCode harness primitive (M1.1-M1.5) Harness primitive for running the OpenCode CLI agent in a sandbox against any OpenAI-compatible endpoint. Stacked on the PR #471 harness-runtime branch; implements the ResourceSession / ResourceSessionFactory contracts. Modes: - black_box: opencode talks directly to the configured endpoint. Verified end-to-end against real OpenAI (gpt-4o-mini) inside an E2B sandbox; the agent produces a correct fizzbuzz and the verifier scores reward=1.0. - transparent_proxy: a per-session FastAPI proxy runs inside the sandbox, forwards /v1/chat/completions to the configured upstream with logprobs=true injected, captures per-turn (messages, completion tokens, logprobs, finish reason) to a JSON-lines trace, and strips logprobs from what opencode sees. Handles both unary and streaming (SSE). Caps max_tokens and auto-translates to max_completion_tokens for gpt-5.x/o* models. Components: - config.py: OpenCodeConfig (generic provider/base_url/api_key/model fields supporting OpenAI, Anthropic, and OpenAI-compatible endpoints; proxy tuning knobs; sandbox_home override for non-E2B backends). - opencode_runtime.py: pure builders for opencode.json, install/run shell commands, and env vars. - task.py: OpenCodeTask pydantic model (instruction + optional setup_shell + file uploads + opaque metadata); coerces from str/dict. - sandbox/base.py: SandboxBackend / SandboxHandle / BgJob Protocols. - sandbox/e2b.py: E2BSandboxBackend with a threaded BgJob wrapper that provides timeout support over E2B's CommandHandle. - harness.py: OpenCodeSession + OpenCodeSessionFactory; in Mode B, installs proxy deps, uploads the proxy module, starts it as a bg job on localhost:7000, rewrites opencode.json to point at the proxy, and forces @ai-sdk/openai-compatible so routing goes through /v1/chat/completions. - interception.py: InterceptionProxy (FastAPI, unary + streaming), per-turn trace capture, CLI entry point for sandbox-side execution. Tests: 37 unit tests plus 3 live integration tests gated on E2B_API_KEY / OPENAI_API_KEY. Known limitation: OpenAI's gpt-5.x chat family refuses logprob requests, so Mode B live validation against OpenAI requires gpt-4o-mini or older. vLLM (the intended training-time upstream) returns logprobs natively. * feat(opencode_env): live vLLM validation + post-rollout summary End-to-end Mode B now verified against a live vLLM tunnel: - Qwen/Qwen3.5-4B on 2x A100 (tp=2, 16K ctx) via `vllm serve` - Cloudflared tunnel exposes the endpoint publicly - E2B sandbox runs opencode against the tunnel through the in-sandbox proxy - Proxy captures 4 turns, 36 tokens with real per-token logprobs - Agent produces correct fizzbuzz.py, verifier scores reward=1.0 - 55.86s total (sandbox + install + 4 LLM turns + verify) Changes: - config: add proxy_disable_thinking flag; plumbed through harness -> proxy via --disable-thinking CLI arg. - interception: inject chat_template_kwargs.enable_thinking=false on forwarded requests when enabled (Qwen3/Qwen3.5 tokenizer hook); also split the request handler into unary + streaming paths (SSE) and auto-translate max_tokens -> max_completion_tokens for gpt-5.x/o* models. - sandbox/e2b: E2B's commands.run(background=True) has a default server-side timeout=60 that kills long-running opencode bg jobs; pass timeout=0 to disable it. - live_watch: RolloutSummary / collect_rollout_summary / print_rollout_summary — post-rollout structured report reading proxy trace, opencode event log, and the workdir listing + file contents. - tests/test_harness_live_vllm: end-to-end Mode B test against a live vLLM tunnel (gated on VLLM_TUNNEL_URL + E2B_API_KEY), asserts logprobs are captured with shape matching completion tokens. * docs(opencode_env): README with Mode A + Mode B quickstarts and full config * feat(opencode_env): retries, error surfacing, model override Fixes the server-mode rollout path. Under load E2B can return sandboxes that aren't yet accepting commands, curl-install can transiently fail, and opencode's internal title-generation call can emit a stripped model id (e.g. ``Qwen3.5-4B`` instead of ``Qwen/Qwen3.5-4B``) that vLLM rejects with 404. Previously the proxy silently swallowed upstream error bodies and returned an empty event-stream, which opencode interpreted as an empty assistant turn. Changes: - harness: ``_wait_for_sandbox_ready`` probes ``echo ok`` up to 15x before issuing commands. - harness: ``_exec_with_retry`` wraps install / pip-deps / extra-setup with exponential backoff, up to 3 attempts, bailing on deterministic errors (non-empty stderr). - interception: ``ProxyConfig.model_override`` rewrites the ``model`` field on every forwarded request to the exact upstream id, bypassing opencode's provider-prefix quirks. Plumbed through as ``--model-override`` on the CLI. - interception: ``_proxy_streaming`` now inspects upstream status before committing to an SSE response — non-2xx returns a JSON error response to opencode AND logs the full upstream body to proxy.log, so the caller sees the real failure reason. - harness: ``_start_proxy`` passes ``--model-override`` built from ``config.model`` so the upstream always sees the right id. - harness: proxy deps install now uses ``_exec_with_retry`` too. Verified via local uvicorn: fizzbuzz + fibonacci tasks both succeed end-to-end through the server path in 19-20s with reward=1.0, 3-4 productive turns, and real per-token logprobs captured on every turn. * feat(opencode_env): add serve driver + opencode_client (Phase 2b primitive) Adds the infrastructure for driving opencode via its HTTP server instead of the CLI. This is the Phase 2b foundation — fine-grained MCP tools in the consumer server wrap these primitives. Changes: - opencode_client.OpenCodeServerClient: typed httpx wrapper over the OpenAPI spec at /doc. Sync and async methods for create_session, send_message / send_prompt_async, list_messages, get_session, get_all_status, abort, plus stream_events / astream_events (SSE) and a wait_for_ready helper. Base64 basic-auth when OPENCODE_SERVER_PASSWORD is set. - harness.OpenCodeSession: new ``driver: Literal["cli", "serve"]`` field. driver="cli" is today's `opencode run` path. driver="serve" stores serve_public_url + serve_client + serve_session_id on the session. start_agent() dispatches on driver; wait_for_completion() polls /session/:id for idle when driver="serve". New abort() method hits /session/:id/abort for cancellation. - harness.OpenCodeSessionFactory: new ``driver`` constructor arg plumbs through to create(). _start_serve() runs opencode serve bound to 0.0.0.0:4096 as a bg job, probes it internally via curl, then uses the sandbox backend's get_host(4096) to build a public URL (E2B returns https://4096-<sandbox_id>.e2b.app). Fails fast if the backend doesn't support get_host. - tests/test_opencode_client.py: 7 unit tests covering URL/method/body shape, auth header, prompt text extraction, abort bool, limit param, wait_for_ready polling, SSE event helpers. Uses httpx MockTransport patched via monkeypatch — no live opencode serve needed. - tests/test_harness.py: _FakeSandbox now responds to the health-probe "echo ok" command so the existing factory tests work after the Phase-1 reliability layer landed. Verified: 34 unit tests pass (7 new + 27 existing), driver=cli path unchanged. End-to-end E2B spike confirms: sandbox_id assigned in 0.4s opencode install 2.5s opencode serve --port 4096 --hostname 0.0.0.0 listening in 1s sandbox.get_host(4096) returns https://4096-<id>.e2b.app external /doc returns HTTP 200 with OpenAPI spec external POST /session returns real session metadata Next: wire 4 new MCP tools (start_rollout / get_state / abort_rollout / finalize_rollout) in the consumer env + SSE endpoint for live rollout events. Ship to HF Space. * fix(opencode_env): bump proxy-start wait from 10s to 60s Uvicorn+fastapi cold boot inside E2B can take >10s under load; the tight probe loop was producing false 'proxy did not start within 10s' errors. 60s cap at 0.5s intervals keeps the retry fast while tolerating the slow path. * fix(opencode_env): cwd into workdir, fix idle detection, preserve reasoning Three audit fixes surfaced by end-to-end testing through the deployed env server: 1. `_start_serve` now `cd`s into workdir_path(config) before launching opencode serve. Without this, the agent writes files to $HOME and RolloutResult.workdir_files (reading /home/user/workdir) comes back empty — the "rollout succeeded but nothing appeared" symptom. 2. `wait_for_completion` idle check was `status.get("idle")` but opencode's /session/status returns `{"type":"idle"}`, not `{"idle":true}`. Every serve-driver rollout silently timed out at agent_timeout_s. Now checks `status.get("type") == "idle"` and adds structured logging on every tick. 3. Interception proxy now preserves `delta.reasoning` on streaming chunks and surfaces it as `message.reasoning` on the assembled response. HF Router's Qwen3.5 thinking mode returns reasoning as a separate field from content; previously it was dropped. 4. `upstream_model` no longer strips the Qwen/ org prefix — full `config.model` is forwarded as the model-override so both vLLM (served as `Qwen/Qwen3.5-4B`) and HF Router (requires `Qwen/<repo>:<provider>`) work. 5. Structured logging at every factory.create phase so operators can see exactly which step is stuck (sandbox, bootstrap, proxy, serve, wait_for_completion). * test(opencode_env): drop live integration tests requiring external secrets The four live tests (OpenAI / vLLM / mode-B / E2B) required OPENAI_API_KEY, VLLM_URL, or E2B_API_KEY to execute and were development-time fixtures rather than CI checks. The core functionality is already covered offline by test_harness.py (end-to-end factory lifecycle against a mock sandbox + mock OpenAI endpoint), test_interception.py (proxy forward + per-turn record assembly), test_opencode_client.py (serve client over httpx mocks), and test_sandbox_base.py (E2BSandboxBackend key-required unit). * feat(opencode_env): deployable env with HF Space, Gradio UI, and 3-endpoint catalog Wraps the existing OpenCode harness primitive in a deployable OpenEnv environment that can run as an HF Space, exposing a single MCP ``run_rollout`` tool plus a Gradio web UI at /web. Highlights ---------- - Single MCP tool ``run_rollout`` accepting a uniform Task shape (instruction + setup[] + verify[] bash commands), reward = passed_verify / total or override via /home/user/logs/verifier/reward.txt. - Endpoint shorthand catalog (``vllm`` / ``openai`` / ``hf_router``) that resolves base_url / api_key / model from env vars + sane defaults. - In-sandbox FastAPI proxy (``transparent_proxy`` mode) injects logprobs=true and captures per-token logprobs for GRPO training. - Optional ``black_box`` mode skips the proxy for SFT / eval rollouts. - Pre-baked E2B template (``opencode-rl``) drops sandbox cold start from ~2min to ~6s by shipping opencode + proxy deps in the image. - Streaming Gradio UI: /run handler is a generator that yields a live phase log (sandbox boot → setup → agent → verify → collect) so the user sees progress instead of a spinner. - HF Space deployed at AdithyaSK/opencode-env, end-to-end verified against vLLM, OpenAI, and HF Router (all 3 reward=1.0 on the binary_search smoke task). Layout ------ envs/opencode_env/ {client.py, models.py, __init__.py} # HTTP client + pydantic {config.py, harness.py, opencode_runtime.py, task.py} # primitive (CLI-only) server/{app.py, opencode_environment.py, gradio_ui.py, catalog.py, Dockerfile} # FastAPI + Gradio + MCP sandbox/{base.py, e2b.py, interception.py, build_template.py} # E2B + proxy + template {pyproject.toml, openenv.yaml, uv.lock, README.md, .dockerignore, .gitignore} Removed (CLI-only refactor) --------------------------- - harness.py: dropped the ``opencode serve`` driver path (~270 LOC). - Deleted opencode_client.py, live_watch.py, env-local tests/. CI / tests ---------- - New tests/envs/test_opencode_env.py: 14 unit tests (no E2B, no LLM, no network) covering catalog resolution, model serialization, and task coercion. Plus one @pytest.mark.integration test that runs opencode end-to-end against the deployed Space (skipped by default). - sandbox/__init__.py: e2b import wrapped in try/except so the package loads cleanly without e2b installed (CI-friendly). - Added opencode-env to .github/workflows/docker-build.yml matrix so the image is built and pushed to GHCR alongside other envs. openenv-core dependency ----------------------- Currently pinned to the ``opencode-harness`` branch via git because PyPI's ``openenv-core`` (0.2.x) does not yet ship the ``openenv.core.harness`` module that this env imports. Switch to ``openenv-core[core]>=0.2.2`` once RFC 5 / PR #471 ships in a published release. The intended end-state is documented inline in pyproject.toml. * docs(opencode_env): add examples/opencode_env_simple.py Minimal end-to-end example: hits the deployed HF Space, runs a binary_search rollout via the MCP run_rollout tool, prints the reward + per-turn logprobs + the file the agent produced. Mirrors the per-env convention in ``examples/`` (echo_mcp_demo.py / coding_env_inference.py / atari_simple.py etc.). Defaults point at ``https://adithyask-opencode-env.hf.space``; override with ``OPENCODE_ENV_SPACE`` to target a different Space or local container. Requires ``OPENAI_API_KEY`` in the environment (passed in the request body, no Space secret required). Swap ``endpoint="openai"`` for ``"vllm"`` or ``"hf_router"`` to exercise the other backends. * fix: secure opencode proxy command --------- Co-authored-by: burtenshaw <ben.burtenshaw@gmail.com> * fix: address ci docs and formatting * feat(tutorials): add SFT warm-up tutorial with reasoning_gym collect support (#636) * feat(collect): add reasoning_gym support + --dataset-config + --system-prompt Extends `openenv collect` to support reasoning_gym environments: - Register ReasoningGymSessionFactory in _build_session_factory - Add --dataset-config option (JSON string) for env-specific config - Add --system-prompt option to override the default system prompt - Add ReasoningGymSessionFactory in envs/reasoning_gym_env/harness.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(harness): generate random seed per episode when none provided reasoning_gym server requires seed when dataset_name is specified. CollectRunner never passes a seed, so generate one per episode to ensure variety across collected rollouts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(llm_client): use max_completion_tokens for OpenAI client Newer OpenAI models (gpt-5-mini, o1, o3) reject max_tokens and require max_completion_tokens instead. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(llm_client): use max_completion_tokens only for OpenAI API, not self-hosted Newer OpenAI models require max_completion_tokens; self-hosted OpenAI-compatible endpoints (vLLM, Ollama, TGI) only support max_tokens. Add use_max_completion_tokens flag to OpenAIClient, enabled automatically by create_llm_client for the openai provider and left off for self-hosted endpoints via --llm-endpoint. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(llm_client): omit temperature for OpenAI API newer models gpt-5-mini and other newer OpenAI models only accept the default temperature (1) and reject any explicit value. Omit temperature entirely when use_max_completion_tokens is set; self-hosted endpoints are unaffected. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(collect): add rich progress bar to CollectRunner Shows episode count, collected count, running avg reward, and elapsed time during collection — previously the loop ran silently. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(tutorials): add SFT warm-up tutorial with end-to-end validated notebook Adds docs/source/tutorials/sft-warmup.md and examples/sft_warmup.ipynb. Tutorial collects rollouts via CollectRunner Python API, pushes to Hub, filters by reward, and fine-tunes Qwen3-1.7B with SFTTrainer. Validated end-to-end: 0% → 64% format compliance, 4% → 60% accuracy on chain_sum. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(tutorials,collect): address PR review findings - Scope use_max_completion_tokens to models that require it (gpt-5-mini, o1, o3, o4-mini) — was incorrectly applied to all OpenAI models, breaking gpt-4o and other standard models - Remove duplicate YOUR_HF_USERNAME re-declaration in tutorial section 10 - Add missing asyncio import in tutorial section 10 code block - Fix prose: max_seq_length → max_length (correct SFTConfig field name) - Fix usort import order in collect.py and harness/collect.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(collect,tutorials): address Greptile review findings - Move _MAX_COMPLETION_TOKENS_PREFIXES to module level as frozenset - Use prefix matching for versioned model names (o1-2024-12-17, etc.) - Replace asyncio.run() with await in tutorial section 10 (Jupyter compat) - Use json.dumps for tool_call text in to_qwen3_messages (safe escaping) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: burtenshaw <ben.burtenshaw@gmail.com> * fix: address harness review feedback * fix: format harness lint * fix: address harness review comments --------- Co-authored-by: Sergio Paniego Blanco <sergiopaniegoblanco@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Adithya S K <adithyaskolavi@gmail.com>

sergiopaniego and others added 5 commits April 17, 2026 17:21

meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Apr 17, 2026

Darktex requested changes Apr 17, 2026

View reviewed changes

fix: address collect feedback

d311f15

burtenshaw approved these changes Apr 18, 2026

View reviewed changes

burtenshaw merged commit 5414298 into huggingface:feature/harness-interface Apr 18, 2026
1 check passed

sergiopaniego deleted the feature/harness-collect branch April 20, 2026 09:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(harness): rollout collection + openenv collect CLI#560

feat(harness): rollout collection + openenv collect CLI#560
burtenshaw merged 6 commits into
huggingface:feature/harness-interfacefrom
sergiopaniego:feature/harness-collect

sergiopaniego commented Apr 17, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Apr 17, 2026

Uh oh!

Darktex left a comment

Uh oh!

burtenshaw commented Apr 18, 2026

Uh oh!

burtenshaw commented Apr 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

sergiopaniego commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of Change

Alignment Checklist

RFC Status

Test Plan

Claude Code Review

Related

Uh oh!

greptile-apps Bot commented Apr 17, 2026

Uh oh!

Darktex left a comment

Choose a reason for hiding this comment

Alignment Review Report

Automated Checks

Tier 1: Fixes Required

Tier 2: Alignment Discussion

Uh oh!

burtenshaw commented Apr 18, 2026

Uh oh!

burtenshaw commented Apr 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sergiopaniego commented Apr 17, 2026 •

edited

Loading