docs(tutorials): add end-to-end walkthrough

[sergiopaniego]

Summary

Adds an end-to-end tutorial that goes from pip install to a trained model in one page. It uses reasoning_gym_env (the OpenEnv wrapper around the Reasoning Gym library) as the environment, Qwen3-1.7B as the base model, and TRL's GRPOTrainer with the environment_factory pattern for training. The accompanying notebook lives at examples/grpo_reasoning_gym/end_to_end_walkthrough.ipynb and is generated from the markdown.

This is a first version. Once #471 is merged, some parts can be updated — in particular the evaluation, which today reads the reward delta from trainer.state.log_history and can be replaced by a held-out pass through the harness.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation
• New environment
• Refactoring

Alignment Checklist

Before submitting, verify:

• I have read .claude/docs/PRINCIPLES.md and this PR aligns with our principles
• I have checked .claude/docs/INVARIANTS.md and no invariants are violated
• I have run /pre-submit-pr (or bash .claude/hooks/lint.sh and tests) and addressed all issues

RFC Status

• Not required (bug fix, docs, minor refactoring)
• RFC exists: #___
• RFC needed (will create before merge)

Test Plan

• Tutorial ran end-to-end on Colab A100 (40 GB) — reward 62.52% → 83.25%, +20.73 pp

Claude Code Review

## Alignment Review Report

### Automated Checks
- Lint: PASS for the PR scope. The hook flags formatting in
  `envs/chat_env/`, `envs/repl_env/`, and `envs/textarena_env/`, none of
  which this PR touches — pre-existing on `main`, out of scope.
- Debug code: CLEAN for the PR scope. The hook flags `console.print`
  calls in `src/openenv/cli/commands/push.py` and pre-existing TODOs in
  `src/openenv/cli/`, none of which this PR touches.

### Open RFCs Context
- **RFC 005 (Agentic Harness Integration)** — *In Review*. This RFC
  covers wrapping external agentic harnesses (OpenClaw, Claude Code,
  Gemini CLI) inside OpenEnv containers. The tutorial uses TRL's
  `GRPOTrainer` directly without an external harness, so it does not
  pre-empt or conflict with RFC 005's design surface.

### Tier 1: Fixes Required
None.

### PR Scope
- 3 files: 1 new tutorial (`docs/source/tutorials/end-to-end-walkthrough.md`,
  319 lines), 1 new notebook (`examples/grpo_reasoning_gym/end_to_end_walkthrough.ipynb`,
  459 lines, generated from the markdown via `jupytext`), and a 2-line
  update to `docs/source/tutorials/index.md` adding the new entry to the
  toctree.
- All cross-links resolve in both Sphinx and Colab (internal links use
  absolute `meta-pytorch.org/OpenEnv/...` URLs so they work in the
  notebook context, where relative paths would break).

### Tier 2: Alignment Discussion

#### Principle Conflicts
None identified. The tutorial:
- Keeps rewards inside the environment (`ReasoningGymTrainEnv.answer`
  reads `result.observation.score` from `env.step` and stores it on
  `self.reward` — no external reward computation).
- Respects client–server separation (the wrapper imports only from
  `reasoning_gym_env` public client surface, never from `server/`).
- Reuses the `environment_factory` recipe from the migrated
  `wordle-grpo.md` rather than introducing a new framework-level pattern.

#### RFC Conflicts
None identified. The "first version" caveat in the Summary points at
[#471](https://github.com/meta-pytorch/OpenEnv/pull/471) /
`feature/harness-collect` as the natural place to swap the
`trainer.state.log_history` shortcut for a held-out eval pass — that is
adjacent to but does not conflict with RFC 005.

### Summary
- 0 mechanical issues to fix (PR scope is clean)
- 0 alignment points for human review
- 0 RFC conflicts to discuss

@adithya-s-k @burtenshaw

[greptile-apps]

Greptile Summary

This PR adds a new end-to-end tutorial covering pip install through a trained model using reasoning_gym_env, Qwen3-1.7B, and TRL's GRPOTrainer with the environment_factory pattern. The tutorial structure and explanations are clear and well-organized, but two P1 defects will break any reader who runs it top-to-bottom:

• MODEL_NAME is undefined — GRPOTrainer(model=MODEL_NAME, ...) in section 8 references a variable that is never assigned anywhere in the notebook, causing an immediate NameError before training starts.
• max_steps mismatch — the code sets max_steps=150 but the explanatory prose directly below says max_steps=100, creating a factual inconsistency that will confuse readers.

Confidence Score: 3/5

Not safe to merge as-is — the notebook raises a NameError on first execution due to undefined MODEL_NAME.

Two P1 defects (undefined MODEL_NAME causing NameError at runtime, and a max_steps code/prose mismatch) prevent the tutorial from working correctly out of the box. These are straightforward fixes but must be resolved before the tutorial can serve its purpose.

Both docs/source/tutorials/end-to-end-walkthrough.md and examples/grpo_reasoning_gym/end_to_end_walkthrough.ipynb carry the same bugs and need to be updated in sync.

Important Files Changed

Filename	Overview
docs/source/tutorials/end-to-end-walkthrough.md	New end-to-end GRPO tutorial; contains a P1 NameError (MODEL_NAME never defined), a P1 max_steps value mismatch between code and prose, a wrong section cross-reference, and a potential StatisticsError on an empty rewards list.
examples/grpo_reasoning_gym/end_to_end_walkthrough.ipynb	Notebook generated from the markdown; carries the same MODEL_NAME NameError, max_steps prose mismatch, and wrong section reference as the source .md file.
docs/source/tutorials/index.md	Two-line toctree update adding the new walkthrough entry; no issues found.

Sequence Diagram

sequenceDiagram
    participant User as User (Colab)
    participant Trainer as GRPOTrainer (TRL)
    participant Env as ReasoningGymTrainEnv
    participant Space as HF Space (reasoning_gym)
    participant Hub as Hugging Face Hub

    User->>Trainer: trainer.train()
    loop Per generation batch
        Trainer->>Env: env.reset(**row)
        Env->>Space: client.reset(dataset_name, config, seed, size)
        Space-->>Env: observation.question
        Env-->>Trainer: question string
        Trainer->>Trainer: Generate num_generations completions
        Trainer->>Env: env.answer(answer)
        Env->>Space: client.step(ReasoningGymAction)
        Space-->>Env: observation.score, correct_answer
        Env-->>Trainer: feedback string + env.reward
        Trainer->>Trainer: GRPO advantage computation and policy update
    end
    Trainer-->>User: trainer.state.log_history
    User->>Hub: trainer.push_to_hub()

Loading

Prompt To Fix All With AI

This is a comment left during a code review.
Path: docs/source/tutorials/end-to-end-walkthrough.md
Line: 251-258

Comment:
**`MODEL_NAME` is undefined — `NameError` at runtime**

`GRPOTrainer(model=MODEL_NAME, ...)` references a variable `MODEL_NAME` that is never defined anywhere in the tutorial. The model identifier `"Qwen/Qwen3-1.7B"` is only mentioned in prose; no cell assigns it to a Python variable. Anyone who runs the notebook top-to-bottom will hit a `NameError` before training even starts.

```suggestion
MODEL_NAME = "Qwen/Qwen3-1.7B"

trainer = GRPOTrainer(
    model=MODEL_NAME,
    reward_funcs=reward_func,
    train_dataset=dataset,
    args=grpo_config,
    environment_factory=ReasoningGymTrainEnv,
)

trainer.train()
```

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: docs/source/tutorials/end-to-end-walkthrough.md
Line: 241

Comment:
**`max_steps` mismatch between code and prose**

The `GRPOConfig` block sets `max_steps=150`, but the explanatory paragraph immediately below says "`max_steps=100` caps the run before saturation." Readers following the prose rationale will be confused about which value is actually in effect. The note about saturation behaviour should reference the value actually in the code (`150`). The same paragraph also says "section 10" for the Hub push but that step is section 9.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: docs/source/tutorials/end-to-end-walkthrough.md
Line: 294-301

Comment:
**`statistics.mean` will crash on an empty list**

If training is interrupted or fewer than 1 log entry contains `"reward"`, `rewards` will be empty and `statistics.mean([])` raises `StatisticsError: mean requires at least one data point`. A simple length guard would prevent a confusing traceback for beginners who interrupt the run early.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: docs/source/tutorials/end-to-end-walkthrough.md
Line: 48

Comment:
**Installing from `@main` of transformers pins to an unstable branch**

`!pip install -q git+https://github.com/huggingface/transformers.git@main` installs the current tip of `main`, which changes daily. For a tutorial meant to be reproducible, pinning to a release tag or PyPI version (e.g., `transformers>=4.51.0`) is strongly preferred.

How can I resolve this? If you propose a fix, please make it concise.

_{Reviews (1): Last reviewed commit: "docs(tutorials): add end-to-end GRPO wal..." | Re-trigger Greptile}

[Darktex]

Note: This is an automated review by Claude Code, not a human review.

---

Alignment Review Report

Automated Checks

• Lint: PASS (pre-existing issues in unrelated files)
• Debug code: CLEAN (PR scope)

Tier 1: Fixes Required

• docs/source/tutorials/end-to-end-walkthrough.md and examples/grpo_reasoning_gym/end_to_end_walkthrough.ipynb — {note} blocks are MyST-Sphinx directives. They render correctly in Sphinx docs but appear as raw fenced code blocks (```{note} ... ```) in Jupyter/Colab. Since there is a prominent "Open in Colab" badge at the top, notebook readers will see broken formatting on all four note blocks. Use plain Markdown > **Note:** ... callouts inside the notebook cells, and keep {note} only in the .md source.

Tier 2: Alignment Discussion

None identified. The tutorial:

• Keeps rewards inside the environment (result.observation.score read from env.step, stored as self.reward — no external computation).
• Respects client-server separation (imports only from reasoning_gym_env public surface: ReasoningGymEnv, ReasoningGymAction).
• Does not expose reset() or simulation controls to the model-under-training.
• Uses the established environment_factory pattern consistent with the Wordle GRPO tutorial.

Additional Polish

The tutorial acknowledges that num_generations=2 is low and that reward_std may hit persistent zero — but the warning appears in section 8's prose, not adjacent to the config value in section 7 where a reader would naturally look for guidance. A one-line comment (# low for memory; raise to 4+ for better advantage signal) on the num_generations=2 line would help readers catch this before training.

Summary

• 1 issue to fix (MyST admonitions break in Colab notebook)
• 0 alignment points for human review

---

Automated review by Claude Code | Learn more

[Darktex]

Note: This is an automated review by Claude Code, not a human review.

---

Alignment Review Report

Automated Checks

• Lint: PASS for PR scope. Pre-existing formatting issues in envs/chat_env/, envs/repl_env/, envs/textarena_env/, and src/openenv/cli/ are untouched by this PR.
• Debug code: CLEAN for PR scope. Pre-existing print statements in src/openenv/core/containers/test_local_docker_provider.py and TODOs in src/openenv/cli/ are untouched by this PR.

Tier 1: Fixes Required

• examples/grpo_reasoning_gym/end_to_end_walkthrough.ipynb (cells containing {note}) - MyST admonition syntax ({note}) is a Sphinx-only directive. In Colab and standard JupyterLab it renders as a fenced code block with {note} as the language specifier, not as a styled callout. Since Colab is the primary audience for .ipynb tutorials, these four note blocks should be converted to plain Markdown blockquotes (e.g. > **Note:** ...) so they render correctly in both Sphinx (via the .md source) and Colab (via the .ipynb copy). The .md source can keep {note} as-is since it is processed by Sphinx.

Tier 2: Alignment Discussion

ALIGNMENT FLAG: External reward_func reads env.reward from a client-side wrapper attribute

• Principle at stake: "Rewards inside environment" (PRINCIPLES.md, RFC 002)
• The concern: The tutorial introduces reward_func(environments, **kwargs) -> list[float] as a top-level function that reads env.reward from the ReasoningGymTrainEnv wrapper. The reward value itself is computed server-side (arrives via result.observation.score), but the indirection through a client-side attribute and an external function is a new teaching pattern. For this tutorial it is benign - the score is computed by the server and merely cached on the wrapper for the TRL callback. However, canonicalizing this pattern in the docs means future tutorial authors may copy it and inadvertently move reward computation logic into the external reward_func body rather than inside the environment. The concern is about what we are teaching, not about this specific instance. A brief inline callout explaining why the reward is safe here (server computed, client just caches) would prevent cargo-culting.
• Suggested reviewer: @Darktex

ALIGNMENT FLAG: Hardcoded personal HF Space URL as the default endpoint

• Principle at stake: "Be hands-on: Provide ready-to-use implementations" and production-readiness (PRINCIPLES.md)
• The concern: base_url="https://sergiopaniego-reasoning-gym.hf.space" and git+https://huggingface.co/spaces/sergiopaniego/reasoning_gym are personal Space URLs in the code that readers copy-paste and run verbatim. The tutorial prose correctly notes these are rate-limited and instructs readers to deploy their own copy, but the code block default leaves personal infrastructure as the load-bearing path for every tutorial run. If the Space goes down, scales down, or is modified, all tutorial runs silently break. Consider whether there is an org-owned Space (e.g. meta-pytorch/reasoning_gym) that should be the canonical default, with the personal Space noted as the historical demo.
• Suggested reviewer: @Darktex

Summary

• 1 mechanical issue to fix (MyST note syntax in .ipynb renders as raw code in Colab)
• 2 alignment points for human review (reward pattern pedagogy, hardcoded personal Space URL)

---

Automated review by Claude Code | Learn more

[Darktex]

Note: This is an automated review by Claude Code, not a human review.

---

Summary

Well-written tutorial, structurally sound, imports and API calls match the current reasoning_gym_env public surface exactly. One quality issue: MyST directive fences inside the Jupyter notebook cells will not render in Colab/Jupyter and should be converted to plain markdown callouts.

Tier 1 — Bugs / Quality

• examples/grpo_reasoning_gym/end_to_end_walkthrough.ipynb — Three markdown cells contain MyST-style ```{note} fences (e.g. the "T4 won't fit" note, the "Replace the base_url" note, the "chat_template_kwargs" note, and the "delta is measured during training" note). MyST directives are processed by Sphinx during docs builds but Jupyter/Colab renders them as literal fenced code blocks, so readers opening the notebook in Colab see ugly raw-text boxes instead of rendered callouts. Fix: convert each ```{note}\n...\n``` block in the notebook to a plain > **Note:** ... blockquote. The .md doc file can keep the {note} directives because it is Sphinx-rendered only.

• docs/source/tutorials/end-to-end-walkthrough.md line 47: !pip install -q --no-deps git+https://huggingface.co/spaces/sergiopaniego/reasoning_gym installs directly from a personal HF Space (not the canonical openenv-core package path). The tutorial body already warns readers to substitute their own deployment, but the hardcoded personal Space URL in the primary install cell will silently break for any reader whose Space clone diverges or is deleted. Consider replacing with !pip install -q --no-deps git+https://github.com/meta-pytorch/OpenEnv#subdirectory=envs/reasoning_gym_env (or a PyPI release once available) and reserving the Space install for a clearly-labelled "shortcut" note.

Tier 2 — Alignment

• The reset() and step() calls in ReasoningGymTrainEnv are invoked by the TRL trainer (training orchestration), not by the agent being trained — this is the correct split and does not violate the "agents cannot reset" invariant. No flag needed.
• reward_func reads env.reward, which was set inside answer() from result.observation.score returned by env.step() — reward computation stays inside the environment boundary. Aligned with RFC 002.
• Client imports are from reasoning_gym_env public surface (__init__.py), no server-side imports. Aligned with the client-server separation invariant.
• The tutorial acknowledges that the reward delta metric is a training-curve shortcut and explicitly flags the held-out eval path (via harness, #471) as follow-up work. Aligned with the "honest about limitations" expectation in docs.

Verdict

Two concrete quality fixes requested: MyST directives in the notebook (renders badly in Colab) and the hardcoded personal Space URL in the install cell. No invariant or alignment issues.

---

Automated review by Claude Code | Learn more

[burtenshaw]

Approved per maintainer merge request after required checks passed.