Skip to content

docs: improve model-call capture guide - #2415

Merged
ritaneves merged 3 commits into
mainfrom
rfneves/model-call-cap-docs
Aug 10, 2026
Merged

docs: improve model-call capture guide#2415
ritaneves merged 3 commits into
mainfrom
rfneves/model-call-cap-docs

Conversation

@ritaneves

@ritaneves ritaneves commented Aug 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • add a copy-paste model call capture workflow from environment startup through inspecting ng_model_call_capture
  • show representative attachment and CaptureStore JSONL output, including gaps and the full metrics shape
  • lead with the consumer workflow and move custom-agent correlation helpers into an advanced section
  • clarify that agent observations are currently emitted by the Claude Code agent
  • add model call capture to the model-server landing-page card grid
  • keep the latest and v0.5.0 documentation copies aligned

Why

Addresses NVBug 6534797. The existing page described capture internals but did not show readers how to enable capture in a real Gym run, where to find the result, or the complete output shape.

Validation

  • pytest -q tests/unit_tests/test_fern_docs_links.py
  • ruff check tests/unit_tests/test_fern_docs_links.py
  • git diff --check

Release cherry-pick companion: #2416

@copy-pr-bot

copy-pr-bot Bot commented Aug 7, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@ritaneves ritaneves mentioned this pull request Aug 7, 2026
Merged
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

🌿 Preview your docs: https://nvidia-preview-rfneves-model-call-cap-docs.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

@ritaneves
ritaneves force-pushed the rfneves/model-call-cap-docs branch from e1adeab to df29153 Compare August 10, 2026 13:02
@ritaneves
ritaneves marked this pull request as ready for review August 10, 2026 15:14
@ritaneves
ritaneves force-pushed the rfneves/model-call-cap-docs branch from 8cecd92 to da089f6 Compare August 10, 2026 15:23
@ritaneves
docs: improve model-call capture guide
e618ba0 
Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com>
@ritaneves
docs: mirror model-call capture guide to v0.5.0
b40e02c 
Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com>
@ritaneves
docs: label capture record example as synthetic
a7db2fd 
Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com>
@ritaneves
ritaneves force-pushed the rfneves/model-call-cap-docs branch from da089f6 to a7db2fd Compare August 10, 2026 15:27
@ritaneves
ritaneves merged commit 3260c98 into main Aug 10, 2026
45 checks passed
@ritaneves
ritaneves deleted the rfneves/model-call-cap-docs branch August 10, 2026 15:47
ritaneves added a commit that referenced this pull request Aug 10, 2026
@ritaneves
docs: cherry-pick model-call capture guide to r0.5.0 (#2416)
f8d4318 
## Summary

Release-branch companion to primary PR #2415 for NVBug 6534797.

- cherry-pick the model-call capture documentation improvements onto
`r0.5.0`
- apply the workflow, examples, agent-observation scope, and
discoverability card to both `latest` and `v0.5.0`
- preserve release-line-specific documentation while resolving the
cherry-pick conflicts
- explicitly identify the raw `CaptureStore` example as synthetic while
documenting its exact persisted field shape

## Validation

- `pytest -q tests/unit_tests/test_fern_docs_links.py` — 13 tests and 15
subtests passed
- `ruff check tests/unit_tests/test_fern_docs_links.py`
- `git diff --check origin/r0.5.0...HEAD`

---------

Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com>
This was referenced Oct 12, 2026
Closed
Merged
ehosseiniasl pushed a commit to ehosseiniasl/Gym that referenced this pull request Oct 13, 2026
@qiaochuz-nv @claude
docs(model-server): finish model-call capture page polish (NVIDIA-NeM…
88a7c0b 
…o#4550)

## What does this PR do?

Finishes the Model-Call Capture page review from NVBug 6534797. NVIDIA-NeMo#2415
rewrote the page and covered most of the report. This PR closes the
items still open on `main`:

- Title the page **Model-Call Capture**, like its sibling pages, and use
that title on the index card and on links that name the page.
- Link "global config" in *Configure capture* to the
[Configuration](/main/reference/configuration) reference.
- In the field table, show `—` as the default for
`model_call_capture_dir`. Its description now says it is required when
`observability_enabled` is true.
- Replace the custom-agent helper paragraphs with a table: one row per
helper and when to use it.
- Replace the remaining "kill-shaped" jargon on the main docs tree, in
the Metrics page's description of `omitted` outcomes, with the
`kill_shaped` class name and what it covers.

Fixes NVBug [6534797](https://nvbugspro.nvidia.com/bug/6534797).
Closes NVIDIA-NeMo#4494.

### Cause

The bug's 2026-08-06 comment listed polish items for the page. NVIDIA-NeMo#2415
(`3260c981e`) addressed the original report, adding the copy-paste run,
samples, and index card. Its review raised none of the comment's medium
and low items, so they were left as-is:

- **Title casing.** The title stayed `"Model-call capture"`. 120 of the
124 page titles under `fern/versions/latest/pages` are Title Case,
including every sibling in `model-server/`. Links that name the page
used two different casings, neither of them the Title Case form.
- **Configure capture section.** The section did not link to the
Configuration reference, and the Default cell read `required when
enabled`.
- **Custom-agent helpers.** The section still listed the five helpers in
running prose.
- **"kill-shaped" jargon.** NVIDIA-NeMo#2415 removed the phrase from this page.
NVIDIA-NeMo#3638 (`7321716b7`) later added "a kill-shaped failure" to
`observability/metrics.mdx`.

### Fix

Edits are in `fern/versions/latest/pages` only:

- **`model-server/model-call-capture.mdx`:**
  - Title.
  - Configuration link.
- Default cell. The requirement matches the runtime check in
`nemo_gym/base_responses_api_model.py`, which raises
`model_call_capture_dir is required when observability_enabled=true`.
- Helper table. Checked against `nemo_gym/base_responses_api_agent.py`
and how `simple_agent` uses the helpers: `url_path_for_run` prefixes the
run's self-call to `/v1/responses`, and `url_path_for_request` carries
that prefix on to the model call.
- **Pages that link to it:**
  - `model-server/index.mdx`: card title.
- `model-server/anthropic-messages.mdx`,
`observability/rollout-evidence.mdx`,
`observability/harness-conformance.mdx` and
`training-tutorials/external-agent-harnesses.mdx`: link label.
- Mid-sentence prose that uses the lowercase noun "model-call capture"
is unchanged.
- **`observability/metrics.mdx`:** an `omitted` outcome is now "a
no-persist result written to neither, such as a `kill_shaped` failure: a
Slurm SIGTERM, a dead Ray actor, or an out-of-memory kill". This matches
`evaluation/aggregate-metrics.mdx` and the `kill_shaped` comment in
`nemo_gym/rollout_collection.py`.
- **Not changed:**
- The versioned folders (`v0.5.0` through `v0.7.0`). They are frozen GA
snapshots, which `fern/README.md` says change only by deliberate
back-port.
- The H1 the bug calls optional. No `model-server/` sibling page has
one.
- **Tests:** `tests/unit_tests/test_fern_docs_links.py` gets one test
per item. The existing index-card test now expects the new title on
`latest` and the shipped title on the snapshots.

### Before the fix

Gym `7f16ec541` (`main`), with this PR's test file:

```bash
pytest tests/unit_tests/test_fern_docs_links.py -q -rf --tb=line
```

```text
AssertionError: Lists differ: [] != ['observability/metrics.mdx']
AssertionError: '—' != 'required when enabled'
AssertionError: Regex didn't match: '(?m)^\\| `url_path_for_run\\(' not found in "## If you build a custom agent ...
AssertionError: Regex didn't match: '(?m)^\\| `url_path_for_request\\(' not found in ...
AssertionError: Regex didn't match: '(?m)^\\| `base_url_for_run\\(' not found in ...
AssertionError: Regex didn't match: '(?m)^\\| `rollout_id_from_run\\(' not found in ...
AssertionError: '](/main/reference/configuration)' not found in '## Configure capture\n\nThe CLI workflow above sets these top-level global-config fields: ...
AssertionError: 'title: "Model-Call Capture"' not found in '---\ntitle: "Model-call capture"\n ...
AssertionError: '<Card title="Model-Call Capture" href="/model-server/model-call-capture">' not found in ...
FAILED tests/unit_tests/test_fern_docs_links.py::TestFernDocsLinks::test_latest_docs_explain_kill_shaped_failures_in_plain_language
FAILED tests/unit_tests/test_fern_docs_links.py::TestFernDocsLinks::test_model_call_capture_config_table_gives_the_capture_dir_no_default
FAILED tests/unit_tests/test_fern_docs_links.py::TestFernDocsLinks::test_model_call_capture_links_global_config_fields_to_the_configuration_reference
FAILED tests/unit_tests/test_fern_docs_links.py::TestFernDocsLinks::test_model_call_capture_title_matches_every_link_that_names_the_page
9 failed, 19 passed, 16 subtests passed
```

Link labels that name the page on `main`, from
`fern/versions/latest/pages`:

```bash
grep -rno '\[Model-call [Cc]apture\]([^)]*)' .
```

```text
training-tutorials/external-agent-harnesses.mdx:150:[Model-call capture](/main/model-server/model-call-capture#if-you-build-a-custom-agent)
observability/rollout-evidence.mdx:39:[Model-call Capture](/main/model-server/model-call-capture)
observability/harness-conformance.mdx:141:[Model-call Capture](/main/model-server/model-call-capture)
model-server/anthropic-messages.mdx:141:[Model-call capture](/model-server/model-call-capture)
```

### After the fix

Gym `ed8420b24` (this branch), same commands:

```text
23 passed, 21 subtests passed
```

The grep finds no mixed-case labels; every label that names the page is
now `Model-Call Capture`.

### Validation

- `pytest tests/unit_tests/test_fern_docs_links.py`: before 9 failed /
19 passed; after 23 passed.
- `cd fern && npm run check` (`fern check`): 0 errors, 1 warning, the
same as `main`. The warning is "Missing redirects check skipped: not
authenticated".
- `pre-commit run --files <changed files>`: passed.
- QA testcase: none links this bug. No `nmfw_tests` case reads the
edited text.
- Rollout evidence: N/A. This is a docs-only change with no runtime
behavior.

### Compatibility

Page URLs and anchors are unchanged. Only the nav label changes, from
"Model-call capture" to "Model-Call Capture".

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: qiaochuz <qiaochuz@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
public — a7db2fd1 Deployed Aug 10, 2026 by copy-pr-bot[bot] via release / finalize / notify #1249
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kajalj22 kajalj22 kajalj22 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ritaneves @kajalj22