docs: document resume_from_cache for rollout collection #1239

Background

resume_from_cache is a config parameter on RolloutCollectionConfig that enables checkpoint-and-resume for interrupted rollout collection runs. The functionality is fully implemented and works for any rollout collection — both ng_collect_rollouts and ng_e2e_collect_rollouts — across all environments (not just benchmarks). However, user-facing documentation is minimal.

Current state

Code (complete):

• resume_from_cache field on RolloutCollectionConfig (default: False)
• On first run, writes a *_materialized_inputs.jsonl file containing the fully expanded inputs (after num_repeats, limit, etc.)
• Results are written incrementally with flush() after each completed rollout, so partial output survives crashes
• On resume, diffs completed (task_index, rollout_index) pairs against materialized inputs to determine remaining work
• Prints a clear summary: "Found X original input rows, Y already run, Z still need to be run"

Docs (incomplete):

• resume_from_cache is not listed in the ng_collect_rollouts parameter table in the CLI reference
• The only mention in the Fern docs is a two-line <Tip> in run-benchmark-suite.mdx (v0.2)
• ~10 individual benchmark/server READMEs use +resume_from_cache=True in example commands but none explain what it does
• The debugging skill documents the stale-cache footgun but users have no way to learn this proactively

What to document

1. Add resume_from_cache to the CLI reference parameter table for ng_collect_rollouts (and ng_e2e_collect_rollouts if applicable). This is the most basic gap — the parameter exists but isn't listed.

2. Explain saved artifacts and where they live:

   • <output>_materialized_inputs.jsonl — the fully expanded input rows (what was the input after num_repeats, limit, prompt_config, etc.)
   • <output>.jsonl — results written incrementally, one line per completed rollout
   • How the output path determines the materialized inputs path (.with_stem(stem + "_materialized_inputs"))

3. Document resume behavior:

   • How matching works: (task_index, rollout_index) pair comparison
   • What "already run" means (present in output file) vs. "still need to be run"
   • That results are flushed per-rollout so partial output is safe after crashes
   • Graceful fallback: if either file is missing, resume is skipped with a message

4. Document the known footgun:

   • If you change config/schema/data between runs and resume from cache, materialized inputs are stale
   • When to delete the materialized inputs file and start fresh
   • The resume_from_cache=False default deliberately clears output to avoid this

5. Recovery workflow (end-to-end):

   • Run crashes or is interrupted → restart the same command with +resume_from_cache=true → only remaining rows are run → final output contains all results
   • Relevance to long-running jobs on Slurm / large-scale collection where interruptions are expected

Suggested location

This could be:

• A section in the existing rollout collection docs / getting-started page
• An expansion of the CLI reference with a "Resume interrupted runs" section
• A short how-to page under troubleshooting

Probably not a standalone page — the feature is simple enough that a well-placed section covers it.

Scope

• In scope: Documenting the existing resume_from_cache functionality
• Out of scope: New features, code changes, or enhancements to the resume mechanism itself