[rng1995]

Summary

v0.5.0 published successfully to https://docs.nvidia.com/skills/skillevaluator/, and the Harbor 0.24 content was already live. This PR fills the gaps that remained: features in the 0.5.0 changelog that no page described, and statements the v0.5.0 CLI or source contradicts. Every page under docs/ was checked against --help output, the source, and the changelog.

New content

• Tier 3 → "How standard grading judges a trial":
  • Covers which signals are deterministic and which are LLM judges, and how trajectories are rebuilt from opencode.txt and codex.txt when trajectory.json is missing (feat(tier3): rebuild ATIF from OpenCode and Codex agent logs #119).
  • Shows the evidence bundle each judge sees and what truncation means for scoring (feat(tier3): make evidence bundle budgets and final response limit configurable #160).
  • Explains structured JSON output (response_format / output_config), with schema downgrade on 400/422, and retries within the per-judge time budget (feat(tier3): enforce structured JSON schemas and rate-limit retries in LLM judges #145).
• Environment variables:
  • New sections: "LLM retries and judge time budget", "Judge evidence budgets" (SKILL_EVAL_*_BUDGET, SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT), and "Harbor process environment" (.env / .env.local not read; allowlisted environment; python-dotenv ≥ 1.2.0).
  • New rows: LLM_JUDGE_FALLBACK_MODELS and SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE.

Corrections

• CLI reference:
  • Adds validate --workers and --previous-version.
  • Lists all 24 --env-mode values, plus wandb → cwsandbox and the nine backends that can't be selected.
  • Adds the reserved --ek names and the rejection of kwargs a backend doesn't accept.
  • Fixes the --timeout-multiplier scope and default.
  • Notes that --refine doesn't need the tier3 extra to read existing results.
  • Gives the dedup-scan report basename.
• Exit codes (CLI reference, CI, Tier 1): removes code 3. ExitCode.RUNTIME_ERROR is never used, and an unexpected error exits 1. Code 2 is now described as a Click usage error.
• Custom graders:
  • Corrects the separate-verifier precedence: a step tests/ directory is a dedicated verifier only with a Dockerfile/docker-compose.yaml or docker_image.
  • Case IDs come from the staged task.
  • Custom graders get no trajectory rebuild.
  • Covers dual-arm [task] name suffixes.
• Reports: adds the contents of trials/, trajectory_source / omitted in reward.json, and the logical-attempt overall when standard and custom rewards are mixed.
• Agents & sandboxes / configuration:
  • Unknown --ek keys are rejected; drop modal_sandbox_v2.
  • stream and enable_environment_dir_upload are reserved.
  • With NVIDIA Build, OpenCode follows SKILL_EVAL_LLM_MODEL, while Codex and Claude Code keep Super.
  • Export credentials in the shell, because .env is not loaded.
• Others:
  • Tier 1: non-string YAML keys are reported.
  • Eval datasets: refining from existing results works without tier3.
  • Installation: tier3 extra pins.
  • Benchmark rollout: Tier 3 runs by default.
  • CI: SARIF in the gate command.
  • Developer guide: CITATION.cff excerpt.
  • Page lists in docs/README.md and docs/AGENTS.md.

Verification

• fern check: 0 errors. The only warning is the redirects check, which needs FERN_TOKEN.
• A script confirmed every .mdx#anchor cross-link resolves to a heading. Added lines contain no bare < or {.
• Docs-related and golden tests (test_benchmark_rollout_docs, test_oss_packaging, tests/golden, …): 687 passed, 2 skipped.
• Exit codes checked live: a missing path and an unknown flag both exit 2.

Merging publishes to docs.nvidia.com through publish-docs.yml.

🤖 Generated with Claude Code