Repository navigation
feat(launcher): add fast benchmark profiles and named resumable runs - #3602
Open
christian-muertz wants to merge 1 commit into
Open
christian-muertz wants to merge 1 commit into
christian-muertz wants to merge 1 commit into
Conversation
christian-muertz
force-pushed
the
codex/benchmark-fast-runs
branch
from
September 22, 2026 09:39
e791870 to
466dc5b
Compare
This was referenced Sep 22, 2026
yaoyu-33
added
area:orchestration
christian-muertz
force-pushed
the
codex/benchmark-fast-runs
branch
from
September 22, 2026 12:10
466dc5b to
5d3398c
Compare
7 tasks done
christian-muertz
force-pushed
the
codex/benchmark-fast-runs
branch
2 times, most recently
from
September 22, 2026 14:38
bdca7eb to
9d0edd4
Compare
github-actions
Bot
added
the
sla:triage-overdue
christian-muertz
force-pushed
the
codex/benchmark-fast-runs
branch
2 times, most recently
from
September 23, 2026 10:04
e2a5190 to
81f39be
Compare
Add shared fast serving settings, benchmark overrides, named output directories, and serialized restart submissions. Preserve the supplied implementation while applying it on top of the public launcher branch. Co-authored-by: Brian Fitzgerald <bfitzgerald@nvidia.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Christian Muertz <cmrtz@nvidia.com>
christian-muertz
force-pushed
the
codex/benchmark-fast-runs
branch
from
September 23, 2026 10:51
81f39be to
f851606
Compare
This was referenced Oct 22, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Brian Fitzgerald's fast benchmark profiles and named resumable runs on top of #3600.
Named runs share a results directory and queue successive Slurm jobs with a singleton dependency, allowing each job to resume cached rollouts. Fast Lightning profiles cover Terminal-Bench 2.1 (Terminus and OpenCode), SWE-bench Verified/Multilingual, and IFBench. They share capped generation settings, a pinned vLLM nightly, and routing of failed agent runs to the failures sidecar. Includes the development-checkout cache-directory and cleanup-script fixes.
The four-task smoke profile requires a separately prepared
benchmarks/terminal_bench_2_1/data/smoke4_distinct.jsonl; no dataset or local configuration is included in this PR. Fast settings change evaluation budgets, so results should be identified as using these profiles.Validation: repository-wide pre-commit checks, Bash syntax, YAML base-config references, and 16 mocked named/unnamed launcher invocations passed. Secret scanning passed. No live evaluation jobs were submitted; this ports the existing changes without retuning the benchmark behavior.
Stack: #3599 → #3600 → this PR. Review against
codex/benchmark-launcher.