Skip to content

fix: resolve CritPt benchmark config interpolation and add critpt_agent README #1642

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
linj-glitch merged 1 commit into from
Jun 22, 2026
Merged

fix: resolve CritPt benchmark config interpolation and add critpt_agent README #1642

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion resources_servers/critpt/configs/critpt.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ critpt_resources_server:
critpt:
entrypoint: app.py
domain: other
api_key: ${artificial_analysis_api_key}
api_key: ${oc.select:artificial_analysis_api_key,null}
verified: false
description: Research-level physics problems scored by the Artificial Analysis API
value: Evaluate model performance on research-level physics reasoning
Expand Down
22 changes: 22 additions & 0 deletions responses_api_agents/critpt_agent/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# CritPt Agent

Custom two-turn agent for the CritPt benchmark (research-level physics problems). For each problem it:

1. **Turn 1 — solve:** sends the problem to the model and collects its reasoning and conclusion.
2. **Turn 2 — fill the template:** strips the Turn 1 thinking blocks (`<think>` / `<thinking>`), then asks
the model to populate the problem's `code_template` using the Turn 1 conclusion as context (the Turn 2
prompt instructs the model not to reason again).

The accumulated Turn 2 output is submitted to the CritPt resources server's `/verify`, which scores it via
the Artificial Analysis API.

## Configuration

- `resources_server`: the CritPt resources server instance to seed sessions and verify against
- `model_server`: the model server used for generation
- `turn2_prompt_fpath`: the Turn 2 user-prompt template filled with the problem's `code_template`
(e.g. `benchmarks/critpt/prompts/turn2.yaml`)

The full wiring (resources server + this agent + example dataset) lives in
`resources_servers/critpt/configs/critpt.yaml`; the benchmark dataset is narrowed in
`benchmarks/critpt/config.yaml`.
Loading