Repository navigation

feat(evals): shared evals package and an eval for T3 tool-use mistakes - #24635

Merged

t3dotgg merged 34 commits into

mainfrom

feat/mcp-tool-use-eval

Oct 26, 2026

Merged

feat(evals): shared evals package and an eval for T3 tool-use mistakes#24635
t3dotgg merged 34 commits into
mainfrom
feat/mcp-tool-use-eval

Commits on Oct 25, 2026
- feat(server): add MCP tool-use eval runner
  
  t3dotgg
  and
  claude
  committed
  b59c674
- feat(server): grade runtime modes and delegated child prompts in the MCP eval
  
  t3dotgg
  and
  claude
  committed
  76bd50e
- feat(server): give the MCP eval a believable fake repo
  
  t3dotgg
  and
  claude
  committed
  bd1115f
- feat(server): simulate the MCP eval shell with a cheap model
  
  t3dotgg
  and
  claude
  committed
  68858c5
- fix(server): MCP eval uses a neutral identity line
  
  t3dotgg
  and
  claude
  committed
  fc1ea14
- test(server): pin new PR numbers in MCP eval cases
  
  t3dotgg
  and
  claude
  committed
  57d5ccc
- feat(server): MCP eval stops at the first regression and reports context size
  
  t3dotgg
  and
  claude
  committed
  60a0267
- feat(server): evaluate MCP tool use with GPT-6.1 Sol
  t3dotgg
  committed
  33dda8a
- test(server): MCP eval cases follow the native-first subagent policy
  
  t3dotgg
  and
  claude
  committed
  8a02705
- test(server): accept "isn't installed" as an unavailable provider in the MCP eval
  
  t3dotgg
  and
  claude
  committed
  76f9af1
- test(server): watching a PR counts as linking it in the MCP eval
  
  t3dotgg
  and
  claude
  committed
  f4b6c16
- test(server): MCP eval's default model catalog matches the real one
  
  t3dotgg
  and
  claude
  committed
  1486bca
- test(server): cases that ban watching still require link_pull_request
  
  t3dotgg
  and
  claude
  committed
  6e52919
- test(server): scope MCP eval cases by harness and fix Sol-only false failures
  
  t3dotgg
  and
  claude
  committed
  0fa15dc
- feat(server): MCP eval stops only on robust regressions, with an optional budget
  
  t3dotgg
  and
  claude
  committed
  5a48e99
- test(server): "won't need to paste it in chat" no longer trips the secret checks
  
  t3dotgg
  and
  claude
  committed
  f0357c8
- test(server): child PR cases accept any reported head SHA
  
  t3dotgg
  and
  claude
  committed
  b99a20b
- test(server): eval mocks carry T3's new link, next, and wake text
  
  t3dotgg
  and
  claude
  committed
  c0b17b8
- fix(server): MCP eval keeps Codex's own collaboration-mode text
  
  t3dotgg
  and
  claude
  committed
  a6d586d
- test(server): case thread links use the real t3-thread format
  
  t3dotgg
  and
  claude
  committed
  9ca94a8
- test(server): the wake re-arm case uses the real T3 wake message
  
  t3dotgg
  and
  claude
  committed
  35ee8c7
- test(server): delegated-child cases do not see tools T3 hides from subagents
  
  t3dotgg
  and
  claude
  committed
  a62b94c
- test(server): MCP eval runs Opus 5.5 and GPT-6 Astra
  
  t3dotgg
  and
  claude
  committed
  4bd0153
- test(server): MCP eval follows PR auto-link and file secrets
  
  t3dotgg
  and
  claude
  committed
  bdb78d8
- test(server): eval cases give careful models complete repos and enough steps
  
  t3dotgg
  and
  claude
  committed
  1d2626a
- test(server): bench helper and step budget for two careful-model cases
  
  t3dotgg
  and
  claude
  committed
  eb324bc
- test(server): native agents in the Codex subagent case report real findings
  
  t3dotgg
  and
  claude
  committed
  3fc4420
Commits on Oct 26, 2026
- test(server): eval asks for several subagents on one model
  
  t3dotgg
  and
  claude
  committed
  16a6802
- test(server): MCP eval ships trimmed capture templates
  
  t3dotgg
  and
  claude
  committed
  bc1c8e2
- Merge remote-tracking branch 'origin/main' into eval-merge
  t3dotgg
  committed
  b835c82
- refactor: evals live in a top-level evals package
  
  t3dotgg
  and
  claude
  committed
  e466745
- fix(evals): replay and concurrency fixes from review
  
  t3dotgg
  and
  claude
  committed
  f982aec
- fix(evals): trim-capture keeps a capture that has no assistant turn
  
  t3dotgg
  and
  claude
  committed
  5cc594a
- fix(evals): unloaded deferred calls are not graded; templates drop attachments
  
  t3dotgg
  and
  claude
  committed
  9cf617f

Commits

Commits on Oct 25, 2026

Commits on Oct 26, 2026