[rng1995]

Summary

This PR moves SkillEvaluator's default models to GPT-6.1 Sol (OpenAI / Codex) and Claude Opus 5.5 (Anthropic / Bedrock / claude-code) across Tiers 1–3. It also lands the request-shape, refusal and runtime changes those models need. Opened as a draft until the live smoke runs listed under Rollout gates pass.

Use	Before	After
OpenAI chat default: Tier 1 verifier and rubric, Tier 2 dedup, Tier 3 judges, native codex	gpt-5.6-sol	gpt-6.1-sol
Anthropic / Bedrock chat default: same roles, native claude-code	claude-opus-5 / us.anthropic.claude-opus-5	claude-opus-5-5 / us.anthropic.claude-opus-5-5
Gateway codex / claude-code	openai/openai/gpt-5.6-sol / aws/anthropic/bedrock-claude-opus-5	openai/openai/gpt-6.1-sol / aws/anthropic/bedrock-claude-opus-5-5
SkillSpector LLM stage	Followed the chat default	Own defaults. openai: gpt-6.1-sol at effort medium. anthropic / bedrock: stay on claude-opus-5 / us.anthropic.claude-opus-5 until a SkillSpector release supports Claude 5.5. Install pinned to v2.12.0.
opencode	Followed the chat default	Decoupled; values unchanged (GPT-6.1 Sol has no Chat Completions tool calls)
Agent CLIs	@latest	Claude Code 2.1.292, Codex 0.161.0; codex effort high (unchanged), claude-code medium for Claude ≥ 5.5

Changes

• 1883751 feat(tier1): decouple the SkillSpector model from provider chat defaults and pin v2.12.0
• 68ba452 feat(llm): gate reasoning-model request shape, detect refusals and size budgets per model family
  • One helper covers the reasoning models (gpt-5*/gpt-6*). Those requests send no temperature and use max_completion_tokens.
  • Token budgets: GPT-5/6 at least 32,768; other models 16,384 for verifier, rubric, dedup and judges (8,192 report suggestions, 16,000 dataset generation). On the Anthropic API, Claude Opus 4 and 4.1 are capped at the SDK's 8,192-token non-streaming limit.
  • Per-role effort, applied only to GPT-6 (not on Bedrock) and to Claude ≥ 5.5 on the anthropic provider (including a custom ANTHROPIC_BASE_URL); Bedrock routes keep the model default:
    • verifier high;
    • dedup, rubric and judges medium;
    • never xhigh or max.
  • Refusal and truncation errors are typed, with an Anthropic server-side fallback and, in the Tier 3 judge, a fallback to Opus 4.8. The verifier keeps the complete verdicts of a truncated reply.
  • The verifier's JSON field reasoning is renamed rationale; parsers accept both.
  • RAGAS is skipped for reasoning judges.
  • Provenance records the judge model, effort and whether a fallback was used.
  • Lift excludes mixed-judge cases, including multi-step trials where any step used a fallback judge.
• 585940c feat(tier3): pin Claude Code 2.1.292 and Codex 0.161.0 and set Claude 5.5 effort (local-mode version preflight)
• 6d24e34 feat(gateway): default gateway agents to GPT-6.1 Sol and Claude Opus 5.5
• fff8311 feat(openai): default OpenAI to GPT-6.1 Sol (the last-resort judge is now keyed by provider)
• 7b3431c feat(anthropic): default Anthropic and Bedrock to Claude Opus 5.5
• f0d98cf ci(gitleaks): allowlist the Bedrock model ID in SkillSpector bridge tests
• c3f75a6 docs(provider-config): drop stale default-model wording from effort comments
• 29e2926 fix(llm): keep complete verifier verdicts from a truncated reply
• ef1a715 fix(llm): clamp Anthropic max_tokens to the SDK limit and rebuild schema downgrades per provider
• 402eaa7 fix(provider-config): pin SkillSpector defaults as literals and document effort scope
• c592290 fix(tier3): carry step judge models into multi-step logical rewards
• 749ce18 test(llm): cover medium effort for dedup verdicts and the dimension and insights judges
• a6d3721 refactor(tier3): build the eval-core judge payload from the shared client's helpers

Tests

• Full suite: 11,399 passed, 20 skipped. One timing-sensitive local-mode test failed under parallel load and passes when rerun alone. Two Docker sidecar tests also fail on main in this environment. The 2 failures are tests/test_harbor_secure_docker_environment.py sidecar tests; they need a protected execution state and fail identically on main locally.
• ruff check . clean; git diff --check clean; every commit signed off (DCO); the open-source boundary scan passes.
• New coverage:
  • the reasoning-model helper and its in-container mirror (drift test);
  • per-role effort;
  • refusal and truncation typing, and fallbacks;
  • rationale/reasoning parsing;
  • SkillSpector decoupling, forwarded knobs and the effort default;
  • CLI pins and local preflight;
  • provider-keyed last-resort judge;
  • verifier salvage of a truncated reply;
  • the Anthropic SDK non-streaming token limit (real SDK, mock transport);
  • mixed-judge lift for multi-step trials;
  • medium effort for dedup verdicts and the dimension and insights judges.

Rollout gates (before marking ready)

1. Native OpenAI smoke run on gpt-6.1-sol: judge, SkillSpector stage (medium), verifier (high), dedup (medium).
2. Native Anthropic and Bedrock smoke runs: the refusal surface and claude-code at medium; the Anthropic server-side fallback firing, and on Bedrock the Tier 3 judge's Opus 4.8 retry (host-side Bedrock calls have no refusal fallback).
3. Gateway: one docker and one local trial per agent.
4. Judge bridging on frozen trajectories, new vs previous default, within a 3-run noise band.

Follow-up: once a SkillSpector release supports Claude 5.5 structured output, move SkillSpector's anthropic and bedrock defaults to Opus 5.5 and bump the pin.

Plan: GPT-6.1 Sol and Claude Opus 5.5 defaults

Plan: GPT-6.1 Sol and Claude Opus 5.5 defaults

Model changes by tier

Tier	Use	Current → new	Effort
T1	SkillSpector LLM scan	openai gpt-5.6-sol → gpt-6.1-sol; anthropic claude-opus-5 and bedrock us.anthropic.claude-opus-5 unchanged, now decoupled from the chat default	Sol medium (set by SE); Claude unset
T1 / T2	FindingVerifier; rubric-eval; dedup verdicts	Follow the provider default: openai gpt-5.6-sol → gpt-6.1-sol; anthropic claude-opus-5 → claude-opus-5-5; bedrock us.anthropic.claude-opus-5 → us.anthropic.claude-opus-5-5	verifier high; rubric medium; dedup medium (Bedrock: model default)
T3	codex agent	gpt-5.6-sol → gpt-6.1-sol; gateway openai/openai/gpt-5.6-sol → openai/openai/gpt-6.1-sol	high (existing pin)
T3	claude-code agent	claude-opus-5 → claude-opus-5-5; bedrock us.anthropic.claude-opus-5 → us.anthropic.claude-opus-5-5; gateway aws/anthropic/bedrock-claude-opus-5 → aws/anthropic/bedrock-claude-opus-5-5	medium (pinned)
T3	opencode agent	Decoupled, values unchanged: gateway nvidia/nvidia/nemotron-3-super-120b-long-ctx; native openai openai/gpt-5.6-sol (GPT-6.1 Sol has no tool calls on Chat Completions)	default
T3	Judges, report, dataset generation	Follow the provider default; the last-resort judge becomes provider-keyed instead of gpt-5.6-sol for every provider	medium (Bedrock: model default)
Prov	Gateway chat default	nvidia/nvidia/nemotron-3-super-120b-long-ctx, unchanged	—

Why GPT-6.1 Sol or Opus 5.5

Use	Pick	Deciding reasons
codex agent, OpenAI defaults	GPT-6.1 Sol	Current OpenAI model; Codex 0.161 default; half the price of gpt-5.6-sol ($2/$10 vs $4/$20 per MTok)
claude-code agent, Anthropic and Bedrock defaults	Opus 5.5	Current Opus; Claude Code default; 20% cheaper than Opus 5 per token
SkillSpector on Anthropic and Bedrock	Opus 5 (unchanged)	Opus 5.5 rejects the forced tool call that SkillSpector v2.12.0 sends; moves once a SkillSpector release supports Claude 5.5

Effort settings

Role	Effort	Why
FindingVerifier	high	A wrong false_positive hides a real finding; low volume
SkillSpector (GPT-6.1 Sol), rubric-eval, dedup, judges	medium	Stable scoring; dedup DUPLICATE blocks; many calls per scan
Any Tier 1/2 role	never xhigh/max	Short classification: more latency and truncation, no accuracy evidence
codex / claude-code	high / medium	Pinned so CLI-default changes cannot shift scores
Any Tier 1/2/3 role on Bedrock	model default	SkillEvaluator sends no effort on Bedrock routes

Prerequisites in this PR

Change	Why
Pin Claude Code 2.1.292 and Codex 0.161.0 (local-mode version preflight)	Opus 5.5 needs ≥ 2.1.280, GPT-6.1 Sol needs ≥ 0.159.1
Reasoning-model checks cover gpt-5*/gpt-6*; token budgets: GPT-5/6 at least 32,768, other models 16,384 (capped at the Anthropic SDK's non-streaming limit for Claude Opus 4/4.1)	Otherwise temperature / max_tokens cause 400s and reasoning truncates small caps
Refusal and truncation detection, Opus 4.8 fallback	Classifier refusals otherwise read as empty output
SkillSpector model decoupled from the chat default; install pinned to v2.12.0	A Claude 5.5 chat default would break the SkillSpector scan

Rollout gates (before merge)

1. Native OpenAI smoke run: judge, SkillSpector stage, verifier, dedup on gpt-6.1-sol.
2. Native Anthropic and Bedrock smoke runs: the refusal surface, claude-code at medium, the Anthropic server-side fallback, and the Tier 3 judge's Bedrock retry on Opus 4.8 (host-side Bedrock calls have no refusal fallback).
3. Gateway: one docker and one local trial per agent.
4. Judge bridging on frozen trajectories (new vs previous default) within a 3-run noise band.

Follow-up: SkillSpector anthropic/bedrock scans move to Opus 5.5 after a SkillSpector release supports Claude 5.5 structured output.

🤖 Generated with Claude Code

[rng1995]

Automated review: 9 findings (0 P1, 2 P2, 7 P3); 8 posted inline.

Findings outside the diff or about the description:

• PR_DESCRIPTION: [P3] Description overstates Bedrock effort and fallback coverage and mislabels token budgets

The description says per-role effort is "applied only to gpt-6.x and Claude ≥ 5.5", and the plan table gives Bedrock us.anthropic.claude-opus-5-5 the verifier high and rubric/dedup medium values. In the code, effective_reasoning_effort returns None for every bedrock route, so the new Bedrock default gets no effort in any role. The CHANGELOG states this correctly ("on the Anthropic API"). Rollout gate 2 expects "fallbacks firing" on Bedrock, but host-side Bedrock calls (verifier, rubric, dedup) have no refusal fallback; only the in-container Tier 3 judge has one. "Token ceilings: GPT 32768, Claude 16384" is also inaccurate. For GPT-5/6, 32768 is a floor (max(max_tokens, 32768)). 16384 applies to every non-reasoning model, not only Claude. Dataset generation uses 16000 and report suggestions 8192.

Suggested fix: Say effort goes to GPT-6 (non-Bedrock routes) and to Claude 5.5+ on the anthropic provider, and mark Bedrock as "model default" in the effort table. Limit gate 2's Bedrock fallback check to the Tier 3 judge. Reword the budgets to: "GPT-5/6 at least 32,768; other models 16,384 for verifier, rubric, dedup and judges (8,192 report suggestions, 16,000 dataset generation)."

[rng1995]

Review follow-up for findings outside the diff:

• A truncated reply now discards every verifier verdict, including the complete ones: Addressed in 29e2926. LLMClient.process now parses the partial text of a truncated reply for subclasses that set salvage_truncated_response, and FindingVerifier sets it. last_failure still records the truncation, and the partial-verdict message names the cause. Added OpenAI and Anthropic tests where a finish_reason=length/max_tokens reply with two complete lines and one cut-off line returns two verdicts, and I corrected the llm_judge comment.
• The 16384 default trips the Anthropic SDK's per-model non-streaming guard for Opus 4/4.1: Addressed in ef1a715. The shared client now caps Anthropic max_tokens at the SDK's MODEL_NONSTREAMING_TOKENS entry when one exists (8,192 for Opus 4 and 4.1). That covers the verifier, rubric, judges and dataset generation, which all go through LLMClient. The new test sends claude-opus-4-1-20250805 and claude-opus-4-0 requests through the real SDK with a mock httpx transport, and it fails without the cap. I updated the provider_config comment and the CHANGELOG.
• Effort docstring says native Anthropic only, but any Anthropic base URL gets effort: Addressed in 402eaa7. I kept the existing behaviour (pinned by test_anthropic_gateway_gets_effort_without_server_side_fallback) and reworded the docstring. It now says Claude 5.5+ gets effort on every anthropic endpoint, including a custom ANTHROPIC_BASE_URL, and that Bedrock and OpenAI-compatible Claude routes keep the model default. The llm_judge comment matches.
• SkillSpector's openai and nv_build defaults still alias the chat defaults: Addressed in 402eaa7. Every SKILLSPECTOR_DEFAULT_MODELS entry is now a literal, and test_provider_config asserts the literals. test_skillspector_model_does_not_follow_a_changed_chat_default now covers all five providers.
• Mixed-judge lift exclusion does nothing for multi-step trials without a root reward: Addressed in c592290. _logical_attempt_rewards now carries the union of its steps' judge models per judge metric in a private _judge_models key, which _judge_models_by_case reads through a new _reward_judge_models helper. Added a multi-step test to test_harbor_metrics_truth where one arm's step fell back to Opus 4.8. The case is now excluded and counted, and the test fails without the fix.
• Description overstates Bedrock effort and fallback coverage and mislabels token budgets: Not changed. Updated the PR description. Effort is scoped to GPT-6 off Bedrock and Claude 5.5+ on the anthropic provider, and Bedrock is shown as the model default in the effort tables. Gate 2's Bedrock fallback check is limited to the Tier 3 judge. The token budgets now read 'GPT-5/6 at least 32,768; other models 16,384 ... (8,192 report suggestions, 16,000 dataset generation)'.
• No tests cover effort wiring for dedup and the dimension and insights judges: Addressed in 749ce18. Added a parametrized test that runs the deduplication LLM path (IntraSkillValidator with mocked embeddings and clusters), DimensionJudge and InsightsJudge against a mocked OpenAI SDK on gpt-6.1-sol. It asserts reasoning_effort == 'medium' in the request kwargs.
• Generic schema-fallback helper now special-cases Anthropic's effort key: Addressed in ef1a715. _call_with_schema_fallback now takes a build_kwargs(include_schema) callable, and the Anthropic and OpenAI branches each build their own request with or without the schema. The 'effort' in schema_option block is gone. The existing schema-downgrade effort test still passes.
• _chat_completion_payload has no production caller, so the drift test checks a dead copy: Addressed in a6d3721. llm_judge._chat_completion_payload now composes the client's _token_limit_kwargs, _temperature_kwargs and _build_openai_response_format with effective_reasoning_effort, so the template drift test compares against what LLMClient sends. I removed the duplicate _is_native_openai_chat_url; its canonical and deceptive URL cases now assert the payload's token key.