Skip to content

feat(doctor): surface per-model reasoning-effort support and context-window limits - #302

Merged
Jason Robert (jrob5756) merged 2 commits into
mainfrom
feature/301-doctor-model-capabilities
Jul 15, 2026
Merged

feat(doctor): surface per-model reasoning-effort support and context-window limits#302
Jason Robert (jrob5756) merged 2 commits into
mainfrom
feature/301-doctor-model-capabilities

Conversation

@jrob5756

@jrob5756 Jason Robert (jrob5756) commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

Extends conductor doctor --models (and the underlying ProviderDiagnostic / --json output) to surface, per model:

  1. Reasoning-effort support — which reasoning.effort levels a model accepts, plus its default.
  2. Context window — the model's prompt / output / total token limits.

Approach

  • Added an optional AgentProvider.get_model_capabilities(model) hook (mirrors the existing get_max_prompt_tokens / get_model_pricing pattern in providers/base.py; base default returns None, never raises).
  • Copilot: implemented via client.list_models() — reasoning fields live at the top level of Model, not nested under capabilities.
  • Claude: implemented via the existing thinking-model heuristic (is_claude_thinking_model) plus get_max_prompt_tokens; Anthropic exposes no output/total-context split, so those fields stay None.
  • Claude Agent SDK / Hermes / openai-agents: inherit the base None default, rendered as n/a.
  • ProviderDiagnostic.models changed from list[str] to list[ModelDiagnostic] (id + capability fields), populated per-model with per-model failure isolation so one bad model does not drop the rest of the list.
  • conductor doctor --models now renders a separate per-provider Models detail table (Model / Reasoning efforts / Default / Prompt / Output / Context); the Providers table's Models column shows a count instead of raw ids. The JSON models field is now a list of capability objects (not plain id strings — this is a breaking shape change for --json consumers).

Bug fix (found while implementing)

CopilotProvider._validate_reasoning_effort_for_model read capabilities.supported_reasoning_efforts, but the installed SDK (github-copilot-sdk>=1.0.0) exposes that field at the top level of Model. The lookup always returned None, so per-model reasoning-effort validation (including the max-rejection behavior from #299) was silently a no-op against the real SDK. Fixed the read location; updated the test mock shape to match the real SDK.

Testing

  • uv run ruff check src tests / ruff format --check — clean.
  • uv run ty check src — clean (one pre-existing, unrelated warning).
  • uv run pytest -m "not install_scripts" — 3942 passed, 21 skipped, 1 pre-existing failure unrelated to this change (test_copilot_large_write.py, fails identically on main — hits a live backend that rejects an unavailable model).
  • Added/updated tests in test_base.py, test_copilot.py, test_claude.py, test_diagnostics.py, test_doctor.py.

Docs

Updated docs/cli-reference.md (new "Per-model capabilities" section) and CHANGELOG.md (Added + Fixed entries).

Closes #301

Jason Robert (jrob5756) and others added 2 commits July 15, 2026 14:48
@jrob5756
feat(doctor): surface per-model reasoning-effort support and context-…
1bb4e29 
…window limits

Add an optional AgentProvider.get_model_capabilities() hook (mirrors the
existing get_max_prompt_tokens / get_model_pricing pattern) that reports,
per model: which reasoning.effort levels it accepts, its default effort,
and its prompt/output/context-window token limits.

- Copilot implements the hook via client.list_models() (reasoning fields
  live at the top level of Model, not nested under capabilities).
- Claude implements it via the existing thinking-model heuristic plus
  get_max_prompt_tokens (Anthropic exposes no output/total-context split).
- Other providers inherit the base None default, rendered as n/a.

diagnostics.ProviderDiagnostic.models changes from list[str] to
list[ModelDiagnostic] (id + capability fields), populated per-model with
per-model failure isolation so one bad model does not drop the rest.

conductor doctor --models now renders a separate per-provider Models
detail table (Model / Reasoning efforts / Default / Prompt / Output /
Context); the Providers table Models column shows a count instead of
raw ids. The JSON models field is now a list of capability objects.

Also fixes a latent bug found while implementing this: Copilot's
_validate_reasoning_effort_for_model read
capabilities.supported_reasoning_efforts, but the installed SDK exposes
that field at the top level of Model, so the per-model reasoning-effort
check (including max-rejection) was silently a no-op against the real SDK.

Closes #301

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ab369757-28a7-4bee-ab9d-6c7eab2f6543
@jrob5756
fix(doctor): address code review findings for per-model capabilities
bb32166 
- diagnostics._build_model_diagnostics: the per-model try/except only
  wrapped the get_model_capabilities call, not the subsequent attribute
  reads on its result. A malformed capabilities object escaped the guard
  and wiped out every already-built model for that provider. Now the
  whole per-model construction is guarded, and simplified to build
  ModelDiagnostic kwargs from ModelCapabilityInfo.to_dict() instead of
  copying each field by hand.
- claude.get_model_capabilities: split the reasoning-effort heuristic and
  the max_prompt_tokens lookup into independently-guarded try/except
  blocks, so the method upholds "never raise" standalone rather than
  relying on its caller's defensive catch. Also fixed a misleading
  "dashboard use case" reference in the docstring.
- diagnostics.ModelDiagnostic: made frozen, matching its sibling value
  objects (ModelCapabilityInfo, ModelPricing, CredentialEnvVar) with no
  functional cost since it is always built in one step.
- docs/cli-reference.md, CHANGELOG.md: corrected an inaccurate claim that
  claude-agent-sdk / hermes / openai-agents render n/a per field in a
  Models detail table row. They do not implement list_models either, so
  they never get a detail table at all -- only n/a in the Providers
  table summary.
- test_copilot.py: reverted unused kwargs on TestReasoningEffort._make_model
  left over from before the get_model_capabilities tests split into their
  own helper class.
- Added regression tests: mixed success/failure model ordering, a
  malformed capabilities object, supported_reasoning_efforts=[] render
  as "none" in the doctor CLI, and Claude's reasoning fields staying
  populated when the SDK is unavailable or model is not a string.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ab369757-28a7-4bee-ab9d-6c7eab2f6543
@jrob5756
Jason Robert (jrob5756) marked this pull request as ready for review July 15, 2026 20:44
@jrob5756
Jason Robert (jrob5756) merged commit f909478 into main Jul 15, 2026
9 checks passed
@jrob5756
Jason Robert (jrob5756) deleted the feature/301-doctor-model-capabilities branch July 15, 2026 20:45
@jrob5756 Jason Robert (jrob5756) mentioned this pull request Jul 15, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

doctor: surface per-model reasoning-effort support and context-window limits

1 participant

@jrob5756