Skip to content

doctor: surface per-model reasoning-effort support and context-window limits #301

Description

Summary

Extend conductor doctor --models (and the underlying ProviderDiagnostic/--json output) to surface, per model:

  1. Reasoning effort support — which reasoning.effort levels (low/medium/high/xhigh) a model accepts, plus its default.
  2. Context window — the model's prompt/output/total token limits.

Motivation

Today conductor doctor --models only lists model ids (_models_cell in src/conductor/cli/doctor.py). Users configuring reasoning.effort or worried about prompt size have no way to discover per-model limits without hitting a ValidationError at runtime or reading provider source. This data is already fetched internally for validation — it just isn't surfaced to the user.

Current state (investigated)

  • Copilot provider (src/conductor/providers/copilot.py):

    • list_models() already calls client.list_models(), which returns Model objects with:
      • capabilities.supports.reasoning_effort (bool)
      • supported_reasoning_efforts: list[str] | None
      • default_reasoning_effort: str | None
      • capabilities.limits.max_prompt_tokens, max_output_tokens, max_context_window_tokens
    • _validate_reasoning_effort_for_model already reads supported_reasoning_efforts for validation — same data, not exposed to diagnostics.
    • get_max_prompt_tokens() already surfaces max_prompt_tokens for the dashboard context-window bar, but doctor doesn't call it.
  • Claude provider (src/conductor/providers/claude.py):

    • list_models() / get_max_prompt_tokens() use Anthropic's models.list(), which only exposes max_input_tokens — no output/total split, no per-model reasoning-effort list.
    • Reasoning support is a static heuristic: is_claude_thinking_model() (prefix match on claude-3-7-, claude-opus-4, claude-sonnet-4, claude-haiku-4) — a boolean "supports extended thinking", not a level list. All 4 Conductor effort levels map to fixed token budgets (EFFORT_TO_BUDGET_TOKENS in src/conductor/providers/reasoning.py) when supported.
  • Claude Agent SDK provider: declares reasoning_effort: None in CAPABILITIES (no concept at all) — should report "n/a" for both new columns.

Proposed approach

  • Add optional provider hooks alongside the existing get_max_prompt_tokens / get_model_pricing pattern in AgentProvider (src/conductor/providers/base.py), e.g. get_model_capabilities(model) -> ModelCapabilityInfo | None, returning:
    • supported_reasoning_efforts: list[str] | None
    • default_reasoning_effort: str | None
    • max_prompt_tokens, max_output_tokens, max_context_window_tokens: int | None
    • Default implementation returns None (safe degradation to "n/a"), matching the existing optional-hook contract (never raise).
  • Copilot: implement via the existing client.list_models() cache.
  • Claude: implement using is_claude_thinking_model() + the 4 Conductor levels for reasoning, and max_input_tokens for prompt tokens only (no output/total available from the SDK).
  • Claude Agent SDK (experimental, reasoning_effort capability = None): leave unimplemented (base default → "n/a").
  • Extend ProviderDiagnostic (src/conductor/providers/diagnostics.py) and gather_provider()/gather() to populate this per-model when --models is passed.
  • Extend _models_cell / add new columns in src/conductor/cli/doctor.py (_render_providers) for Rich table output, and mirror new fields in DoctorReport.to_dict() for --json.
  • Degrade gracefully to n/a/ wherever a provider or model doesn't report the data — must not fail doctor or affect its exit code.

Out of scope

  • Bumping the github-copilot-sdk pin (currently >=1.0.0, installed 1.0.1, latest stable 1.0.6) — not required since the needed fields already exist in 1.0.1. Tracked separately if desired.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions