Introduce experimental provider tier with capability declaration + validate-time cross-checks

[jrob5756]

Summary

Introduce a formal experimental provider tier with a provider capability declaration so that workflows can be statically validated against what each provider actually supports.

Motivated by two in-flight provider PRs (#104 claude-agent-sdk, #235 hermes) that exposed the gap: both delegate parts of the agentic loop to an upstream SDK/framework and therefore can't honor every parity rule in AGENTS.md (streaming events, MCP passthrough, reasoning.effort, structured-output schema, resume, etc.). Rather than rejecting these case-by-case or silently letting parity rules erode, formalize a tier with explicit allowed carve-outs and a static validator that catches workflow ↔ provider mismatches at conductor validate time.

Goals

• Land a small, opinionated framework that feat(providers): add claude-agent-sdk provider #104 can become the first member of, and Feat/hermes provider #235 can be evaluated against.
• Keep the supply-chain blast radius small (per-provider extras, no bulk [experimental] extra).
• Surface mismatches early via conductor validate; surface runtime mismatches via console + JSONL.
• Avoid experimental becoming permanent purgatory: written promotion criteria.

Non-goals (for v1)

• A plugin / entry-point architecture. Tier is metadata, not package structure.
• A runtime --allow-experimental opt-in. Optional extras already gate installation.
• Automatic feature shimming. If a feature isn't supported, surface it — don't silently swallow.
• A separate conductor.providers.experimental namespace. Promotion shouldn't require an import-path change.

Design

1. Provider capability descriptor (applies to ALL providers, not just experimental)

Each provider declares its capabilities so conductor validate and the runtime can cross-check. Capabilities are orthogonal to tier — Copilot/Claude declare them too. Without that, the validator only catches issues in experimental, which is backwards.

Proposed mechanism (TBD during implementation): a ProviderCapabilities Pydantic model attached to each AgentProvider subclass as a class-level CAPABILITIES attribute. The factory exposes capabilities for a configured provider without needing to instantiate it (so validate works without API keys).

Proposed initial capability vocabulary (lock this list before letting providers declare against it):

Capability	Type	Notes
tier	stable | experimental	The tier label itself lives on the descriptor.
mcp_tools	bool	Honors runtime.mcp_servers and workflow tools:.
workflow_tools_passthrough	bool	Per-agent tools: allowlist is enforced (vs. ignored).
streaming_events	bool	Emits agent_message / agent_reasoning incrementally.
agent_reasoning_events	bool	Emits agent_reasoning for thinking content.
reasoning_effort	list[Literal["low","medium","high","xhigh"]] | None	Supported levels; None = not supported.
structured_output	Literal["native","prompt_injection","none"]	How output: schema is enforced.
interrupt	bool	Esc/Ctrl+G cancels mid-call.
max_session_seconds	bool	Per-agent timeout honored.
checkpoint_resume	bool	Session state survives across resume.
usage_tracking	bool	Reports input_tokens / output_tokens / model.
concurrent_safe	bool	Safe to run N copies in parallel groups / for_each.
upstream_pin	str | None	E.g., "hermes-agent==0.15.2". Surfaced in the experimental banner.
maintainer	str | None	E.g., "@external-contributor (best-effort)".

2. conductor validate cross-checks

For every agent in the workflow, validate compares declared workflow features against the configured provider's capabilities. Examples:

• Workflow sets reasoning.effort: high but provider's reasoning_effort doesn't include "high" → error.
• Workflow has mcp_servers: but provider's mcp_tools: false → error.
• Agent declares output: schema and provider's structured_output: "prompt_injection" → warning (works, but flaky).
• Parallel group references a provider with concurrent_safe: false → error.
• Workflow has any tools: allowlist but provider's workflow_tools_passthrough: false → warning (ignored).

Default behavior: validate exits non-zero on errors; warnings print but don't fail. Mirror the existing validator style.

3. Runtime surfacing

• Console banner at run start when any agent uses an experimental provider:

  ⚠ Experimental provider in use: hermes (hermes-agent==0.15.2, maintainer: @external (best-effort))
    Limitations: no streaming events, no MCP, reasoning_effort ignored, structured_output via prompt injection.
    See docs/providers/experimental.md for stability policy.
  

• JSONL event log: add provider_tier (and upstream_pin) to each agent's provider block in workflow_started.system — so dashboards and downstream log tooling can render the badge consistently.
• Web dashboard: render an "experimental" badge on agent nodes that use an experimental provider (consumes the JSONL field — no separate API).
• Per-call warnings for dynamic mismatches that validate can't catch statically (e.g., the selected model within a provider doesn't support the requested reasoning effort). Use the existing console event subscriber path; don't reach for logging.warning (it bypasses Rich formatting — see existing repo convention).

4. Optional-extras install policy

Per-provider extras, not a bulk [experimental] bucket. Keeps each provider's dependency graph isolated and limits supply-chain blast radius (a real concern for any provider wrapping a fast-moving 0.x upstream).

[project.optional-dependencies]
claude-agent-sdk = ["claude-agent-sdk==X.Y.Z"]
hermes = ["hermes-agent==X.Y.Z"]

The provider module imports its upstream lazily so missing extras surface as a clear "install with pip install conductor[hermes]" error from the factory, not an ImportError at module load.

5. AGENTS.md addition

New "Experimental Providers" section that names:

• Allowed carve-outs. Experimental providers MAY waive: streaming events, agent_reasoning events, MCP tool passthrough, workflow tools: allowlist enforcement, reasoning.effort support, native structured output (prompt-injection acceptable), checkpoint_resume, concurrent_safe, max_session_seconds.
• Non-negotiable rules. Experimental providers MUST still uphold: AgentProvider lifecycle (validate_connection / execute / close), AgentOutput shape (even if fields are None), raising real exceptions on real errors (no silent failure), declaring accurate ProviderCapabilities, providing a smoke test (import + construct + dry-run).
• Stability disclaimer. The YAML surface area for an experimental provider may change between minor Conductor releases. Pin Conductor when relying on one.

6. Promotion criteria (written down so the tier doesn't become permanent purgatory)

A provider promotes from experimental → stable when ALL of:

• Full parity capabilities declared (no carve-outs in active use across the test suite).
• Named maintainer with a track record of responding to issues.
• ≥6 months of green CI on a real-API integration test (behind a marker, runs nightly or on release).
• Upstream is ≥1.0 with a stated stability promise (or is a long-stable 0.x with no breaking minor releases for ≥6 months).
• At least one non-trivial workflow in examples/ exercising the provider end-to-end.

7. CI

• Per experimental provider: a smoke test (import + construct + dry-run + lint) that runs on every PR.
• Real-API integration tests gated behind an existing or new pytest marker (real_api or similar), skipped by default.
• The lint/format/typecheck pipeline must cover the new provider files (existing make check workflow should pick this up automatically).

Acceptance criteria

• ProviderCapabilities Pydantic model exists and every existing provider (Copilot, Claude, plus feat(providers): add claude-agent-sdk provider #104 once merged) declares accurate values.
• Factory exposes capabilities for any configured provider name without instantiating it.
• conductor validate cross-checks workflow features against provider capabilities and produces errors/warnings per the table above, with at least the following covered: mcp_tools, reasoning_effort, structured_output, workflow_tools_passthrough, concurrent_safe.
• Console banner prints once at run start for any experimental provider, including upstream_pin and maintainer.
• workflow_started.system JSONL event includes provider_tier and upstream_pin per agent's provider.
• Web dashboard renders an "experimental" badge on agent nodes whose provider is experimental.
• pip install conductor[<provider>] installs each experimental provider's upstream pin; missing-extra path produces a clear error from the factory.
• docs/providers/experimental.md exists: explains the tier, the allowed carve-outs, the promotion criteria, the stability disclaimer.
• AGENTS.md gets an "Experimental Providers" section covering allowed carve-outs, non-negotiable rules, and promotion criteria.
• At least one experimental provider (feat(providers): add claude-agent-sdk provider #104 claude-agent-sdk) declares tier: experimental and the capability set that matches its real behavior.
• At least one workflow in examples/ exercises the experimental provider end-to-end (smoke test in CI, integration test behind a marker).

Out of scope (track separately if pursued)

• Full plugin/entry-point architecture (importlib.metadata providers from 3rd-party packages).
• Conductor-as-a-Hermes-plugin (inverse integration via Hermes's model-provider-plugin / programmatic-integration surface).
• Per-provider --allow-experimental runtime gate.
• Capability negotiation at workflow-load time (e.g., automatically picking a different provider when the configured one lacks a capability).

Sequencing

1. Merge feat(providers): add claude-agent-sdk provider #104 as-is (or after small touch-ups). This becomes the canonical "agent-loop delegated to upstream SDK" experimental provider — narrow, predictable carve-outs.
2. Implement this issue. Reclassify feat(providers): add claude-agent-sdk provider #104 as tier: experimental and declare its accurate capabilities. Backfill capabilities for Copilot/Claude as tier: stable.
3. Re-evaluate Feat/hermes provider #235 against the framework. Hermes-as-stateless-provider may fit cleanly as experimental once carve-outs are formalized; the open design question becomes the concurrent_safe capability and what happens when a parallel group references a concurrent_safe: false provider (validate-time error vs runtime serialization).

Related

• PR feat(providers): add claude-agent-sdk provider #104 — claude-agent-sdk provider (intended first experimental member).
• PR Feat/hermes provider #235 — hermes provider (to be evaluated against the framework once landed).
• AGENTS.md "Provider Parity" section — the rules this issue carves explicit exceptions to.

[jrob5756]

Follow-up actions from PR #104 review

Posting the comprehensive review findings here so they can be tracked and addressed in a follow-up PR after #104 merges.

🔴 Critical (security / correctness / parity)

1. Silent tool over-privileging — tools: [] is dropped; agent silently gets full claude_code preset (filesystem/bash). Security regression. claude_agent_sdk.py:87,106-113
2. MCP servers / temperature / max_tokens silently dropped at factory boundary with no warning. factory.py:97-107
3. agent_turn_start events emitted at wrong points — awaiting_model fires after response arrives; {"turn": N} fires at iteration end. Breaks dashboard spinner and JSONL iteration boundaries (AGENTS.md provider parity rules).
4. Token double-counting — AssistantMessage usage + ResultMessage usage summed, but ResultMessage.usage is the cumulative session total. Reports ~2× actual tokens; breaks cost/budget math.
5. Misleading error suggestion — every non-ProviderError exception gets "Check that the claude CLI is installed", including auth/network/JSON-parse errors. claude_agent_sdk.py:198-201
6. Doc claim "SDK handles retries" is unverified — provider sets is_retryable=False, so workflow-level retries also won't fire. Verify against the SDK and cite, or soften wording in AGENTS.md / docs/providers/comparison.md.

🟠 Important

7. _max_session_seconds is dead code — accepted in constructor, stored, never read. Factory forwards it. Either wire to asyncio.wait_for (mirror claude.py:1040-1044) or drop the param + document in parity notes.
8. except Exception swallows asyncio.CancelledError — breaks structured concurrency and interrupts. Re-raise CancelledError explicitly.
9. is_retryable=False hardcoded on ResultMessage.is_error — Anthropic 429s, 5xx, overload are misclassified as non-retryable. Inspect stop_reason and classify.
10. validate_connection() is a no-op — only re-checks the import already verified in __init__. Should probe for the claude CLI binary (shutil.which) and/or run a trivial connectivity check. Compare to claude.py:265-285.
11. JSON-parse fallback to {"response": ...} when agent.output is declared — declared schema contract silently violated; downstream routes get undefined fields. At minimum logger.warning; ideally raise ValidationError.
12. comparison.md regresses Claude row to "No (Phase 1)" tool support — looks unintentional; out of scope for PR feat(providers): add claude-agent-sdk provider #104. Verify and likely revert.

🟡 Test coverage gaps

• No assertion on output.model field (parity contract)
• No tests for per-agent model / max_agent_iterations overrides
• agent_turn_start tested by set-membership only — would not catch wrong ordering (issue feature: logging redesign #3)
• pending_tools not tested across message boundaries with out-of-order tool results
• is_retryable=False flag never asserted on raised ProviderError
• close() never tested
• tools silently-dropped behavior not pinned by a regression test
• test_execute_interrupt_signal sets event pre-iteration; mid-stream interrupt untested

🟢 Minor / polish

• Missing Google-style docstrings on helpers (_build_output, _process_assistant_blocks, _process_tool_results, etc.)
• Magic numbers [:500] / [:200] / [:150] → named constants (the 500 in _process_tool_results is load-bearing — it truncates what dashboard/JSONL see)
• "text" vs "TextBlock" dual matching needs a one-line comment explaining why both checks exist
• claude-sonnet-4-6 vs claude-sonnet-4-5 default model ID — verify the alias is real
• README install command pin drift (uv add claude-agent-sdk vs >=0.1.0 in source)
• Cargo-culted # noqa: F401 on yield line in test (should be removed or relabeled)
• CLI imports (is_verbose, _file_console) at execute time — wrap in try/except ImportError so CLI coupling doesn't break library use

Connection to this issue

Several of the critical findings (#1, #2, #7, #10) are exactly the kind of silent capability mismatches an experimental tier + capability declaration would surface at validate time rather than at runtime — strong motivation for the work proposed here.