Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 55 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -202,11 +202,62 @@ When modifying any provider, check all other providers for the same change. The

#### `claude_agent_sdk.py` parity notes

The Claude Agent SDK provider (`claude_agent_sdk.py`) delegates the agentic loop to the `claude` CLI via the `claude-agent-sdk` package. This achieves **event and output parity** but the following are managed by the SDK rather than Conductor:
The Claude Agent SDK provider (`claude_agent_sdk.py`) is the canonical
**experimental** provider — see the "Experimental Providers" section
below for the carve-out policy. It delegates the agentic loop to the
`claude` CLI via the `claude-agent-sdk` package. This achieves **event
and output parity** but the following are managed by the SDK rather than
Conductor:

- **Retry and error handling**: The `claude-agent-sdk` package does **not** retry API failures (429s, 5xx, network errors) internally — its built-in retry logic covers only filesystem operations. Conductor wraps SDK errors in `ProviderError` and uses `stop_reason` / error subtype to set `is_retryable`, so workflow-level `retry:` configuration drives all retry behavior. Plan for transient failures with explicit `retry:` blocks in your workflow.
- **Tool execution**: Tools and MCP servers are managed by the `claude` CLI's own configuration. The provider rejects workflow-level `runtime.mcp_servers` at the factory and refuses any non-empty per-agent `tools:` list (workflow tool names do not translate to CLI tool IDs). An agent with `tools: []` runs with no tools; omitting `tools:` grants the full `claude_code` preset.
- **Runtime config**: `temperature` and `max_tokens` are rejected at the factory — the CLI controls sampling behavior.

### Experimental Providers

Some providers delegate part of the agentic loop to an upstream SDK or
framework and cannot honor every parity rule above. Rather than reject
them or let parity silently erode, Conductor formalizes an
**experimental tier** with explicit allowed carve-outs and a static
validator that catches workflow ↔ provider mismatches at `conductor
validate` time. See `docs/providers/experimental.md` for the full
stability policy.

**Capability declaration.** Every provider — stable or experimental —
declares a class-level `CAPABILITIES: ProviderCapabilities` attribute
(see `src/conductor/providers/capabilities.py`). The descriptor is a
contract: behavior must match what the provider declares. Lying in the
descriptor undermines the framework.

**Allowed carve-outs** for experimental providers (declared as `False` /
`None` on the descriptor):

- `mcp_tools` — workflow-level `runtime.mcp_servers` is not forwarded
- `workflow_tools_passthrough` — per-agent `tools:` allowlist is not enforced
- `streaming_events` — events emitted only at completion (not incrementally)
- `agent_reasoning_events` — no thinking/reasoning event surfacing
- `reasoning_effort` — provider has no reasoning-effort concept
- `structured_output: "prompt_injection"` — schema enforced via prompt injection only
- `interrupt` — mid-call interrupt not honored (still cancels between iterations)
- `max_session_seconds` — wall-clock session timeout silently ignored
- `checkpoint_resume` — session state does not survive `conductor resume`

**Non-negotiable rules** experimental providers MUST uphold:

- `AgentProvider` lifecycle (`validate_connection` / `execute` / `close`).
- `AgentOutput` shape on every successful execution (fields may be `None`).
- Raise real exceptions on real errors — no silent failure swallowing.
- Declare accurate `ProviderCapabilities` matching observed behavior.
- Provide a smoke test that exercises construct + execute paths against
a mocked SDK.
- Maintain `concurrent_safe: true`, or fail validation when used in
parallel/for_each groups with `max_concurrent > 1`.

**Promotion criteria** (experimental → stable) are documented in
`docs/providers/experimental.md` — full parity capabilities, named
maintainer, real-API integration test, ≥6 months stable upstream,
end-to-end example workflow.

- **Retry and error handling**: The SDK handles retries, backoff, and parse recovery internally. The provider wraps SDK errors in `ProviderError` but does not implement its own retry logic.
- **Tool execution**: Tools and MCP servers are managed by the `claude` CLI's own configuration. Workflow-level `tools` and `runtime.mcp_servers` fields are ignored.
- **Runtime config**: `temperature`, `max_tokens`, and `timeout` are not configurable per-workflow — they are controlled by the CLI.
### Run / Resume Parity

The `run` and `resume` commands must accept the same flags wherever a flag is meaningful for a resumed run. When adding a new flag to `run`, add it to `resume` too unless there's a specific reason it cannot apply.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,7 @@ workflow:
default_model: claude-sonnet-4-6
```

Requires the `claude` CLI to be installed and authenticated. Install the SDK: `uv add claude-agent-sdk`
Requires the `claude` CLI to be installed and authenticated. Install the SDK: `uv add 'claude-agent-sdk>=0.1.0'`

> **Note:** The `claude-agent-sdk` provider delegates tool and MCP server management to the `claude` CLI. Workflow-level `tools` and `runtime.mcp_servers` fields are ignored — configure these through your Claude Code settings instead.

Expand Down
22 changes: 17 additions & 5 deletions docs/providers/comparison.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ This guide helps you choose between GitHub Copilot, Anthropic Claude, and Claude

| Feature | Copilot | Claude | Claude Agent SDK | Winner |
|---------|---------|--------|------------------|--------|
| **Tier** | Stable | Stable | Experimental ([#241](https://github.com/microsoft/conductor/issues/241)) | Copilot / Claude |
| **Context Window** | per-model (SDK-reported) | per-model (SDK-reported) | 200K | Tie |
| **Pricing Model** | Subscription ($10-39/mo) | Pay-per-token | Via Claude Code CLI | Depends |
| **Setup** | GitHub auth | API key | `claude` CLI auth | Copilot (easier) |
Expand All @@ -19,6 +20,14 @@ This guide helps you choose between GitHub Copilot, Anthropic Claude, and Claude
| **Multi-provider** | No | Yes (via Conductor) | No | Claude |
| **Agentic Loop** | SDK-managed | Manual (provider code) | SDK-managed (delegated to CLI) | Depends |

> **About the experimental tier.** `claude-agent-sdk` declares specific
> capability carve-outs (no MCP, no per-agent tools allowlist, no
> reasoning_effort, no checkpoint resume). `conductor validate` catches
> workflows that depend on those features against this provider, and the
> CLI prints a one-time banner when the workflow runs. See
> [docs/providers/experimental.md](./experimental.md) for the stability
> policy and promotion criteria.

## When to Use Copilot

### ✅ Choose Copilot if:
Expand Down Expand Up @@ -128,21 +137,24 @@ agents:
- Settings inherited from your Claude Code environment

3. **You want the SDK to manage the agentic loop**
- Retry logic, tool execution, and structured output handled by the SDK
- Tool execution and structured output handled by the SDK / CLI
- Less provider code to maintain
- Native interrupt support
- **Note:** the SDK does *not* retry transient API errors internally — Conductor classifies SDK errors and surfaces them so workflow-level `retry:` drives recovery.

4. **You need streaming with Claude models**
- Real-time message streaming (unlike the raw Claude provider)
- Typed message objects for each event

### Important: Tools and MCP Servers

The `claude-agent-sdk` provider delegates tool and MCP server management entirely to the `claude` CLI. This means:
The `claude-agent-sdk` provider does not bridge workflow-level tools/MCP into the CLI. Concretely:

- Workflow-level `tools` and `runtime.mcp_servers` fields are **ignored** — configure tools and MCP servers through your Claude Code settings instead
- The full Claude Code toolset (WebSearch, Bash, Read, Write, etc.) is available automatically
- `temperature`, `max_tokens`, and `timeout` are also managed by the CLI and not configurable per-workflow
- `runtime.mcp_servers` — **rejected at the factory** with a clear error. Translation to the CLI's MCP configuration is not implemented. Configure MCP servers through your Claude Code settings instead.
- Per-agent `tools: []` — disables all tools for that agent.
- Per-agent `tools: [list]` — **refused loudly**. Workflow tool names do not translate to Claude CLI tool IDs; silently passing them through would risk granting the wrong native tool.
- Omitting `tools:` entirely — grants the full `claude_code` preset (filesystem, bash, web), matching the bare `claude` CLI experience.
- `temperature` and `max_tokens` are **rejected at the factory** — sampling behavior is controlled by the CLI.

### Example Claude Agent SDK Workflow

Expand Down
106 changes: 106 additions & 0 deletions docs/providers/experimental.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Experimental Providers

Conductor ships **stable** providers that uphold every parity rule in
`AGENTS.md`, and **experimental** providers that delegate part of the
agentic loop to an upstream SDK or framework and therefore cannot honor
every rule. This page documents what "experimental" means, what carve-outs
are allowed, and how a provider moves from experimental to stable.

## Why a separate tier?

Every provider declares a `ProviderCapabilities` descriptor (see
`src/conductor/providers/capabilities.py`). `conductor validate`
cross-checks workflow features against those declarations and surfaces
mismatches before runtime. Experimental providers can declare specific
capabilities as `False` without breaking the validator — but their tier
label is visible everywhere they're used so operators are never surprised
by missing features.

When you run a workflow that uses an experimental provider, the CLI
prints a one-time banner per provider:

```text
┌─────────────────────────────────────────────────────────────────────┐
│ ⚠ Experimental provider in use: claude-agent-sdk │
│ (claude-agent-sdk>=0.1.0) maintained by @lesandiz (best-effort) │
│ Limitations: no MCP servers, no per-agent tools allowlist, │
│ reasoning_effort ignored, no checkpoint resume. │
│ See docs/providers/experimental.md for stability policy. │
└─────────────────────────────────────────────────────────────────────┘
```

The web dashboard surfaces the same information as an **exp** badge on
every agent node whose resolved provider has `tier: experimental`.

## Allowed carve-outs

An experimental provider MAY declare any of the following capabilities
as `False` / `None`. Each carve-out is surfaced via the banner and the
validator so the operator can plan accordingly.

| Capability | Carve-out meaning |
|---|---|
| `mcp_tools` | Provider does not forward `runtime.mcp_servers`. Workflows that declare MCP servers against this provider fail validation. |
| `workflow_tools_passthrough` | Provider does not honor per-agent `tools:` allowlists. Workflows that declare a non-empty allowlist against this provider fail validation. |
| `streaming_events` | Provider emits events only at completion (not incrementally). |
| `agent_reasoning_events` | Provider does not surface thinking/reasoning content. |
| `reasoning_effort` | Provider has no reasoning-effort concept; an agent declaring `reasoning.effort: <level>` fails validation. |
| `structured_output: "prompt_injection"` | Schema is enforced via prompt injection rather than a native JSON mode. Validation emits a warning (not an error) for experimental providers; stable providers are silent. |
| `interrupt` | Provider does not monitor `interrupt_signal`. Esc/Ctrl+G still aborts at iteration boundaries but cannot return partial output mid-call. |
| `max_session_seconds` | Provider does not enforce a wall-clock session timeout. Agents that set `max_session_seconds` fail validation. |
| `checkpoint_resume` | Provider session state does not survive `conductor resume` (re-runs the agent from scratch). |

## Non-negotiable rules

Experimental tier does NOT exempt a provider from:

- The `AgentProvider` lifecycle: `validate_connection()`, `execute()`,
`close()`.
- Returning an `AgentOutput` of the expected shape (even when individual
fields like `model` or token counts are `None`).
- Raising real exceptions on real failures — no silent error swallowing.
- Declaring **accurate** `ProviderCapabilities`. Lying in the descriptor
defeats the whole framework. If behavior cannot be honored under all
conditions, declare the weaker capability value.
- Providing a smoke test (`tests/test_providers/test_<name>.py`) that
exercises construct + execute paths against a mocked SDK.
- Maintaining `concurrent_safe: true` *or* failing validation when used
in parallel/for_each groups with `max_concurrent > 1`.

## Promotion criteria: experimental → stable

To prevent the tier from becoming permanent purgatory, every promotion
requires ALL of:

1. Full parity capabilities declared — no carve-outs in active use across
the test suite.
2. Named maintainer with a track record of responding to issues.
3. ≥6 months of green CI on a real-API integration test (behind a
pytest marker, run nightly or on release).
4. Upstream is ≥1.0 with a stated stability promise, or is a long-stable
0.x with no breaking minor releases for ≥6 months.
5. At least one non-trivial workflow in `examples/` that exercises the
provider end-to-end.
6. AGENTS.md "Experimental Providers" section updated to remove the
provider from the experimental table.

## Stability disclaimer

The YAML surface area for an experimental provider may change between
minor Conductor releases. Pin Conductor when relying on one.

Optional-dependency extras (`pip install conductor[<provider>]`) isolate
each experimental provider's upstream dependency graph so that
adopting one does not inflate the install surface for others.

## Current experimental providers

| Provider | Upstream pin | Maintainer | Capability carve-outs |
|---|---|---|---|
| `claude-agent-sdk` | `claude-agent-sdk>=0.1.0` | `@lesandiz (best-effort)` | no `mcp_tools`, no `workflow_tools_passthrough`, no `reasoning_effort`, `prompt_injection` structured output, no `checkpoint_resume` |

## See also

- `AGENTS.md` — "Provider Parity" section (the rules experimental providers carve out from) and "Experimental Providers" section (rules they must still uphold)
- `src/conductor/providers/capabilities.py` — `ProviderCapabilities` schema
- Issue [#241](https://github.com/microsoft/conductor/issues/241) — design rationale
105 changes: 105 additions & 0 deletions examples/experimental-claude-agent-sdk.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# Experimental Provider Showcase: claude-agent-sdk
#
# This example demonstrates the **experimental** claude-agent-sdk provider
# end-to-end. The provider delegates the agentic loop to the `claude` CLI
# via the `claude-agent-sdk` package — see docs/providers/experimental.md
# for the stability policy and full list of capability carve-outs.
#
# When you run this workflow, you will see a one-time banner in the console
# (and an "exp" badge on each agent node in the web dashboard) advertising
# the experimental tier. That is by design — it surfaces tier metadata so
# operators are never surprised by missing features.
#
# What this example exercises:
# - Provider tier surfacing (banner + dashboard badge)
# - Structured output via the SDK's prompt-injection JSON schema
# - Per-agent `tools: []` (the only safe per-agent allowlist for this
# provider — see #241 / A1 for why non-empty lists are refused)
# - `max_session_seconds` enforcement (now wired — see #241 / A7)
#
# Pre-requisites:
# pip install conductor[claude-agent-sdk]
# # Or with uv:
# uv add 'claude-agent-sdk>=0.1.0'
# # Plus the `claude` CLI:
# npm install -g @anthropic-ai/claude-code
# claude login
#
# Usage:
# conductor run examples/experimental-claude-agent-sdk.yaml \
# --input topic="multi-agent workflow orchestration"
#
# Validation only (no execution / no API calls):
# conductor validate examples/experimental-claude-agent-sdk.yaml

workflow:
name: experimental-claude-agent-sdk-showcase
description: >
End-to-end smoke example for the experimental claude-agent-sdk
provider. Two LLM agents (analyze + summarize) chained linearly,
each with a declared output schema and a session timeout.
version: "1.0.0"
entry_point: analyze

runtime:
provider: claude-agent-sdk
# claude-sonnet-4-5 is the SDK's documented default; adjust to your
# Claude Code backend (Vertex AI / Bedrock / anthropic.com).
default_model: claude-sonnet-4-5

input:
topic:
type: string
required: true
description: The topic to analyze and summarize.

agents:
- name: analyze
description: Analyze the topic and produce structured findings.
prompt: |
Analyze the following topic and produce:
1. A short 1-2 sentence analysis of why it matters
2. Three concrete facets that anyone learning about it should know

Topic: {{ workflow.input.topic }}
# Empty tools list = no tools available to this agent. Required for
# the security boundary documented in #241 / A1: non-empty lists are
# refused loudly because workflow tool names do not translate to
# Claude CLI tool IDs.
tools: []
# Wall-clock timeout for the entire agent execution. Honored by the
# provider per #241 / A7.
max_session_seconds: 120
output:
analysis:
type: string
description: One- or two-sentence analysis.
facets:
type: array
description: Three concrete facets to know.
items:
type: string
routes:
- to: summarize

- name: summarize
description: Condense the analyzer's output into a tight one-paragraph summary.
prompt: |
Given this analysis and the facets, write a single tight paragraph
(3-4 sentences) suitable for a TL;DR. Be specific, not generic.

Analysis: {{ analyze.output.analysis }}
Facets: {% for f in analyze.output.facets %}
- {{ f }}{% endfor %}
tools: []
max_session_seconds: 60
output:
summary:
type: string
description: One-paragraph TL;DR.
routes:
- to: $end

output:
analysis: "{{ analyze.output.analysis }}"
summary: "{{ summarize.output.summary }}"
Loading
Loading