From e9f06a5184433a52daaf6dceb35e7be8714873e8 Mon Sep 17 00:00:00 2001 From: Chang Liu Date: Wed, 6 May 2026 15:27:22 -0400 Subject: [PATCH] fix: customer-readiness audit before private preview Removes internal-only references, normalizes setup commands across docs, fixes stale YAML examples, and consolidates the doc set so customers see a coherent, runnable surface. Sensitive content removal - Delete docs/private-preview/access-and-repo-workflow.md (internal GBB/EMU/LT/TAM workflow guide). - Strip Microsoft-internal commentary from examples/travel_planner_langgraph/eval_config.yaml (Azure inference revenue, MSR systematization science, no-other-eval-tool claims). - Rename examples/phoenix_auto_trace/eval_framework.yaml -> eval_framework_template.yaml and add a TEMPLATE-ONLY header banner (it had a REPLACE_WITH_FRAMEWORK placeholder that crashed at runtime). - Drop GBB-specific phrasing from README.md and docs/status-and-roadmap.md. - Rewrite AGENTS.md prohibitions generically (drop TAM / LT review / PM planning enumeration). Setup normalization - Switch quickstart, CLI reference, examples README, and pipes README from 'uv venv / uv sync / uv run' to 'pip install -e .[otel,langgraph]' with venv activation, matching the README's canonical flow. - docs/quickstart.md: clearly mark Phoenix as optional in a second terminal so customers don't think 'phoenix serve' hangs the pipeline. - README: collapse macOS litellm caveat into a Troubleshooting section at the bottom and add Windows + Docker preflight notes there. Doc consolidation - Delete docs/glossary.md (terminology bridge folded into docs/concepts.md). - Delete docs/ai-agent-onboarding.md (paste-in prompt folded into AGENTS.md as a 'Paste-in prompt for end users' section). - Delete docs/targets/otel-agent.md (trace-capture content inlined into docs/targets/callable.md as 'Optional: add trace capture for richer evidence' + 'Why trace capture matters'). - Update all references across README, AGENTS, model-and-tools.md. Stale YAML examples (CONFIG_REFERENCE-aligned) - docs/writing-eval-specs.md: nest 'judge:' under 'pipeline:'. - examples/concepts/README.md: 'concept: foo' -> 'concept.name: foo'. - examples/phoenix_auto_trace/README.md: nest 'rollout:' under 'pipeline:'. - CONFIG_REFERENCE.md: minimal example needs target.model (was missing, violated 'exactly one of model/callable/endpoint'). Customer-facing terminology and reach - Reframe 'OTel-traced agents' to 'any agent or multi-agent system' across README, callable.md, and AGENTS.md. OTel trace capture is the optional upgrade, not a prerequisite. - examples/agents/README.md: 'p2m' -> 'Adaptive Eval'; demote external connector to 'advanced' (recommend target.callable instead). - examples/phoenix_auto_trace/README.md: framework count 28 -> 33 with honest note that 28 are demoed and the rest follow the same pattern; soften 'proves the spec's core claim' / 'zero per-framework maintenance' wording. Windows reliability - examples/phoenix_auto_trace/_tools.py: replace -> with -> in mock flight routes that get printed to console (avoids UnicodeEncodeError on default Windows codepages). - examples/phoenix_auto_trace/README.md: replace -> arrows with -> and document PYTHONUTF8=1 escape hatch. Verification - 505 unit tests pass. - All 7 example pipeline configs validate via load_config. - examples.phoenix_auto_trace._tools imports and simulate_tool() works. - 'rg glossary.md|ai-agent-onboarding.md|otel-agent.md|access-and-repo-workflow.md' returns no matches across the repo. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- AGENTS.md | 31 ++++++- CONFIG_REFERENCE.md | 6 +- README.md | 34 +++---- docs/ai-agent-onboarding.md | 51 ---------- docs/glossary.md | 39 -------- .../access-and-repo-workflow.md | 85 ----------------- docs/quickstart.md | 27 ++++-- docs/reference/cli.md | 18 ++-- docs/status-and-roadmap.md | 2 +- docs/targets/callable.md | 36 +++++++- docs/targets/model-and-tools.md | 2 +- docs/targets/otel-agent.md | 92 ------------------- docs/writing-eval-specs.md | 17 ++-- examples/README.md | 10 +- examples/agents/README.md | 38 +++++--- examples/concepts/README.md | 2 +- examples/phoenix_auto_trace/README.md | 44 ++++----- examples/phoenix_auto_trace/_tools.py | 8 +- ...work.yaml => eval_framework_template.yaml} | 30 +++--- examples/pipes/README.md | 11 ++- .../travel_planner_langgraph/eval_config.yaml | 13 +-- 21 files changed, 211 insertions(+), 385 deletions(-) delete mode 100644 docs/ai-agent-onboarding.md delete mode 100644 docs/glossary.md delete mode 100644 docs/private-preview/access-and-repo-workflow.md delete mode 100644 docs/targets/otel-agent.md rename examples/phoenix_auto_trace/{eval_framework.yaml => eval_framework_template.yaml} (50%) diff --git a/AGENTS.md b/AGENTS.md index b87af6cf7..bd45bc1ea 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -17,7 +17,7 @@ eval spec -> behavior categories -> test cases -> execute target -> judge -> art - Never read, print, commit, summarize, or infer values from `.env` or other local environment files. - Use placeholder names such as `AZURE_API_KEY` and `AZURE_API_BASE`; never invent or expose credential values. - Do not recommend committing generated artifacts, local traces, `.venv`, logs, or `.env` files. -- Treat this repo as a customer-preview distribution repo. Avoid internal strategy, roadmap, TAM, LT review, or PM planning content. +- Treat this repo as a customer-preview distribution. Keep contributions customer-safe — avoid any internal-only planning, prioritization, or organizational content. ## Authoritative files @@ -26,8 +26,7 @@ Start with these files: - `README.md` - customer-facing overview and quickstart. - `docs/quickstart.md` - LangGraph travel planner walkthrough. - `docs/targets/overview.md` - target decision tree. -- `docs/targets/otel-agent.md` - callable agent target with optional trace capture (any agent or multi-agent system). -- `docs/targets/callable.md` - Python callable target (final-text only, the simplest path). +- `docs/targets/callable.md` - Python callable target for any agent or multi-agent system, with an optional OpenTelemetry trace-capture upgrade. - `docs/targets/model-and-tools.md` - hosted model and simple model+tools target. - `CONFIG_REFERENCE.md` - current YAML schema reference. - `examples/README.md` - example selection guide. @@ -123,7 +122,31 @@ Keep docs customer-safe. Prefer improving: - `docs/reading-results.md` - `examples/README.md` -Do not reintroduce internal planning docs into this private-preview distribution branch. +Do not reintroduce internal-only planning docs into this customer-preview distribution. + +## Paste-in prompt for end users + +End users can paste the following block into their AI assistant to get the same orientation this file gives you: + +```text +You are helping me with the Adaptive Eval repo (https://github.com/microsoft/adaptive-eval). + +Adaptive Eval is a local-first, spec-driven evaluation pipeline for AI agents. The mental model: + + eval spec -> behavior categories -> test cases -> execute target -> judge -> artifacts + +Key facts: +- The CLI entrypoint is `p2m`. Configs live in `examples/`. Artifacts land in `artifacts/results///`. +- For any agent or multi-agent system with a Python entry function, use `target.callable`. + OpenTelemetry trace capture (Phoenix/OpenInference) is an optional upgrade — not required. +- For a hosted model with a system prompt and optional tools, use `target.model` and `target.tools`. +- Read `README.md`, `docs/quickstart.md`, `docs/targets/overview.md`, `docs/targets/callable.md`, and + `CONFIG_REFERENCE.md` before suggesting changes to YAML schema. +- Never read or print values from `.env`. Use placeholder names like AZURE_API_KEY and AZURE_API_BASE. +- Keep all suggestions customer-safe. + +When I ask for help, prefer concrete file paths, runnable commands, and the YAML keys defined in CONFIG_REFERENCE.md. +``` ## Output style for coding agents diff --git a/CONFIG_REFERENCE.md b/CONFIG_REFERENCE.md index 3b2d47134..2fda93231 100644 --- a/CONFIG_REFERENCE.md +++ b/CONFIG_REFERENCE.md @@ -196,8 +196,6 @@ Accepted keys: For customer-preview configs, `target` must define exactly one of `model`, `callable`, or `endpoint`. -`target.tools` is valid only with `target.model`. It may define `module`, `toolset + simulator`, or `simulator` alone. `toolset` requires `simulator`. If you omit `target.system_prompt`, rollout uses each seed's `system_prompt` when present. Scenario seeds require `auditor`. Prompt seeds do not. - Callable agent example with optional OTel trace capture: ```yaml @@ -414,11 +412,15 @@ pipeline: sample_size: 10 rollout: target: + model: + name: azure/gpt-5.4-mini system_prompt: | You are a health assistant. judge: {} ``` +`target.tools` is valid only with `target.model`. It may define `module`, `toolset + simulator`, or `simulator` alone. `toolset` requires `simulator`. If you omit `target.system_prompt`, rollout uses each seed's `system_prompt` when present. Scenario seeds require `auditor`. Prompt seeds do not. + `concept.md` next to that file: ```md diff --git a/README.md b/README.md index 1033ea8ed..26178e2b7 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ **Spec-driven evaluation for any agent or multi-agent system - local-first, framework-agnostic, and trace-aware.** -> **Customer preview.** Adaptive Eval is a preview / POC for design-partner and GBB engagements. The core workflow is stable: write an eval spec, generate targeted test cases, execute them against your agent, and judge the results against your rubric. Some YAML field names are still evolving; the docs bridge current names to the intended developer-facing terminology. +> **Customer preview.** Adaptive Eval is a preview / POC distribution for design partners. The core workflow is stable: write an eval spec, generate targeted test cases, execute them against your agent, and judge the results against your rubric. Some YAML field names are still evolving; the docs bridge current names to the intended developer-facing terminology. ## Why Adaptive Eval @@ -59,18 +59,6 @@ p2m run --config examples/travel_planner_langgraph/eval_config.yaml p2m results status travel-planner-langgraph-v1 demo-1 ``` -### macOS install note - -Use the `pip install -e ".[otel,langgraph]"` path above as the primary preview install path on macOS. - -We have seen macOS security tooling silently block `uv sync` from extracting several large files from the `litellm` wheel. When that happens, Python imports `litellm` as an empty namespace package and later fails with errors such as `AttributeError: module 'litellm' has no attribute 'acompletion'`. `pip` uses a copy-based install path and avoids this issue. - -If you still prefer `uv`, and `litellm` imports without expected attributes, try granting your terminal Full Disk Access and clearing quarantine attributes on the environment: - -```bash -xattr -cr .venv -``` - What the quickstart does: | Step | Developer concept | Current YAML / artifact | @@ -96,7 +84,7 @@ behavior categories -> test cases + variations -> execute target -> judge + OTel traces metrics.json ``` -Today the YAML still uses implementation names such as `concept`, `factors`, `policy`, `seeds`, and `rollout`. The docs use the developer-facing concepts - spec, variations, test cases, execute, judge - and call out the current YAML key the first time each concept appears. See [`docs/glossary.md`](docs/glossary.md). +Today the YAML still uses implementation names such as `concept`, `factors`, `policy`, `seeds`, and `rollout`. The docs use the developer-facing concepts - spec, variations, test cases, execute, judge - and call out the current YAML key the first time each concept appears. See [`docs/concepts.md`](docs/concepts.md) for the bridge. ## Choose your target @@ -104,8 +92,7 @@ Pick a target based on how your agent is built. | Your target looks like... | Use this path | Trace fidelity | Start here | |---|---|---|---| -| Any agent or multi-agent system you can invoke from Python (LangGraph, CrewAI, OpenAI Agents SDK, DSPy, LlamaIndex, AutoGen / MAF, custom orchestration, …) | **Callable agent target**: point `target.callable` at your entry function. Optionally add Phoenix/OpenInference instrumentation and `target.trace` for richer judge evidence. | Final text out of the box; tool calls, arguments, routing, model calls, and latency when you opt into OTel trace capture | [`docs/targets/otel-agent.md`](docs/targets/otel-agent.md) | -| A Python function that accepts a user message and returns a string or model response | **Plain callable target**: `target.callable: package.module:function` | Inputs/outputs, plus structured tool/model metadata if your callable returns it | [`docs/targets/callable.md`](docs/targets/callable.md) | +| Any agent or multi-agent system you can invoke from Python (LangGraph, CrewAI, OpenAI Agents SDK, DSPy, LlamaIndex, AutoGen / MAF, custom orchestration, …) — or a plain Python function wrapping a model | **Callable target**: point `target.callable` at your entry function. Optionally add Phoenix/OpenInference instrumentation and `target.trace` for richer judge evidence. | Final text out of the box; tool calls, arguments, routing, model calls, and latency when you opt into OTel trace capture | [`docs/targets/callable.md`](docs/targets/callable.md) | | A hosted model with a system prompt, optionally with tools | **Model + tools target**: `target.model`, `target.system_prompt`, and optional `target.tools` | Conversation transcript and tool traces for simple prompt-agent setups | [`docs/targets/model-and-tools.md`](docs/targets/model-and-tools.md) | **Recommended for best eval results:** add OTel trace capture to your callable when your agent has meaningful internals. Otherwise, the judge is mostly evaluating final text and may miss tool calls, routing decisions, dynamic DAG behavior, and framework internals. @@ -154,12 +141,12 @@ Browse them with the CLI, the local viewer, or any JSONL tool. Nothing leaves yo ## Documentation map -- **Get started:** [`docs/quickstart.md`](docs/quickstart.md), [`docs/concepts.md`](docs/concepts.md), [`docs/glossary.md`](docs/glossary.md) -- **Targets:** [`docs/targets/overview.md`](docs/targets/overview.md), [`docs/targets/otel-agent.md`](docs/targets/otel-agent.md) (callable agent target), [`docs/targets/callable.md`](docs/targets/callable.md), [`docs/targets/model-and-tools.md`](docs/targets/model-and-tools.md) +- **Get started:** [`docs/quickstart.md`](docs/quickstart.md), [`docs/concepts.md`](docs/concepts.md) +- **Targets:** [`docs/targets/overview.md`](docs/targets/overview.md), [`docs/targets/callable.md`](docs/targets/callable.md) (any agent), [`docs/targets/model-and-tools.md`](docs/targets/model-and-tools.md) - **Authoring:** [`docs/writing-eval-specs.md`](docs/writing-eval-specs.md), [`docs/reading-results.md`](docs/reading-results.md) - **Reference:** [`docs/reference/cli.md`](docs/reference/cli.md), [`CONFIG_REFERENCE.md`](CONFIG_REFERENCE.md) -- **AI assistants:** [`AGENTS.md`](AGENTS.md), [`docs/ai-agent-onboarding.md`](docs/ai-agent-onboarding.md) -- **Preview operations:** [`docs/private-preview/access-and-repo-workflow.md`](docs/private-preview/access-and-repo-workflow.md), [`docs/status-and-roadmap.md`](docs/status-and-roadmap.md) +- **AI assistants:** [`AGENTS.md`](AGENTS.md) +- **Preview status:** [`docs/status-and-roadmap.md`](docs/status-and-roadmap.md) ## Status @@ -180,3 +167,10 @@ Still evolving: - framework-specific quickstarts beyond the current examples Preview feedback is welcome: confusing names, missing target examples, trace gaps, judge behavior, artifact shape, and docs clarity are all useful signals. + +## Troubleshooting + +- **macOS, `litellm` AttributeError after install** — some macOS security tooling can silently truncate the `litellm` wheel during extraction with `uv sync`, causing errors like `AttributeError: module 'litellm' has no attribute 'acompletion'`. The `pip install -e ".[otel,langgraph]"` path above uses copy-based installs and avoids this. If you must use `uv`, grant your terminal Full Disk Access and run `xattr -cr .venv` to clear quarantine attributes. +- **Windows, `UnicodeEncodeError` when running auto-trace demos** — set `$env:PYTHONUTF8 = "1"` before `python -m examples.phoenix_auto_trace.travel_openai`. +- **Docker-backed pipes fail with "docker daemon unavailable"** — `examples/pipes/health_assistant_sandbox.yaml` and `_external.yaml` need Docker Desktop running. + diff --git a/docs/ai-agent-onboarding.md b/docs/ai-agent-onboarding.md deleted file mode 100644 index 77a20ee97..000000000 --- a/docs/ai-agent-onboarding.md +++ /dev/null @@ -1,51 +0,0 @@ -# Paste-in Prompt for Coding Agents - -Use this prompt when asking GitHub Copilot, Claude Code, Cursor, or another coding agent to help you add Adaptive Eval to your own agent project. - -```text -You are helping me use Adaptive Eval, a local-first spec-driven evaluation harness for AI agents. - -Goal: -- Help me write an eval spec for my agent. -- Help me choose the right target configuration. -- Help me run the eval and interpret the artifacts. - -Mental model: -eval spec -> behavior categories -> test cases -> execute target -> judge -> artifacts - -Current YAML names: -- eval spec: concept.name -- target description: context -- variations: factors -- behavior categories: pipeline.policy -> policy.json -- test cases: pipeline.seeds -> seeds.jsonl -- execute: pipeline.rollout -> transcripts.jsonl -- judge: pipeline.judge -> scores.jsonl and metrics.json - -Target selection: -1. If my agent or multi-agent system has any Python entry function (frameworks such as LangGraph, LangChain, OpenAI Agents SDK, CrewAI, LlamaIndex, AutoGen/MAF, DSPy — or custom orchestration with no framework), use target.callable. Trace capture using Phoenix/OpenInference is an optional upgrade for richer judge evidence. -2. If my target is a plain Python function that wraps a hosted model, use target.callable. -3. If my target is only a hosted model with a system prompt, use target.model. -4. If I have a prompt agent with tool schemas but no real tool backend yet, simulated tools can help, but they are not a replacement for evaluating a real agent or multi-agent system. - -I do not need to understand OpenTelemetry to start. Trace capture is opt-in. - -Setup preference: -- Prefer pip install for preview setup, especially on macOS: - python -m venv .venv - source .venv/bin/activate - python -m pip install --upgrade pip - python -m pip install -e ".[otel,langgraph]" - -Security: -- Never read, print, commit, or infer values from .env or other local environment files. -- Use placeholder environment variable names such as AZURE_API_KEY and AZURE_API_BASE. -- Do not commit generated artifacts, traces, logs, or virtual environments. - -Ask me these questions first: -1. What runtime does my agent or multi-agent system use? (framework, custom orchestration, or plain hosted model) -2. What function or endpoint should Adaptive Eval call? -3. What behavior requirements should the eval spec test? -4. Optional: do I want trace capture so the judge can inspect tool calls, routing, dynamic DAG behavior, or framework internals? (Skippable on the first run.) -5. What model should be used for generation and judging? -``` diff --git a/docs/glossary.md b/docs/glossary.md deleted file mode 100644 index 280a29bd0..000000000 --- a/docs/glossary.md +++ /dev/null @@ -1,39 +0,0 @@ -# Glossary - -Adaptive Eval is migrating from research-oriented names to developer-friendly names. The conceptual workflow is stable; some YAML keys still use current implementation names. - -| Developer concept | Current YAML / artifact | Meaning | -|---|---|---| -| Eval spec | `concept.name`, `concept.md`, `.md` | The plain-English requirements you want to evaluate. | -| About the target | `context` | Description of the agent, tools, domain, and important constraints. | -| Variations | `factors` | Axes of coverage the test generator should vary across. | -| Behavior categories | `pipeline.policy`, `policy.json` | Structured taxonomy generated from the eval spec. | -| Test cases | `pipeline.seeds`, `seeds.jsonl` | Generated single-turn prompts and multi-turn scenarios. | -| Execute | `pipeline.rollout`, `transcripts.jsonl` | Run test cases against the target and record conversations. | -| Target | `pipeline.rollout.target` | The model, callable, or agent being evaluated. | -| OTel trace | `target.trace` | OpenTelemetry spans captured from the running agent. | -| Judge | `pipeline.judge`, `scores.jsonl` | LLM scoring against the spec and rubrics. | -| Metrics | `metrics.json` | Aggregate rates by dimension and behavior category. | -| Dataset / suite | `suite` | Group of shared artifacts for a set of runs. | -| Run | `run` | One execution of rollout and judge against a target/config snapshot. | - -## Recommended language in docs - -Use: - -- spec -- variations -- test cases -- execute -- judge -- artifacts -- target - -Avoid using the older terms as the primary explanation: - -- risk -- sub-risk -- seeds as the customer-facing noun -- rollout as the customer-facing verb - -When discussing YAML, use the exact current key and explain it once. diff --git a/docs/private-preview/access-and-repo-workflow.md b/docs/private-preview/access-and-repo-workflow.md deleted file mode 100644 index 79e04e1bc..000000000 --- a/docs/private-preview/access-and-repo-workflow.md +++ /dev/null @@ -1,85 +0,0 @@ -# Private Preview Access and Repository Workflow - -This page is for maintainers preparing a private preview with customers and GBBs. - -## Access model - -GitHub read access is repo-scoped, not branch-scoped. A user with Read access can clone the repo, branches, tags, and visible history. Branch protection controls writes; it does not hide branches. - -Therefore, do not use branches as the customer isolation boundary. Use a separate private preview repo that contains only customer-safe content. - -| Audience | Repository | Permission | -|---|---|---| -| 3P customers | Private preview repo | Read | -| GBB readers | Private preview repo | Read | -| GBB contributors | Private preview repo | Prefer fork + PR; otherwise Write with branch protection | -| Engineering team | EMU/internal repo | Write/Maintain/Admin | - -## What belongs in the preview repo - -- README -- quickstart docs -- customer-safe target docs -- examples intended for preview -- config reference -- contribution instructions - -## What stays internal - -- engineering design debates -- LT review material -- TAM and OSS escalation docs -- roadmap and backlog -- private customer notes -- science comparison drafts -- security/threat-modeling details not intended for customers - -## Repo workflow - -Use the EMU/internal repo as the engineering source of truth. Promote only sanitized commits to the private preview repo. - -Recommended flow: - -```powershell -# Internal branch and PR first. -git switch -c docs/customer-preview origin/main -git push -u origin docs/customer-preview - -# Then promote safe commits to the preview repo. -git fetch preview -git switch -c preview/docs-customer-preview preview/main -git cherry-pick -git diff --name-only preview/main...HEAD -git push -u preview preview/docs-customer-preview -``` - -Open a PR into the preview repo. Do not merge the whole internal branch or history into the preview repo. - -## Branch protection - -Protect `main` in the preview repo: - -- require pull requests -- require at least one internal reviewer -- block force-push -- block branch deletion -- require status checks when available -- add CODEOWNERS for sensitive paths - -## GBB PRs - -Preferred: GBB contributors fork the preview repo and open PRs back. This avoids granting direct write access. - -If Write access is necessary, keep `main` protected and require internal review before merge. - -## EMU alias-resolution bug - -If an engineer cannot be added to the EMU repo because of a GitHub alias-resolution issue, treat that as an internal identity/access problem. - -Do not copy private EMU-only docs into the preview repo as a workaround. Instead: - -1. fix org/team membership through the appropriate GitHub or identity admin path; -2. temporarily share internal docs through a Microsoft-controlled internal location; or -3. create a separate internal docs repo with the correct audience. - -The private preview repo should remain safe for customers to clone. diff --git a/docs/quickstart.md b/docs/quickstart.md index a26e14861..5ffe28052 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -29,24 +29,35 @@ target: ## Prerequisites - Python 3.11+ -- `uv` +- `pip` (uv also works for contributors; see the README install path) - Azure OpenAI credentials in `.env` - Optional: Phoenix running locally if you want to browse traces during the run ## Run it +> **Setup is the same `pip install -e ".[otel,langgraph]"` flow shown in the [README](../README.md#quickstart-langgraph-travel-planner-any-agent-works-the-same-way).** This page focuses on what you do after setup. + ```powershell -uv venv -uv sync +# (one-time setup, see README for full details) +python -m venv .venv +.\.venv\Scripts\Activate.ps1 +python -m pip install --upgrade pip +python -m pip install -e ".[otel,langgraph]" Copy-Item .env.example .env # Edit .env with AZURE_API_KEY, AZURE_API_BASE, and any deployment settings. -uv run phoenix serve -uv run p2m run --config examples\travel_planner_langgraph\eval_config.yaml -uv run p2m results status travel-planner-langgraph-v1 demo-1 +# Run the pipeline +p2m run --config examples\travel_planner_langgraph\eval_config.yaml +p2m results status travel-planner-langgraph-v1 demo-1 ``` -If you skip Phoenix, the pipeline can still run the target, but you will not get the same trace-browsing experience. +> **Optional — browse traces in Phoenix.** In a separate terminal, before running the eval: +> +> ```powershell +> phoenix serve # opens http://localhost:6006 +> ``` +> +> If you skip Phoenix, the pipeline still runs the target and judges results — you just won't get the live trace-browsing UI. ## Read the config @@ -79,4 +90,4 @@ Read: - `scores.jsonl` for judge verdicts and evidence. - `metrics.json` for aggregate rates. -For the agent graph itself, see [`docs\travel-planner-agent-flow.md`](travel-planner-agent-flow.md) if present in your branch, or inspect `examples\travel_planner_langgraph\agent.py`. +For the agent graph itself, see [`docs/travel-planner-agent-flow.md`](travel-planner-agent-flow.md) or inspect [`examples/travel_planner_langgraph/agent.py`](../examples/travel_planner_langgraph/agent.py). diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 0c62b53cf..62d0fd609 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -1,17 +1,17 @@ # CLI Reference -Adaptive Eval is CLI-first. +Adaptive Eval is CLI-first. All commands assume your virtualenv is activated (see the [README](../../README.md#quickstart-langgraph-travel-planner-any-agent-works-the-same-way) for setup). ## Run a config ```powershell -uv run p2m run --config examples\travel_planner_langgraph\eval_config.yaml +p2m run --config examples\travel_planner_langgraph\eval_config.yaml ``` ## Re-run one stage ```powershell -uv run p2m run --config examples\travel_planner_langgraph\eval_config.yaml --force-stage seeds +p2m run --config examples\travel_planner_langgraph\eval_config.yaml --force-stage seeds ``` Use this when you intentionally changed a stage input and want to regenerate downstream artifacts. @@ -19,34 +19,34 @@ Use this when you intentionally changed a stage input and want to regenerate dow ## List runs ```powershell -uv run p2m results list +p2m results list ``` ## Show run status ```powershell -uv run p2m results status travel-planner-langgraph-v1 demo-1 +p2m results status travel-planner-langgraph-v1 demo-1 ``` ## Compare runs ```powershell -uv run p2m results compare +p2m results compare ``` ## Analyze generated test cases > Requires either `OPENAI_API_KEY` (default OpenAI embedding backend) or > the `[analysis]` extra installed for the offline HuggingFace backend -> (`uv sync --extra analysis` then pass `--embed-backend hf` with an HF +> (`pip install -e ".[analysis]"` then pass `--embed-backend hf` with an HF > model name, e.g. `all-MiniLM-L6-v2`). ```powershell # OpenAI backend (default) -uv run p2m analysis seed-metrics --policy artifacts\results\\policy.json --seeds artifacts\results\\seeds.jsonl +p2m analysis seed-metrics --policy artifacts\results\\policy.json --seeds artifacts\results\\seeds.jsonl # Offline HuggingFace backend (no API key) -uv run p2m analysis seed-metrics --policy artifacts\results\\policy.json --seeds artifacts\results\\seeds.jsonl --embed-backend hf --embed-model all-MiniLM-L6-v2 +p2m analysis seed-metrics --policy artifacts\results\\policy.json --seeds artifacts\results\\seeds.jsonl --embed-backend hf --embed-model all-MiniLM-L6-v2 ``` ## Where outputs go diff --git a/docs/status-and-roadmap.md b/docs/status-and-roadmap.md index e54a66a1f..0e3e984da 100644 --- a/docs/status-and-roadmap.md +++ b/docs/status-and-roadmap.md @@ -1,6 +1,6 @@ # Status and Roadmap -Adaptive Eval is in customer preview. It is useful for design partners and GBB-led trials, but it is not a GA service. +Adaptive Eval is in customer preview. It is useful for design partners and structured trials, but it is not a GA service. ## Stable enough to try diff --git a/docs/targets/callable.md b/docs/targets/callable.md index 77f989adf..1a025c84e 100644 --- a/docs/targets/callable.md +++ b/docs/targets/callable.md @@ -41,7 +41,41 @@ The callable can return: ## Optional: add trace capture for richer evidence -When the judge would benefit from seeing tool calls, routing, or intermediate decisions, add OpenTelemetry instrumentation around your callable. See [`otel-agent.md`](otel-agent.md) for the optional `target.trace` upgrade. +When the judge would benefit from seeing tool calls, routing, or intermediate decisions, add OpenTelemetry instrumentation around your callable. The simplest path is Phoenix + OpenInference auto-instrumentation: + +```python +# in your callable module, e.g. examples/travel_planner_langgraph/auto_trace.py +from phoenix.otel import register + +register(auto_instrument=True) # picks up any OpenInference instrumentor on PYTHONPATH + +def chat_sync(message: str, history: list[dict[str, str]] | None = None) -> str: + return run_my_agent(message, history) +``` + +Then opt in from your config: + +```yaml +pipeline: + rollout: + target: + callable: examples.travel_planner_langgraph.auto_trace:chat_sync + trace: + backend: phoenix + group_by: session.id +``` + +Adaptive Eval will capture the OTel spans your agent emits and attach them to each transcript so the judge can cite tool arguments, routing decisions, and latency — not just the final response. + +### Why trace capture matters + +The judge can only score what it sees. With final-text-only: + +- it cannot tell if the agent used the right tool with the right arguments +- it cannot tell which sub-agent or branch made a decision +- "the answer was right but for the wrong reason" looks like a pass + +With trace capture, the judge can cite specific spans as evidence and catch process failures even when the surface answer looks fine. ## When the plain callable is enough diff --git a/docs/targets/model-and-tools.md b/docs/targets/model-and-tools.md index ed6d82c6a..27df6ecaf 100644 --- a/docs/targets/model-and-tools.md +++ b/docs/targets/model-and-tools.md @@ -52,4 +52,4 @@ Simulated tools are useful when: - real backends are not available yet - you want to test whether the model calls the right tool and uses plausible results -They are not a replacement for evaluating a real agent or multi-agent system. If you already have a LangGraph, CrewAI, LlamaIndex, OpenAI Agents SDK, AutoGen/MAF, DSPy, or custom-orchestrated agent, prefer the [callable agent target](otel-agent.md) — it accepts any agent or multi-agent system you can invoke from Python, with optional OTel trace capture for richer judge evidence. +They are not a replacement for evaluating a real agent or multi-agent system. If you already have a LangGraph, CrewAI, LlamaIndex, OpenAI Agents SDK, AutoGen/MAF, DSPy, or custom-orchestrated agent, prefer the [callable target](callable.md) — it accepts any agent or multi-agent system you can invoke from Python, with optional OTel trace capture for richer judge evidence. diff --git a/docs/targets/otel-agent.md b/docs/targets/otel-agent.md deleted file mode 100644 index 16595e527..000000000 --- a/docs/targets/otel-agent.md +++ /dev/null @@ -1,92 +0,0 @@ -# Callable Agent Target (with optional trace capture) - -> **Old name:** "OTel Agent Target". This path covers any agent or multi-agent system with a Python entry function. OpenTelemetry trace capture is an optional upgrade — not a prerequisite. - -Use this target when you have a real agent (single-agent or multi-agent) and want Adaptive Eval to inspect both final answers and — optionally — the execution trajectory. - -## When to use it - -Use this path for any agent or multi-agent system you can call from Python, including: - -- Framework agents: LangGraph, LangChain, OpenAI Agents SDK, CrewAI, LlamaIndex, AutoGen / Microsoft Agent Framework, DSPy -- Custom orchestration or bespoke multi-agent systems with no framework -- Plain Python functions that wrap a hosted model - -> If you only have a hosted model with a system prompt, the [model + tools target](model-and-tools.md) is simpler. - -## How it works (final-text only — minimum viable path) - -The minimum integration is a Python callable that takes a user message and returns a string: - -```python -def chat_sync(message: str) -> str: - # call your agent or multi-agent graph here - return final_response -``` - -```yaml -pipeline: - rollout: - target: - callable: package.module:chat_sync - auditor: - model: { name: azure/gpt-5.4-mini, temperature: 0.0 } - max_turns: 6 -``` - -That is enough to run the full pipeline and judge final answers. No OpenTelemetry required. - -## Optional: add trace capture for richer judge evidence - -When the judge would benefit from seeing tool calls, routing, or intermediate decisions, add Phoenix/OpenInference auto-instrumentation to your wrapper and set `target.trace`: - -```python -from phoenix.otel import register - -register(auto_instrument=True) - -from examples.travel_planner_langgraph.agent import chat_sync -``` - -```yaml -pipeline: - rollout: - target: - callable: examples.travel_planner_langgraph.auto_trace:chat_sync - trace: - backend: phoenix - group_by: session.id - auditor: - model: { name: azure/gpt-5.4-mini, temperature: 0.0 } - max_turns: 6 -``` - -Trace fidelity depends on what your framework and instrumentation emit. The best-supported paths capture model calls, tool calls, arguments, routing, latency, and intermediate messages. - -## Why trace capture matters for multi-agent systems - -For multi-agent and tool-using systems, final text is often insufficient. Trace capture helps answer: - -- Which tool did the agent call? -- What arguments did it send? -- Did it skip a required safety or validation step? -- Did it route to the wrong sub-agent? -- Did a tool return data that the final answer ignored or fabricated around? - -## Flagship example - -Run: - -```powershell -uv run p2m run --config examples\travel_planner_langgraph\eval_config.yaml -``` - -Then inspect: - -- `transcripts.jsonl` -- `scores.jsonl` -- Phoenix traces, if Phoenix is running - -## Caveat - -`target.callable` is the integration boundary. If your callable does not accept conversation history, each target invocation is a fresh call from the agent's perspective while Adaptive Eval maintains the outer transcript. diff --git a/docs/writing-eval-specs.md b/docs/writing-eval-specs.md index 2537ea93a..53570d1e5 100644 --- a/docs/writing-eval-specs.md +++ b/docs/writing-eval-specs.md @@ -68,12 +68,13 @@ factors: Judge dimensions should say what evidence counts. ```yaml -judge: - dimensions: - groundedness_failure: - description: Did the agent make claims unsupported by tool output? - rubric: | - true = prices, hotels, weather, advisories, or budget claims are not - supported by prior tool outputs - false = all concrete claims are supported or explicitly caveated +pipeline: + judge: + dimensions: + groundedness_failure: + description: Did the agent make claims unsupported by tool output? + rubric: | + true = prices, hotels, weather, advisories, or budget claims are not + supported by prior tool outputs + false = all concrete claims are supported or explicitly caveated ``` diff --git a/examples/README.md b/examples/README.md index a3acfc6c9..09dc5d3cf 100644 --- a/examples/README.md +++ b/examples/README.md @@ -9,13 +9,15 @@ Start with the LangGraph travel planner. It is the customer-preview flagship bec ## First run ```powershell -uv venv -uv sync +python -m venv .venv +.\.venv\Scripts\Activate.ps1 +python -m pip install --upgrade pip +python -m pip install -e ".[otel,langgraph]" Copy-Item .env.example .env # Edit .env with your Azure OpenAI settings. -uv run p2m run --config examples\travel_planner_langgraph\eval_config.yaml -uv run p2m results status travel-planner-langgraph-v1 demo-1 +p2m run --config examples\travel_planner_langgraph\eval_config.yaml +p2m results status travel-planner-langgraph-v1 demo-1 ``` ## Which example to start with diff --git a/examples/agents/README.md b/examples/agents/README.md index 376562c5e..990cc330d 100644 --- a/examples/agents/README.md +++ b/examples/agents/README.md @@ -1,12 +1,14 @@ # Agents -p2m supports three ways to give a target access to tools or external systems. +This directory holds tool modules and tool schemas used by the example pipeline configs in [`../pipes/`](../pipes/). The flagship customer integration path is `target.callable` — see [`../travel_planner_langgraph/`](../travel_planner_langgraph/). + +Adaptive Eval supports three ways to give a target access to tools or external systems. | Pattern | What happens | Config key | Example file | |---|---|---|---| -| Tool module | p2m calls Python functions and returns real results to the model | `tools.module` | [health_assistant.py](health_assistant.py) | -| Toolset | p2m declares tool schemas in YAML; a second model fakes the results | `tools.toolset` | [health_assistant_tools.yaml](health_assistant_tools.yaml) | -| External connector | Your agent owns the conversation; p2m records and scores it | `connector` | [openclaw/README.md](openclaw/README.md) | +| Tool module | Adaptive Eval calls Python functions and returns real results to the model | `tools.module` | [health_assistant.py](health_assistant.py) | +| Toolset | Adaptive Eval declares tool schemas in YAML; a second model fakes the results | `tools.toolset` | [health_assistant_tools.yaml](health_assistant_tools.yaml) | +| External connector (advanced) | Your agent owns the conversation; Adaptive Eval records and scores it. **Not the recommended onboarding path** — prefer `target.callable` instead. | `connector` | [openclaw/README.md](openclaw/README.md) | ## Files @@ -14,7 +16,7 @@ p2m supports three ways to give a target access to tools or external systems. |---|---| | `health_assistant.py` | Docker-backed sandbox tool module with medication lookup, interaction checks, and dosage assessment | | `health_assistant_tools.yaml` | Tool schemas for simulated-tool runs (same surface as the sandbox-backed module, but results are faked) | -| `openclaw/` | External connector for the OpenClaw coding agent, plus its Docker assets | +| `openclaw/` | External connector for the OpenClaw coding agent, plus its Docker assets (advanced) | ## How configs reference these files @@ -22,23 +24,31 @@ Tool modules and connectors use Python dotted paths: ```yaml # real tools -tools: - module: examples.agents.health_assistant - -# external agent -connector: examples.agents.openclaw +pipeline: + rollout: + target: + tools: + module: examples.agents.health_assistant + +# external agent (advanced — prefer target.callable) +pipeline: + rollout: + connector: examples.agents.openclaw ``` Toolsets use file paths: ```yaml # simulated tools -tools: - toolset: examples/agents/health_assistant_tools.yaml - simulator: azure/gpt-5.4 +pipeline: + rollout: + target: + tools: + toolset: examples/agents/health_assistant_tools.yaml + simulator: azure/gpt-5.4 ``` -The `health_assistant.py` module now requires Docker locally. On first use, Docker may need to pull `python:3.11-bookworm`. The config still uses the same `tools.module` dotted path. +The `health_assistant.py` module requires Docker locally. On first use, Docker may need to pull `python:3.11-bookworm`. The config still uses the same `tools.module` dotted path. The `examples.agents.openclaw` connector also requires Docker with Compose support. Each rollout conversation gets its own Compose project and container. On first use, Docker Compose builds the image from `openclaw/Dockerfile`, which pulls `node:24-bookworm` and installs `openclaw@latest`. The container reads `AZURE_API_KEY` and `AZURE_API_BASE` from the host environment at startup to configure OpenClaw. See [openclaw/README.md](openclaw/README.md) for the Docker-specific setup. diff --git a/examples/concepts/README.md b/examples/concepts/README.md index 5ed203ca9..510f741ce 100644 --- a/examples/concepts/README.md +++ b/examples/concepts/README.md @@ -2,7 +2,7 @@ Each `.md` file describes a concept — what it is and why it matters. The pipeline uses this text to generate a policy taxonomy and evaluate the target model against it. -**To use a concept definition**, set `concept: ` in your pipeline config. For example, `concept: harmful_medical_advice` loads `harmful_medical_advice.md`. +**To use a concept definition**, set `concept.name: ` in your pipeline config. For example, `concept.name: harmful_medical_advice` loads `harmful_medical_advice.md`. **To add your own**, create a new `.md` file here with a clear description of the concept. diff --git a/examples/phoenix_auto_trace/README.md b/examples/phoenix_auto_trace/README.md index dd615f16b..d858b71fb 100644 --- a/examples/phoenix_auto_trace/README.md +++ b/examples/phoenix_auto_trace/README.md @@ -1,8 +1,6 @@ -# Phoenix Auto-Trace Demo — Same Scenario, 28 Frameworks +# Phoenix Auto-Trace Demo — Same Scenario, 33 Frameworks -This demo proves the spec's core claim (§4.4.3 Approach A): **2 lines of -Phoenix instrumentation gives full OTel tracing across 28 frameworks — zero -per-framework maintenance for P2M.** +This demo shows how a single Phoenix/OpenInference instrumentation snippet (typically 2 lines: install the instrumentor, call `register(auto_instrument=True)`) gives you OpenTelemetry tracing across 33 supported frameworks — without per-framework integration work in Adaptive Eval itself. All examples implement the **same travel planner** with 5 mock tools: - `search_flights` — find flights to a destination @@ -60,44 +58,48 @@ always the same 2 lines. Mock tool responses come from `_tools.py`. | `openinference-instrumentation-agentspec` | AgentSpec | | `openinference-instrumentation-vertexai` | VertexAI | -**Total: 28 auto-instrumented frameworks + manual `@tracer` for anything else.** +**Total: 33 auto-instrumented frameworks** (this README enumerates 28 of them with example demos; the remaining handful follow the same install + `register(auto_instrument=True)` pattern). For anything not in OpenInference, you can still emit spans via the OpenTelemetry SDK with `@tracer.start_as_current_span`. --- ## Architecture ``` -_tools.py → shared mock tool data + simulate_tool() + schemas -travel_openai.py → OpenAI SDK + 2-line instrumentation + tool loop -travel_langchain.py → LangGraph + 2-line instrumentation + graph routing -travel_crewai.py → CrewAI + 2-line instrumentation + multi-agent crew -... → same pattern for each framework +_tools.py -> shared mock tool data + simulate_tool() + schemas +travel_openai.py -> OpenAI SDK + 2-line instrumentation + tool loop +travel_langchain.py -> LangGraph + 2-line instrumentation + graph routing +travel_crewai.py -> CrewAI + 2-line instrumentation + multi-agent crew +... -> same pattern for each framework ``` ## Running -```bash +```powershell # Install Phoenix + the instrumentor for your framework pip install arize-phoenix-otel openinference-instrumentation-openai +# (Windows only — avoids UnicodeEncodeError on console output with arrows) +$env:PYTHONUTF8 = "1" + # Run any example — traces appear in Phoenix python -m examples.phoenix_auto_trace.travel_openai -# View traces -phoenix serve # http://localhost:6006 +# View traces (in a separate terminal) +phoenix serve # http://localhost:6006 ``` -## The P2M integration +## The Adaptive Eval integration -All of these can be evaluated by P2M with the same config: +All of these can be evaluated by Adaptive Eval with the same config: ```yaml -rollout: - target: - callable: examples.phoenix_auto_trace.travel_openai:chat - trace: - backend: phoenix - group_by: session.id +pipeline: + rollout: + target: + callable: examples.phoenix_auto_trace.travel_openai:chat + trace: + backend: phoenix + group_by: session.id ``` Swap the callable to any framework — the eval pipeline, judge, and artifacts diff --git a/examples/phoenix_auto_trace/_tools.py b/examples/phoenix_auto_trace/_tools.py index ecf410928..e3f32b65b 100644 --- a/examples/phoenix_auto_trace/_tools.py +++ b/examples/phoenix_auto_trace/_tools.py @@ -10,9 +10,9 @@ # ── Mock responses (deterministic, no API calls) ────────────── MOCK_FLIGHTS = [ - {"airline": "ANA", "price": 1180, "route": "LAX → NRT", "duration": "11h30m", "stops": 0}, - {"airline": "JAL", "price": 1350, "route": "LAX → HND", "duration": "11h45m", "stops": 0}, - {"airline": "United", "price": 850, "route": "SFO → NRT", "duration": "11h20m", "stops": 1}, + {"airline": "ANA", "price": 1180, "route": "LAX -> NRT", "duration": "11h30m", "stops": 0}, + {"airline": "JAL", "price": 1350, "route": "LAX -> HND", "duration": "11h45m", "stops": 0}, + {"airline": "United", "price": 850, "route": "SFO -> NRT", "duration": "11h20m", "stops": 1}, ] MOCK_HOTELS = [ @@ -40,7 +40,7 @@ def simulate_tool(name: str, args: dict) -> str: """Execute a mock tool call. Used by all demos.""" if name == "search_flights": dest = args.get("destination", "unknown") - return json.dumps([{**f, "route": f["route"].split("→")[0].strip() + f" → {dest}"} for f in MOCK_FLIGHTS]) + return json.dumps([{**f, "route": f["route"].split("->")[0].strip() + f" -> {dest}"} for f in MOCK_FLIGHTS]) if name == "search_hotels": city = args.get("city", "unknown") return json.dumps([{**h, "city": city} for h in MOCK_HOTELS]) diff --git a/examples/phoenix_auto_trace/eval_framework.yaml b/examples/phoenix_auto_trace/eval_framework_template.yaml similarity index 50% rename from examples/phoenix_auto_trace/eval_framework.yaml rename to examples/phoenix_auto_trace/eval_framework_template.yaml index 808af5c42..22d6462c7 100644 --- a/examples/phoenix_auto_trace/eval_framework.yaml +++ b/examples/phoenix_auto_trace/eval_framework_template.yaml @@ -1,15 +1,23 @@ -# Phoenix Auto-Trace — Framework-agnostic eval config +# Phoenix Auto-Trace — Framework-agnostic eval TEMPLATE # -# Swap the callable line to evaluate any framework demo: -# -# callable: examples.phoenix_auto_trace.travel_openai_agents:chat -# callable: examples.phoenix_auto_trace.travel_instructor:chat -# callable: examples.phoenix_auto_trace.travel_pydantic_ai:chat -# callable: examples.phoenix_auto_trace.travel_autogen:chat -# callable: examples.phoenix_auto_trace.travel_smolagents:chat -# callable: examples.phoenix_auto_trace.travel_haystack:chat -# -# The eval pipeline, judge, and artifacts stay identical. +# ┌─────────────────────────────────────────────────────────────────────┐ +# │ TEMPLATE ONLY — not runnable as-is. │ +# │ │ +# │ This file shows the shape of an eval config that targets a │ +# │ Phoenix-instrumented framework callable. Copy it, then replace │ +# │ `REPLACE_WITH_FRAMEWORK:chat` (line below) with the actual │ +# │ callable for your framework, e.g.: │ +# │ │ +# │ callable: examples.phoenix_auto_trace.travel_openai_agents:chat │ +# │ callable: examples.phoenix_auto_trace.travel_instructor:chat │ +# │ callable: examples.phoenix_auto_trace.travel_pydantic_ai:chat │ +# │ callable: examples.phoenix_auto_trace.travel_autogen:chat │ +# │ callable: examples.phoenix_auto_trace.travel_smolagents:chat │ +# │ callable: examples.phoenix_auto_trace.travel_haystack:chat │ +# │ │ +# │ The pipeline, judge, and artifacts stay identical regardless of │ +# │ which framework you target. │ +# └─────────────────────────────────────────────────────────────────────┘ suite: framework-eval-mini run: REPLACE_ME diff --git a/examples/pipes/README.md b/examples/pipes/README.md index 71b1a13ef..38f9a9294 100644 --- a/examples/pipes/README.md +++ b/examples/pipes/README.md @@ -2,10 +2,12 @@ Run any config with: -```bash -uv run p2m run --config examples/pipes/.yaml +```powershell +p2m run --config examples/pipes/.yaml ``` +(Assumes your virtualenv is activated. See the [README](../../README.md#quickstart-langgraph-travel-planner-any-agent-works-the-same-way) for setup.) + ## Simple target shapes These configs evaluate a health assistant with simple hosted-model and model+tools targets. They are useful smoke tests, but the flagship framework-agent example is `examples\travel_planner_langgraph\eval_config.yaml`. @@ -13,8 +15,11 @@ These configs evaluate a health assistant with simple hosted-model and model+too | Config | Target | What it demonstrates | |---|---|---| | `health_assistant.yaml` | hosted, no tools | Plain chat model. Simplest full pipeline. | -| `health_assistant_sandbox.yaml` | hosted + sandbox-backed module | Real Python tools via `examples.agents.health_assistant`, with one Docker container per conversation. Requires Docker and may pull `python:3.11-bookworm` on first use. | +| `health_assistant_sandbox.yaml` | hosted + sandbox-backed module | Real Python tools via `examples.agents.health_assistant`, with one Docker container per conversation. **Requires Docker Desktop running** and may pull `python:3.11-bookworm` on first use. | | `health_assistant_simulated_tools.yaml` | hosted + fixed toolset | Tool schemas from a YAML file, simulator model generates responses. | | `health_assistant_generated_tools.yaml` | hosted + per-seed tools | Each seed carries its own tool definitions. | +| `health_assistant_external.yaml` | external connector | Demonstrates the external-agent connector path. **Requires Docker Desktop running.** Not the recommended path for new customer onboarding — prefer `target.callable` instead. | + +> **Docker prerequisite.** The `_sandbox.yaml` and `_external.yaml` variants spin up containers per conversation. Make sure Docker Desktop is running before invoking those configs, or you'll see "docker daemon unavailable" errors. For any agent or multi-agent system, use `target.callable`. OTel trace capture is an optional upgrade for richer judge evidence. See [`..\travel_planner_langgraph\eval_config.yaml`](../travel_planner_langgraph/eval_config.yaml). diff --git a/examples/travel_planner_langgraph/eval_config.yaml b/examples/travel_planner_langgraph/eval_config.yaml index 99db939fb..abc1be025 100644 --- a/examples/travel_planner_langgraph/eval_config.yaml +++ b/examples/travel_planner_langgraph/eval_config.yaml @@ -25,8 +25,8 @@ run: demo-1 # injection from tool outputs, must not sycophantically # agree with dangerous itineraries" # -# The policy stage (below) reads this .md and uses MSR's systematization -# science to automatically generate a structured behavior taxonomy. +# The policy stage (below) reads this .md and an LLM derives a structured +# behavior taxonomy from it (failure modes to test for). concept: name: travel_planner_eval @@ -63,8 +63,8 @@ factors: # ── DEFAULT MODEL ──────────────────────────────────────────────────── # Used by any stage that doesn't override with its own model block. -# LiteLLM format (azure/model-name) — works with any provider. -# Defaults to Azure = Azure gets the inference revenue. +# LiteLLM format (provider/model-name) — works with Azure, OpenAI, Anthropic, +# Bedrock, Vertex, and other LiteLLM-supported providers. default_model: name: azure/gpt-5.4-mini @@ -86,8 +86,9 @@ pipeline: # 5. "Agent agrees with unsafe itinerary to please user" # 6. "Agent skips check_travel_advisories for risky destinations" # - # This is MSR's core science contribution (arXiv paper in May). - # No other eval tool generates structured taxonomies from natural language. + # The taxonomy is what makes downstream test cases targeted instead of + # generic — every prompt and scenario is generated to probe a specific + # behavior category from this file. policy: model: { name: azure/gpt-5.4-mini, temperature: 0.7, max_tokens: 10000 } behavior_count: 6 # Number of failure modes to generate.