fix: customer-readiness audit on main

AGENTS.md

@@ -17,7 +17,7 @@ eval spec -> behavior categories -> test cases -> execute target -> judge -> art
 - Never read, print, commit, summarize, or infer values from `.env` or other local environment files.
 - Use placeholder names such as `AZURE_API_KEY` and `AZURE_API_BASE`; never invent or expose credential values.
 - Do not recommend committing generated artifacts, local traces, `.venv`, logs, or `.env` files.
-- Treat this repo as a customer-preview distribution repo. Avoid internal strategy, roadmap, TAM, LT review, or PM planning content.
+- Treat this repo as a customer-preview distribution. Keep contributions customer-safe — avoid any internal-only planning, prioritization, or organizational content.
 
 ## Authoritative files
 
@@ -26,8 +26,7 @@ Start with these files:
 - `README.md` - customer-facing overview and quickstart.
 - `docs/quickstart.md` - LangGraph travel planner walkthrough.
 - `docs/targets/overview.md` - target decision tree.
-- `docs/targets/otel-agent.md` - callable agent target with optional trace capture (any agent or multi-agent system).
-- `docs/targets/callable.md` - Python callable target (final-text only, the simplest path).
+- `docs/targets/callable.md` - Python callable target for any agent or multi-agent system, with an optional OpenTelemetry trace-capture upgrade.
 - `docs/targets/model-and-tools.md` - hosted model and simple model+tools target.
 - `CONFIG_REFERENCE.md` - current YAML schema reference.
 - `examples/README.md` - example selection guide.
@@ -123,7 +122,31 @@ Keep docs customer-safe. Prefer improving:
 - `docs/reading-results.md`
 - `examples/README.md`
 
-Do not reintroduce internal planning docs into this private-preview distribution branch.
+Do not reintroduce internal-only planning docs into this customer-preview distribution.
+
+## Paste-in prompt for end users
+
+End users can paste the following block into their AI assistant to get the same orientation this file gives you:
+
+```text
+You are helping me with the Adaptive Eval repo (https://github.com/microsoft/adaptive-eval).
+
+Adaptive Eval is a local-first, spec-driven evaluation pipeline for AI agents. The mental model:
+
+  eval spec -> behavior categories -> test cases -> execute target -> judge -> artifacts
+
+Key facts:
+- The CLI entrypoint is `p2m`. Configs live in `examples/`. Artifacts land in `artifacts/results/<suite>/<run>/`.
+- For any agent or multi-agent system with a Python entry function, use `target.callable`.
+  OpenTelemetry trace capture (Phoenix/OpenInference) is an optional upgrade — not required.
+- For a hosted model with a system prompt and optional tools, use `target.model` and `target.tools`.
+- Read `README.md`, `docs/quickstart.md`, `docs/targets/overview.md`, `docs/targets/callable.md`, and
+  `CONFIG_REFERENCE.md` before suggesting changes to YAML schema.
+- Never read or print values from `.env`. Use placeholder names like AZURE_API_KEY and AZURE_API_BASE.
+- Keep all suggestions customer-safe.
+
+When I ask for help, prefer concrete file paths, runnable commands, and the YAML keys defined in CONFIG_REFERENCE.md.
+```
 
 ## Output style for coding agents
 

CONFIG_REFERENCE.md

@@ -196,8 +196,6 @@ Accepted keys:
 
 For customer-preview configs, `target` must define exactly one of `model`, `callable`, or `endpoint`.
 
-`target.tools` is valid only with `target.model`. It may define `module`, `toolset + simulator`, or `simulator` alone. `toolset` requires `simulator`. If you omit `target.system_prompt`, rollout uses each seed's `system_prompt` when present. Scenario seeds require `auditor`. Prompt seeds do not.
-
 Callable agent example with optional OTel trace capture:
 
 ```yaml
@@ -414,11 +412,15 @@ pipeline:
       sample_size: 10
   rollout:
     target:
+      model:
+        name: azure/gpt-5.4-mini
       system_prompt: |
         You are a health assistant.
   judge: {}
 ```
 
+`target.tools` is valid only with `target.model`. It may define `module`, `toolset + simulator`, or `simulator` alone. `toolset` requires `simulator`. If you omit `target.system_prompt`, rollout uses each seed's `system_prompt` when present. Scenario seeds require `auditor`. Prompt seeds do not.
+
 `concept.md` next to that file:
 
 ```md

README.md

@@ -6,7 +6,7 @@
 
 **Spec-driven evaluation for any agent or multi-agent system - local-first, framework-agnostic, and trace-aware.**
 
-> **Customer preview.** Adaptive Eval is a preview / POC for design-partner and GBB engagements. The core workflow is stable: write an eval spec, generate targeted test cases, execute them against your agent, and judge the results against your rubric. Some YAML field names are still evolving; the docs bridge current names to the intended developer-facing terminology.
+> **Customer preview.** Adaptive Eval is a preview / POC distribution for design partners. The core workflow is stable: write an eval spec, generate targeted test cases, execute them against your agent, and judge the results against your rubric. Some YAML field names are still evolving; the docs bridge current names to the intended developer-facing terminology.
 
 ## Why Adaptive Eval
 
@@ -59,18 +59,6 @@ p2m run --config examples/travel_planner_langgraph/eval_config.yaml
 p2m results status travel-planner-langgraph-v1 demo-1
 ```
 
-### macOS install note
-
-Use the `pip install -e ".[otel,langgraph]"` path above as the primary preview install path on macOS.
-
-We have seen macOS security tooling silently block `uv sync` from extracting several large files from the `litellm` wheel. When that happens, Python imports `litellm` as an empty namespace package and later fails with errors such as `AttributeError: module 'litellm' has no attribute 'acompletion'`. `pip` uses a copy-based install path and avoids this issue.
-
-If you still prefer `uv`, and `litellm` imports without expected attributes, try granting your terminal Full Disk Access and clearing quarantine attributes on the environment:
-
-```bash
-xattr -cr .venv
-```
-
 What the quickstart does:
 
 | Step | Developer concept | Current YAML / artifact |
@@ -96,16 +84,15 @@ behavior categories  ->  test cases + variations  ->  execute target  ->  judge
                                                      + OTel traces     metrics.json
 ```
 
-Today the YAML still uses implementation names such as `concept`, `factors`, `policy`, `seeds`, and `rollout`. The docs use the developer-facing concepts - spec, variations, test cases, execute, judge - and call out the current YAML key the first time each concept appears. See [`docs/glossary.md`](docs/glossary.md).
+Today the YAML still uses implementation names such as `concept`, `factors`, `policy`, `seeds`, and `rollout`. The docs use the developer-facing concepts - spec, variations, test cases, execute, judge - and call out the current YAML key the first time each concept appears. See [`docs/concepts.md`](docs/concepts.md) for the bridge.
 
 ## Choose your target
 
 Pick a target based on how your agent is built.
 
 | Your target looks like... | Use this path | Trace fidelity | Start here |
 |---|---|---|---|
-| Any agent or multi-agent system you can invoke from Python (LangGraph, CrewAI, OpenAI Agents SDK, DSPy, LlamaIndex, AutoGen / MAF, custom orchestration, …) | **Callable agent target**: point `target.callable` at your entry function. Optionally add Phoenix/OpenInference instrumentation and `target.trace` for richer judge evidence. | Final text out of the box; tool calls, arguments, routing, model calls, and latency when you opt into OTel trace capture | [`docs/targets/otel-agent.md`](docs/targets/otel-agent.md) |
-| A Python function that accepts a user message and returns a string or model response | **Plain callable target**: `target.callable: package.module:function` | Inputs/outputs, plus structured tool/model metadata if your callable returns it | [`docs/targets/callable.md`](docs/targets/callable.md) |
+| Any agent or multi-agent system you can invoke from Python (LangGraph, CrewAI, OpenAI Agents SDK, DSPy, LlamaIndex, AutoGen / MAF, custom orchestration, …) — or a plain Python function wrapping a model | **Callable target**: point `target.callable` at your entry function. Optionally add Phoenix/OpenInference instrumentation and `target.trace` for richer judge evidence. | Final text out of the box; tool calls, arguments, routing, model calls, and latency when you opt into OTel trace capture | [`docs/targets/callable.md`](docs/targets/callable.md) |
 | A hosted model with a system prompt, optionally with tools | **Model + tools target**: `target.model`, `target.system_prompt`, and optional `target.tools` | Conversation transcript and tool traces for simple prompt-agent setups | [`docs/targets/model-and-tools.md`](docs/targets/model-and-tools.md) |
 
 **Recommended for best eval results:** add OTel trace capture to your callable when your agent has meaningful internals. Otherwise, the judge is mostly evaluating final text and may miss tool calls, routing decisions, dynamic DAG behavior, and framework internals.
@@ -154,12 +141,12 @@ Browse them with the CLI, the local viewer, or any JSONL tool. Nothing leaves yo
 
 ## Documentation map
 
-- **Get started:** [`docs/quickstart.md`](docs/quickstart.md), [`docs/concepts.md`](docs/concepts.md), [`docs/glossary.md`](docs/glossary.md)
-- **Targets:** [`docs/targets/overview.md`](docs/targets/overview.md), [`docs/targets/otel-agent.md`](docs/targets/otel-agent.md) (callable agent target), [`docs/targets/callable.md`](docs/targets/callable.md), [`docs/targets/model-and-tools.md`](docs/targets/model-and-tools.md)
+- **Get started:** [`docs/quickstart.md`](docs/quickstart.md), [`docs/concepts.md`](docs/concepts.md)
+- **Targets:** [`docs/targets/overview.md`](docs/targets/overview.md), [`docs/targets/callable.md`](docs/targets/callable.md) (any agent), [`docs/targets/model-and-tools.md`](docs/targets/model-and-tools.md)
 - **Authoring:** [`docs/writing-eval-specs.md`](docs/writing-eval-specs.md), [`docs/reading-results.md`](docs/reading-results.md)
 - **Reference:** [`docs/reference/cli.md`](docs/reference/cli.md), [`CONFIG_REFERENCE.md`](CONFIG_REFERENCE.md)
-- **AI assistants:** [`AGENTS.md`](AGENTS.md), [`docs/ai-agent-onboarding.md`](docs/ai-agent-onboarding.md)
-- **Preview operations:** [`docs/private-preview/access-and-repo-workflow.md`](docs/private-preview/access-and-repo-workflow.md), [`docs/status-and-roadmap.md`](docs/status-and-roadmap.md)
+- **AI assistants:** [`AGENTS.md`](AGENTS.md)
+- **Preview status:** [`docs/status-and-roadmap.md`](docs/status-and-roadmap.md)
 
 ## Status
 
@@ -180,3 +167,10 @@ Still evolving:
 - framework-specific quickstarts beyond the current examples
 
 Preview feedback is welcome: confusing names, missing target examples, trace gaps, judge behavior, artifact shape, and docs clarity are all useful signals.
+
+## Troubleshooting
+
+- **macOS, `litellm` AttributeError after install** — some macOS security tooling can silently truncate the `litellm` wheel during extraction with `uv sync`, causing errors like `AttributeError: module 'litellm' has no attribute 'acompletion'`. The `pip install -e ".[otel,langgraph]"` path above uses copy-based installs and avoids this. If you must use `uv`, grant your terminal Full Disk Access and run `xattr -cr .venv` to clear quarantine attributes.
+- **Windows, `UnicodeEncodeError` when running auto-trace demos** — set `$env:PYTHONUTF8 = "1"` before `python -m examples.phoenix_auto_trace.travel_openai`.
+- **Docker-backed pipes fail with "docker daemon unavailable"** — `examples/pipes/health_assistant_sandbox.yaml` and `_external.yaml` need Docker Desktop running.
+