diff --git a/AGENTS.md b/AGENTS.md index 43702d75e..f8a5a8fea 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -114,11 +114,11 @@ assert-ai run --config examples/travel_planner_langgraph/eval_config.yaml Read artifacts in this order: -1. `metrics.json` -2. `scores.jsonl` -3. `inference_set.jsonl` -4. Phoenix/OpenInference traces, if configured -5. `config.yaml` +1. `scores.jsonl` +2. `inference_set.jsonl` +3. Phoenix/OpenInference traces, if configured +4. `config.yaml` +5. `metrics.json` Look for judge evidence, cited turns, tool calls, routing decisions, and trace references. diff --git a/assert_ai/library/judges/README.md b/assert_ai/library/judges/README.md index 975d08cf6..91cb4389d 100644 --- a/assert_ai/library/judges/README.md +++ b/assert_ai/library/judges/README.md @@ -67,4 +67,4 @@ Each dimension has: - **name** — unique identifier used in `scores.jsonl` - **description** — rubric the LLM judge follows (be specific and concrete) - **scale** — `[low, high]` scoring range -- **weight** — relative importance when aggregating into `metrics.json` +- **weight** — relative importance when aggregating scores from `scores.jsonl` into summary rates diff --git a/docs/concepts.md b/docs/concepts.md index fcee20ea2..f8aa0f518 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -64,10 +64,11 @@ In the scoring stage, ASSERT evaluates each trace against the associated behavio `pipeline.judge` scores each output with your dimensions and rubrics. -Outputs: +Output: - `scores.jsonl` -- `metrics.json` + +`metrics.json` (pipeline token-usage telemetry) is written by the runner after all stages complete, not by the judge stage itself. ## Risks and limitations of ASSERT diff --git a/docs/getting-started.md b/docs/getting-started.md index ff8155ded..8b82b448a 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,134 +1,134 @@ -# Getting Started - -This guide covers installation and your first end-to-end evaluation run. - -## Prerequisites - -- Python 3.11+ -- pip -- Model credentials in environment variables (for example `AZURE_API_KEY` and `AZURE_API_BASE` for Azure OpenAI) - -## Install with a quickstart example: LangGraph travel planner - -The flagship example evaluates a multi-tool LangGraph travel planner. The target is reached through `target.callable` — the same integration boundary you would use for any agent or multi-agent system — and Phoenix/OpenInference auto-instrumentation captures the agent's OpenTelemetry spans so the judge can cite tool calls and routing decisions. This is the recommended integration shape for any non-trivial agent. - -### Recommended install path - -bash (macOS / Linux): - -```bash -python -m venv .venv -source .venv/bin/activate -python -m pip install --upgrade pip -python -m pip install -e ".[otel,langgraph]" -cp .env.example .env -``` - -Edit `.env` with credentials for your provider. Defaults match the example's `azure/...` model. Any LiteLLM provider (OpenAI, Anthropic, Bedrock, Vertex, Ollama, and others) works. - -PowerShell (Windows): - -```powershell -python -m venv .venv -.\.venv\Scripts\Activate.ps1 -python -m pip install --upgrade pip -python -m pip install -e ".[otel,langgraph]" -Copy-Item .env.example .env -``` - -### Run your first evaluation - -The example's `auto_trace.py` calls `assert_ai.auto_trace.enable()`, which installs the available OpenInference instrumentors locally so the judge can cite tool calls, routing decisions, model calls, and latency as evidence. It does **not** start a Phoenix server. - -`phoenix serve` is optional — only run it if you want a browser UI to inspect the traces visually. The eval runs and the judge see the same span data either way. - -bash (macOS / Linux): - -```bash -phoenix serve # optional: trace UI on http://localhost:6006 -assert-ai run --config examples/travel_planner_langgraph/eval_config.yaml -``` - -PowerShell (Windows): - -```powershell -phoenix serve # optional: trace UI on http://localhost:6006 -assert-ai run --config examples/travel_planner_langgraph/eval_config.yaml -``` - -Check run status: - -PowerShell (Windows): - -```powershell -assert-ai results status travel-planner-langgraph-v1 demo-1 -``` - -bash (macOS / Linux): - -```bash -assert-ai results status travel-planner-langgraph-v1 demo-1 -``` - -Artifacts are written under: - -```text -artifacts/results/travel-planner-langgraph-v1/demo-1/ -``` - -### Codespaces / VS Code Dev Containers - -[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/microsoft/ASSERT) - -The repo includes a minimal dev container for the LangGraph quickstart. It installs `.[otel,langgraph,dev]`, copies `.env.example` to `.env` if needed, and forwards Phoenix on port `6006`. After container setup, add your provider credentials to `.env` and run the same `assert-ai run` command. - -PowerShell (Windows) — full sequence: - -```powershell -python -m venv .venv -.\.venv\Scripts\Activate.ps1 -python -m pip install --upgrade pip -python -m pip install -e ".[otel,langgraph]" -Copy-Item .env.example .env - -phoenix serve # optional -assert-ai run --config examples/travel_planner_langgraph/eval_config.yaml -assert-ai results status travel-planner-langgraph-v1 demo-1 -``` - -## What just happened - -1. `systematize` expanded the behavior spec into behavior categories. -2. `test_set` generated prompt and scenario test cases. -3. `inference` executed the target for each case. -4. `judge` produced verdicts, evidence, and aggregate metrics. - -What the quickstart does: - -| Step | Developer behavior | Current YAML / artifact | -|---|---|---| -| 1 | **Eval spec**: plain-English behavior requirements | `behavior.name` and `behavior.description` live inline in `eval_config.yaml` | -| 2 | **Behavior categories**: generated failure-mode taxonomy | `pipeline.systematize` writes `taxonomy.json` | -| 3 | **Test cases**: prompts and multi-turn scenarios | `pipeline.test_set` writes `test_set.jsonl` | -| 4 | **Execute**: run the agent and capture traces | `pipeline.inference.target.callable` + `target.trace` write `inference_set.jsonl` | -| 5 | **Judge**: score against your rubric | `pipeline.judge.dimensions` writes `scores.jsonl` and `metrics.json` | - -### CLI helper assistant to create your own config - -Don't want to write YAML by hand? `assert-ai init` starts a conversational LLM assistant that asks about your agent, eval goals, and constraints, then proposes a complete config YAML file to use for your evaluations. - -`assert-ai init` needs an LLM to power the conversation. Pass `--model` with any [LiteLLM model string](https://docs.litellm.ai/docs/providers) and make sure the matching API key is set in your `.env` file (loaded by default) or environment: - -```bash -assert-ai init --model azure/gpt-5.4 -# or skip the first question: -assert-ai init --model azure/gpt-5.4 --describe "A customer-support chatbot with order-lookup and refund tools" -# or edit/extend an existing config: -assert-ai init --model azure/gpt-5.4 --from examples/travel_planner_langgraph/eval_config.yaml -``` - -See [CLI Commands](cli/commands.md) for the full option reference. - -- To learn the config format, see [Config Overview](config/overview.md). -- To inspect outputs in detail, see [Results Guide](guides/results.md). -- To use the local web viewer, see [Run the Local UI Viewer Application](guides/run-local-viewer.md). +# Getting Started + +This guide covers installation and your first end-to-end evaluation run. + +## Prerequisites + +- Python 3.11+ +- pip +- Model credentials in environment variables (for example `AZURE_API_KEY` and `AZURE_API_BASE` for Azure OpenAI) + +## Install with a quickstart example: LangGraph travel planner + +The flagship example evaluates a multi-tool LangGraph travel planner. The target is reached through `target.callable` — the same integration boundary you would use for any agent or multi-agent system — and Phoenix/OpenInference auto-instrumentation captures the agent's OpenTelemetry spans so the judge can cite tool calls and routing decisions. This is the recommended integration shape for any non-trivial agent. + +### Recommended install path + +bash (macOS / Linux): + +```bash +python -m venv .venv +source .venv/bin/activate +python -m pip install --upgrade pip +python -m pip install -e ".[otel,langgraph]" +cp .env.example .env +``` + +Edit `.env` with credentials for your provider. Defaults match the example's `azure/...` model. Any LiteLLM provider (OpenAI, Anthropic, Bedrock, Vertex, Ollama, and others) works. + +PowerShell (Windows): + +```powershell +python -m venv .venv +.\.venv\Scripts\Activate.ps1 +python -m pip install --upgrade pip +python -m pip install -e ".[otel,langgraph]" +Copy-Item .env.example .env +``` + +### Run your first evaluation + +The example's `auto_trace.py` calls `assert_ai.auto_trace.enable()`, which installs the available OpenInference instrumentors locally so the judge can cite tool calls, routing decisions, model calls, and latency as evidence. It does **not** start a Phoenix server. + +`phoenix serve` is optional — only run it if you want a browser UI to inspect the traces visually. The eval runs and the judge see the same span data either way. + +bash (macOS / Linux): + +```bash +phoenix serve # optional: trace UI on http://localhost:6006 +assert-ai run --config examples/travel_planner_langgraph/eval_config.yaml +``` + +PowerShell (Windows): + +```powershell +phoenix serve # optional: trace UI on http://localhost:6006 +assert-ai run --config examples/travel_planner_langgraph/eval_config.yaml +``` + +Check run status: + +PowerShell (Windows): + +```powershell +assert-ai results status travel-planner-langgraph-v1 demo-1 +``` + +bash (macOS / Linux): + +```bash +assert-ai results status travel-planner-langgraph-v1 demo-1 +``` + +Artifacts are written under: + +```text +artifacts/results/travel-planner-langgraph-v1/demo-1/ +``` + +### Codespaces / VS Code Dev Containers + +[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/microsoft/ASSERT) + +The repo includes a minimal dev container for the LangGraph quickstart. It installs `.[otel,langgraph,dev]`, copies `.env.example` to `.env` if needed, and forwards Phoenix on port `6006`. After container setup, add your provider credentials to `.env` and run the same `assert-ai run` command. + +PowerShell (Windows) — full sequence: + +```powershell +python -m venv .venv +.\.venv\Scripts\Activate.ps1 +python -m pip install --upgrade pip +python -m pip install -e ".[otel,langgraph]" +Copy-Item .env.example .env + +phoenix serve # optional +assert-ai run --config examples/travel_planner_langgraph/eval_config.yaml +assert-ai results status travel-planner-langgraph-v1 demo-1 +``` + +## What just happened + +1. `systematize` expanded the behavior spec into behavior categories. +2. `test_set` generated prompt and scenario test cases. +3. `inference` executed the target for each case. +4. `judge` produced verdicts, evidence, and aggregate metrics. + +What the quickstart does: + +| Step | Developer behavior | Current YAML / artifact | +|---|---|---| +| 1 | **Eval spec**: plain-English behavior requirements | `behavior.name` and `behavior.description` live inline in `eval_config.yaml` | +| 2 | **Behavior categories**: generated failure-mode taxonomy | `pipeline.systematize` writes `taxonomy.json` | +| 3 | **Test cases**: prompts and multi-turn scenarios | `pipeline.test_set` writes `test_set.jsonl` | +| 4 | **Execute**: run the agent and capture traces | `pipeline.inference.target.callable` + `target.trace` write `inference_set.jsonl` | +| 5 | **Judge**: score against your rubric | `pipeline.judge.dimensions` writes `scores.jsonl` and `metrics.json` | + +### CLI helper assistant to create your own config + +Don't want to write YAML by hand? `assert-ai init` starts a conversational LLM assistant that asks about your agent, eval goals, and constraints, then proposes a complete config YAML file to use for your evaluations. + +`assert-ai init` needs an LLM to power the conversation. Pass `--model` with any [LiteLLM model string](https://docs.litellm.ai/docs/providers) and make sure the matching API key is set in your `.env` file (loaded by default) or environment: + +```bash +assert-ai init --model azure/gpt-5.4 +# or skip the first question: +assert-ai init --model azure/gpt-5.4 --describe "A customer-support chatbot with order-lookup and refund tools" +# or edit/extend an existing config: +assert-ai init --model azure/gpt-5.4 --from examples/travel_planner_langgraph/eval_config.yaml +``` + +See [CLI Commands](cli/commands.md) for the full option reference. + +- To learn the config format, see [Config Overview](config/overview.md). +- To inspect outputs in detail, see [Results Guide](guides/results.md). +- To use the local web viewer, see [Run the Local UI Viewer Application](guides/run-local-viewer.md). diff --git a/docs/guides/results.md b/docs/guides/results.md index da3dac16b..5ae65327b 100644 --- a/docs/guides/results.md +++ b/docs/guides/results.md @@ -1,63 +1,63 @@ -# Results and artifacts - -ASSERT writes local artifacts and evaluation results under the artifacts folder, sorted by the evaluation suites (configured for each evaluation config YAML specification): - -```text -artifacts/results// -``` - -Run-level outputs are located under each evaluation suite: - -```text -artifacts/results/// -``` - -## Artifact layout and description - -```text -artifacts/results// -├── suite.json -├── taxonomy.json -├── test_set.jsonl -└── / - ├── manifest.json - ├── config.yaml - ├── inference_set.jsonl - ├── scores.jsonl - └── metrics.json -``` - -- `suite.json`: evaluation suite metadata -- `taxonomy.json`: behavior categories generated from your evaluation config YAML in the systematization step of the pipeline. -- `test_set.jsonl`: single turn prompt and multi-turn scenario test cases generated by the test set generation step of the pipeline -- `manifest.json`: stage-by-stage run status and timestamps -- `config.yaml`: frozen config snapshot used for this run -- `inference_set.jsonl`: target outputs plus trace references/events -- `scores.jsonl`: per-case judge verdicts, dimensions, and evidence -- `metrics.json`: aggregate rates by dimension and category, along with token usage metadata - -> **Tip**: After a run, start with `metrics.json` first then see the `scores.jsonl` before inspecting the `inference_set.jsonl` more closely. - -## Useful CLI commands for viewing results - -```bash -assert-ai results list -assert-ai results status -assert-ai results status -assert-ai results compare -assert-ai results compare-suites / / -``` - -See [CLI Commands](../cli/commands.md) for full options. - -## View evaluation suite artifacts and run results in a local UI app - -Access a rich inspector and editing application to view run status, evaluation suite artifacts such as richly rendered taxonomy of behavior categories and their associated policy labels. - -```sh -cd viewer -npm install -npm run dev -``` - -The local hosted UI application server starts at `http://localhost:5174`. Paste this into your browser to open up the inspector view. +# Results and artifacts + +ASSERT writes local artifacts and evaluation results under the artifacts folder, sorted by the evaluation suites (configured for each evaluation config YAML specification): + +```text +artifacts/results// +``` + +Run-level outputs are located under each evaluation suite: + +```text +artifacts/results/// +``` + +## Artifact layout and description + +```text +artifacts/results// +├── suite.json +├── taxonomy.json +├── test_set.jsonl +└── / + ├── manifest.json + ├── config.yaml + ├── inference_set.jsonl + ├── scores.jsonl + └── metrics.json +``` + +- `suite.json`: evaluation suite metadata +- `taxonomy.json`: behavior categories generated from your evaluation config YAML in the systematization step of the pipeline. +- `test_set.jsonl`: single turn prompt and multi-turn scenario test cases generated by the test set generation step of the pipeline +- `manifest.json`: stage-by-stage run status and timestamps +- `config.yaml`: frozen config snapshot used for this run +- `inference_set.jsonl`: target outputs plus trace references/events +- `scores.jsonl`: per-case judge verdicts, dimensions, and evidence +- `metrics.json`: aggregate rates by dimension and category, along with token usage metadata + +> **Tip**: After a run, start with `metrics.json` first then see the `scores.jsonl` before inspecting the `inference_set.jsonl` more closely. + +## Useful CLI commands for viewing results + +```bash +assert-ai results list +assert-ai results status +assert-ai results status +assert-ai results compare +assert-ai results compare-suites / / +``` + +See [CLI Commands](../cli/commands.md) for full options. + +## View evaluation suite artifacts and run results in a local UI app + +Access a rich inspector and editing application to view run status, evaluation suite artifacts such as richly rendered taxonomy of behavior categories and their associated policy labels. + +```sh +cd viewer +npm install +npm run dev +``` + +The local hosted UI application server starts at `http://localhost:5174`. Paste this into your browser to open up the inspector view.