Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions .github/workflows/regression.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,7 @@ jobs:
# CI install set has to include the demo's optional extras even though
# core ASSERT doesn't need them.
run: |
pip install uv
uv pip install --system -e ".[dev,otel]"
python -m pip install -e ".[dev,otel]"

- name: Install viewer npm dependencies
# tests/test_viewer_*.py shell out to `node` against viewer TypeScript
Expand Down Expand Up @@ -93,9 +92,8 @@ jobs:
# sets aligned avoids "passes Tier 1, mysteriously fails Tier 4"
# surprises in the future.
run: |
pip install uv
uv pip install --system -e ".[dev,otel]"
uv pip install --system scipy
python -m pip install -e ".[dev,otel]"
python -m pip install scipy

- name: Detect seed count from labels
id: test_set
Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ Do not recommend an external connector path for customer-preview onboarding.

## Preferred setup commands

For preview customers, prefer `pip` over `uv` in setup instructions:
For preview customers, use `pip` in setup instructions:

```bash
python -m venv .venv
Expand Down
2 changes: 1 addition & 1 deletion assert_eval/analysis/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,4 +37,4 @@ The summary answers four questions about a suite of evaluation runs:

## Dependencies

`stats.py` requires numpy (installed via `uv sync --extra analysis`). The other modules use only the standard library.
`stats.py` requires numpy (for example, install the analysis extras with `python -m pip install -e ".[analysis]"`). The other modules use only the standard library.
2 changes: 1 addition & 1 deletion assert_eval/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ def _handle_missing_analysis_dependency(exc: ModuleNotFoundError) -> None:
missing = getattr(exc, "name", "") or "analysis extras"
_error(
f"Could not import '{missing}'. Install the analysis dependencies first, for example:\n"
" uv sync --extra analysis"
" python -m pip install -e \".[analysis]\""
)


Expand Down
2 changes: 1 addition & 1 deletion assert_eval/core/model_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -597,7 +597,7 @@ def _get_litellm_module() -> Any:
_LITELLM_MODULE = importlib.import_module("litellm")
except ModuleNotFoundError as exc:
raise RuntimeError(
"litellm is not installed. Run `uv sync` in omni/measurements "
"litellm is not installed. Install it with `python -m pip install litellm` "
"before using assert_eval.core.model_client."
) from exc
# Silence noisy litellm warnings that pollute stderr
Expand Down
2 changes: 1 addition & 1 deletion assert_eval/runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -959,7 +959,7 @@ def _run_stages_inner(
run_id = ctx.get('run_id', '')
if suite_id and run_id:
log.info("Inspect results:")
log.info(f" uv run assert-eval results status {suite_id} {run_id}")
log.info(f" assert-eval results status {suite_id} {run_id}")
log.info("View in browser:")
log.info(f" cd viewer && npm run dev (then open http://localhost:5174/suite/{suite_id}/{run_id})")
else:
Expand Down
2 changes: 1 addition & 1 deletion docs/guides/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,6 @@ assert-eval results compare-suites <suite-a>/<run-a> <suite-b>/<run-b>

## 7) Environment-specific fixes

- macOS `litellm` install issue (`AttributeError: module 'litellm' has no attribute 'acompletion'`): some macOS security tooling can silently truncate wheels during `uv sync`. The `pip install -e ".[otel,langgraph]"` path avoids this. If you must use `uv`, grant your terminal Full Disk Access and run `xattr -cr .venv`.
- macOS `litellm` install issue (`AttributeError: module 'litellm' has no attribute 'acompletion'`): some macOS security tooling can silently truncate wheels during package installation. The `pip install -e ".[otel,langgraph]"` path avoids this. If you hit it, grant your terminal Full Disk Access and run `xattr -cr .venv`.
- Windows `UnicodeEncodeError` when running auto-trace demos: set `$env:PYTHONUTF8 = "1"` before `python -m examples.phoenix_auto_trace.travel_openai`.
- Docker-backed Prompt Agent configs fail with `docker daemon unavailable`: ensure Docker Desktop is running for `examples/prompt_agents/health_assistant_sandbox.yaml` and `examples/prompt_agents/health_assistant_external.yaml`.
20 changes: 10 additions & 10 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,14 @@ Start with the LangGraph travel planner. It is the customer-preview flagship bec

```powershell
python -m venv .venv
.\.venv\Scripts\Activate.ps1
./.venv/Scripts/Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e ".[otel,langgraph]"
Copy-Item .env.example .env
# Edit .env with credentials for your provider. The shipped configs use `azure/...` models;
# any LiteLLM provider (OpenAI, Anthropic, Bedrock, Vertex, Ollama, …) works — see https://docs.litellm.ai/docs/providers.

assert-eval run --config examples\travel_planner_langgraph\eval_config.yaml
assert-eval run --config examples/travel_planner_langgraph/eval_config.yaml
assert-eval results status travel-planner-langgraph-v1 demo-1
```

Expand All @@ -29,7 +29,7 @@ Pass `--model` with any [LiteLLM model string](https://docs.litellm.ai/docs/prov
```powershell
assert-eval init --model azure/gpt-5.4-mini
# or seed from an existing example:
assert-eval init --model azure/gpt-5.4-mini --from examples\travel_planner_langgraph\eval_config.yaml
assert-eval init --model azure/gpt-5.4-mini --from examples/travel_planner_langgraph/eval_config.yaml
```

See the [CLI reference](../docs/reference/cli.md#design-a-config-interactively) for all options.
Expand All @@ -38,13 +38,13 @@ See the [CLI reference](../docs/reference/cli.md#design-a-config-interactively)

| Goal | Example | Notes |
|---|---|---|
| Evaluate any agent or multi-agent system (recommended) | `travel_planner_langgraph\eval_config.yaml` | Flagship. Uses `target.callable` with `target.trace.backend: phoenix` so the judge sees tool calls and routing. |
| **See runtime + eval close the loop on a real workflow** | `incident_triage_agent\eval_config_baseline.yaml` + `eval_config_naive_prompt.yaml` + `eval_config_guarded.yaml` + `eval_config_guarded_gepa.yaml` | Joint AgentShield + ASSERT demo. SRE incident-triage agent run across a 4-variant matrix (baseline weak prompt → naïve DO-NOT prompt → ACS gates → ACS + GEPA-optimized prompt) over a 4-axis failure-mode taxonomy to prove the runtime+eval loop and surface the security/overrefusal trade-off. See [`incident_triage_agent\README.md`](incident_triage_agent/README.md) and [`docs\case-study-incident-triage-joint.md`](../docs/case-study-incident-triage-joint.md). |
| Understand framework instrumentation breadth | `phoenix_auto_trace\README.md` | Same travel-planner idea across multiple framework auto-instrumentation paths. |
| Run a simple hosted-model eval | `pipes\health_assistant.yaml` | Good smoke test for a single LLM target with a system prompt. |
| Evaluate a Prompt Agent with planned tools but no backend | `pipes\health_assistant_simulated_tools.yaml` | Uses a fixed tool schema and simulated tool responses. |
| Evaluate a hosted target with Python tool functions | `pipes\health_assistant_sandbox.yaml` | Requires Docker. Use when you want actual tool execution around a hosted model. |
| Evaluate a science research agent with real retrieval tools | `science_research_agent\eval_config.yaml` | Callable-agent example ported from Omni. Uses `web_search`, `fetch_url`, and `file_search`. Run `python -m pip install -e ".[examples]"`, set `TAVILY_API_KEY` for web search, then `assert-eval run --config examples\science_research_agent\eval_config.yaml`. |
| Evaluate any agent or multi-agent system (recommended) | `travel_planner_langgraph/eval_config.yaml` | Flagship. Uses `target.callable` with `target.trace.backend: phoenix` so the judge sees tool calls and routing. |
| **See runtime + eval close the loop on a real workflow** | `incident_triage_agent/eval_config_baseline.yaml` + `eval_config_naive_prompt.yaml` + `eval_config_guarded.yaml` + `eval_config_guarded_gepa.yaml` | Joint AgentShield + ASSERT demo. SRE incident-triage agent run across a 4-variant matrix (baseline weak prompt → naïve DO-NOT prompt → ACS gates → ACS + GEPA-optimized prompt) over a 4-axis failure-mode taxonomy to prove the runtime+eval loop and surface the security/overrefusal trade-off. See [`incident_triage_agent/README.md`](incident_triage_agent/README.md) and [`docs/case-study-incident-triage-joint.md`](../docs/case-study-incident-triage-joint.md). |
| Understand framework instrumentation breadth | `phoenix_auto_trace/README.md` | Same travel-planner idea across multiple framework auto-instrumentation paths. |
| Run a simple hosted-model eval | `pipes/health_assistant.yaml` | Good smoke test for a single LLM target with a system prompt. |
| Evaluate a Prompt Agent with planned tools but no backend | `pipes/health_assistant_simulated_tools.yaml` | Uses a fixed tool schema and simulated tool responses. |
| Evaluate a hosted target with Python tool functions | `pipes/health_assistant_sandbox.yaml` | Requires Docker. Use when you want actual tool execution around a hosted model. |
| Evaluate a science research agent with real retrieval tools | `science_research_agent/eval_config.yaml` | Callable-agent example ported from Omni. Uses `web_search`, `fetch_url`, and `file_search`. Run `python -m pip install -e ".[examples]"`, set `TAVILY_API_KEY` for web search, then `assert-eval run --config examples/science_research_agent/eval_config.yaml`. |

## Layout

Expand Down
2 changes: 1 addition & 1 deletion examples/agents/openclaw/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Run the bundled example like this:
```bash
cp .env.example .env
source .env
uv run assert-eval run --config examples/pipes/health_assistant_external.yaml
assert-eval run --config examples/prompt_agents/health_assistant_external.yaml
```

If you want to validate the Docker assets without running the full pipeline, build the image directly:
Expand Down
36 changes: 18 additions & 18 deletions examples/incident_triage_agent/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,15 +190,15 @@ import DSPy at runtime.
### Run the demo path (A → C)

```powershell
assert-eval run --config examples\incident_triage_agent\eval_config_baseline.yaml
assert-eval run --config examples\incident_triage_agent\eval_config_guarded.yaml
assert-eval run --config examples/incident_triage_agent/eval_config_baseline.yaml
assert-eval run --config examples/incident_triage_agent/eval_config_guarded.yaml
```

### Run the appendix experiments (B and D)

```powershell
assert-eval run --config examples\incident_triage_agent\eval_config_naive_prompt.yaml
assert-eval run --config examples\incident_triage_agent\eval_config_guarded_gepa.yaml
assert-eval run --config examples/incident_triage_agent/eval_config_naive_prompt.yaml
assert-eval run --config examples/incident_triage_agent/eval_config_guarded_gepa.yaml
```

Artifacts land in (`run:` value used directly as the directory name):
Expand All @@ -214,7 +214,7 @@ After the variants finish, re-render the PNG from the real
`scores.jsonl` files:

```powershell
python scripts\render_trade_off.py --suite incident-triage-agent-v1
python scripts/render_trade_off.py --suite incident-triage-agent-v1
```

The script renders the demo-path variants (A and C) with solid markers
Expand Down Expand Up @@ -1154,18 +1154,18 @@ From this folder:
```bash
# Pre-req: AgentShield Python SDK 0.13.x installed (for the AFTER run only)
# and Azure OpenAI creds for gpt-5.4 and gpt-5.4-mini in your repo-root .env.
uv pip install agent-shield
python -m pip install agent-shield

# 1. BEFORE — minimal-prompt baseline.
uv run assert-eval run --config ./eval_config_baseline.yaml
assert-eval run --config ./eval_config_baseline.yaml

# 2. AFTER — same test cases, runtime guardrails engaged.
# (cached systematization/stratification/test_set; only inference + judge re-run)
uv run assert-eval run --config ./eval_config_guarded.yaml
assert-eval run --config ./eval_config_guarded.yaml

# 3. Compare.
uv run assert-eval results status incident-triage-agent-v1 baseline-weak-prompt
uv run assert-eval results status incident-triage-agent-v1 guarded-with-shield
assert-eval results status incident-triage-agent-v1 baseline-weak-prompt
assert-eval results status incident-triage-agent-v1 guarded-with-shield

# 4. Browse inference outputs.
cd ../../viewer && npm install && npm run dev
Expand Down Expand Up @@ -1271,24 +1271,24 @@ From this folder:

```bash
# 1. Install dependencies from the repository project.
uv sync
uv pip install agent-shield
python -m pip install -e .
python -m pip install agent-shield

# 2. Copy env templates, then fill in your Azure values.
cp ../../.env.example ../../.env
cp ./.env.example ./.env

# 3. Optional smoke test: unguarded, then guarded.
uv run python ./agent.py
uv run python ./agent_guarded.py
python ./agent.py
python ./agent_guarded.py

# 4. BEFORE — generate systematization, stratification, test_set, inference, and judge outputs.
uv run assert-eval run --config ./eval_config_baseline.yaml
uv run assert-eval results status incident-triage-agent-v1 baseline-weak-prompt
assert-eval run --config ./eval_config_baseline.yaml
assert-eval results status incident-triage-agent-v1 baseline-weak-prompt

# 5. AFTER — reuse the same test_set; rerun inference and judge against AgentShield.
uv run assert-eval run --config ./eval_config_guarded.yaml
uv run assert-eval results status incident-triage-agent-v1 guarded-with-shield
assert-eval run --config ./eval_config_guarded.yaml
assert-eval results status incident-triage-agent-v1 guarded-with-shield
```

Artifacts land under `artifacts/results/incident-triage-agent-v1/`. The suite-level files are `systematization.json`, `stratification.json`, and `test_set.jsonl`; each run writes `inference_set.jsonl`, `scores.jsonl`, and `metrics.json`.
2 changes: 1 addition & 1 deletion examples/phoenix_auto_trace/travel_langgraph.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
LLM call, tool invocation, and routing decision via Phoenix auto-instrumentation.

Usage:
uv run assert-eval run --config examples/travel_planner_langgraph/eval_config.yaml
assert-eval run --config examples/travel_planner_langgraph/eval_config.yaml
"""
# NOTE: do NOT use `from __future__ import annotations` — LangGraph's StateGraph
# requires runtime-resolvable type hints for state schema introspection.
Expand Down
2 changes: 1 addition & 1 deletion examples/travel_planner_langgraph/agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
→ safety_advisor → itinerary_optimizer

Usage:
uv run assert-eval run --config examples/travel_planner_langgraph/eval_config.yaml
assert-eval run --config examples/travel_planner_langgraph/eval_config.yaml
"""

from __future__ import annotations
Expand Down
3 changes: 0 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -87,9 +87,6 @@ dev = [
"pytest-timeout>=2.2.0",
]

[tool.uv]
index-url = "https://pypi.org/simple"

[project.scripts]
assert-eval = "assert_eval.cli:cli"

Expand Down
Loading
Loading