diff --git a/AGENTS.md b/AGENTS.md index a0c3b0a3d..925028b6c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -75,7 +75,7 @@ source .venv/bin/activate python -m pip install --upgrade pip python -m pip install -e ".[otel,langgraph]" cp .env.example .env -p2m run --config examples/travel_planner_langgraph/eval_config.yaml +assert-eval run --config examples/travel_planner_langgraph/eval_config.yaml ``` Use the PowerShell equivalent on Windows: @@ -86,7 +86,7 @@ python -m venv .venv python -m pip install --upgrade pip python -m pip install -e ".[otel,langgraph]" Copy-Item .env.example .env -p2m run --config examples/travel_planner_langgraph/eval_config.yaml +assert-eval run --config examples/travel_planner_langgraph/eval_config.yaml ``` ## How to help with common tasks @@ -99,7 +99,7 @@ p2m run --config examples/travel_planner_langgraph/eval_config.yaml 4. Add `dimensions` only when systematic variation matters. 5. Configure the target in `pipeline.inference.target`. 6. Add judge dimensions with concrete descriptions and rubrics. -7. Run `p2m run --config `. +7. Run `assert-eval run --config `. ### Debug a failure @@ -138,7 +138,7 @@ Adaptive Eval is a local-first, spec-driven evaluation pipeline for AI agents. T eval spec -> behavior categories -> test cases -> execute target -> judge -> artifacts Key facts: -- The CLI entrypoint is `p2m`. Configs live in `examples/`. Artifacts land in `artifacts/results///`. +- The canonical CLI entrypoint is `assert-eval`; `assert` and `p2m` remain backward-compatible aliases. Configs live in `examples/`. Artifacts land in `artifacts/results///`. - For any agent or multi-agent system with a Python entry function, use `target.callable` with `target.trace`. OpenTelemetry trace capture (Phoenix/OpenInference for 33+ frameworks, or your own OTel SDK spans) is the recommended integration path so the judge can score tool calls and routing, not just final text. - For a hosted model with a system prompt and optional tools, use `target.model` and `target.tools`. diff --git a/README.md b/README.md index 18cfe9136..5958e2763 100644 --- a/README.md +++ b/README.md @@ -40,17 +40,17 @@ cp .env.example .env phoenix serve # Run the full pipeline: spec -> taxonomy -> test cases -> execution -> verdicts. -p2m run --config examples/travel_planner_langgraph/eval_config.yaml +assert-eval run --config examples/travel_planner_langgraph/eval_config.yaml # Inspect the run. -p2m results status travel-planner-langgraph-v1 demo-1 +assert-eval results status travel-planner-langgraph-v1 demo-1 ``` Codespaces / VS Code Dev Containers: [![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/microsoft/adaptive-eval) -The repo includes a minimal dev container for the LangGraph quickstart. It installs `.[otel,langgraph,dev]`, copies `.env.example` to `.env` if needed, and forwards Phoenix on port `6006`. After the container finishes setup, add your provider credentials to `.env` and run the same `p2m run` command above. +The repo includes a minimal dev container for the LangGraph quickstart. It installs `.[otel,langgraph,dev]`, copies `.env.example` to `.env` if needed, and forwards Phoenix on port `6006`. After the container finishes setup, add your provider credentials to `.env` and run the same `assert-eval run` command above. Windows PowerShell equivalent: @@ -62,8 +62,8 @@ python -m pip install -e ".[otel,langgraph]" Copy-Item .env.example .env phoenix serve -p2m run --config examples/travel_planner_langgraph/eval_config.yaml -p2m results status travel-planner-langgraph-v1 demo-1 +assert-eval run --config examples/travel_planner_langgraph/eval_config.yaml +assert-eval results status travel-planner-langgraph-v1 demo-1 ``` What the quickstart does: diff --git a/docs/quickstart.md b/docs/quickstart.md index e413a9d85..a2d957b39 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -47,8 +47,8 @@ Copy-Item .env.example .env # any LiteLLM provider (OpenAI, Anthropic, Bedrock, Vertex, Ollama, …) works — see https://docs.litellm.ai/docs/providers. # Run the pipeline -p2m run --config examples\travel_planner_langgraph\eval_config.yaml -p2m results status travel-planner-langgraph-v1 demo-1 +assert-eval run --config examples\travel_planner_langgraph\eval_config.yaml +assert-eval results status travel-planner-langgraph-v1 demo-1 ``` > **Optional — browse traces in the Phoenix UI.** Span capture happens inside `auto_trace.py` regardless; running `phoenix serve` only adds an interactive UI for browsing them. In a separate terminal, before running the eval: diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 4ba27abe4..d3afe3126 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -5,13 +5,13 @@ Adaptive Eval is CLI-first. All commands assume your virtualenv is activated (se ## Run a config ```powershell -p2m run --config examples\travel_planner_langgraph\eval_config.yaml +assert-eval run --config examples\travel_planner_langgraph\eval_config.yaml ``` ## Re-run one stage ```powershell -p2m run --config examples\travel_planner_langgraph\eval_config.yaml --force-stage test_set +assert-eval run --config examples\travel_planner_langgraph\eval_config.yaml --force-stage test_set ``` Use this when you intentionally changed a stage input and want to regenerate downstream artifacts. @@ -19,19 +19,19 @@ Use this when you intentionally changed a stage input and want to regenerate dow ## List runs ```powershell -p2m results list +assert-eval results list ``` ## Show run status ```powershell -p2m results status travel-planner-langgraph-v1 demo-1 +assert-eval results status travel-planner-langgraph-v1 demo-1 ``` ## Compare runs ```powershell -p2m results compare +assert-eval results compare ``` ## Analyze generated test cases @@ -43,10 +43,10 @@ p2m results compare ```powershell # OpenAI backend (default) -p2m analysis test-set-metrics --taxonomy artifacts\results\\taxonomy.json --test_set artifacts\results\\test_set.jsonl +assert-eval analysis test-set-metrics --taxonomy artifacts\results\\taxonomy.json --test_set artifacts\results\\test_set.jsonl # Offline HuggingFace backend (no API key) -p2m analysis test-set-metrics --taxonomy artifacts\results\\taxonomy.json --test_set artifacts\results\\test_set.jsonl --embed-backend hf --embed-model all-MiniLM-L6-v2 +assert-eval analysis test-set-metrics --taxonomy artifacts\results\\taxonomy.json --test_set artifacts\results\\test_set.jsonl --embed-backend hf --embed-model all-MiniLM-L6-v2 ``` ## Where outputs go diff --git a/examples/README.md b/examples/README.md index 285115c23..bc8f9a983 100644 --- a/examples/README.md +++ b/examples/README.md @@ -17,8 +17,8 @@ Copy-Item .env.example .env # Edit .env with credentials for your provider. The shipped configs use `azure/...` models; # any LiteLLM provider (OpenAI, Anthropic, Bedrock, Vertex, Ollama, …) works — see https://docs.litellm.ai/docs/providers. -p2m run --config examples\travel_planner_langgraph\eval_config.yaml -p2m results status travel-planner-langgraph-v1 demo-1 +assert-eval run --config examples\travel_planner_langgraph\eval_config.yaml +assert-eval results status travel-planner-langgraph-v1 demo-1 ``` ## Which example to start with diff --git a/examples/agents/openclaw/README.md b/examples/agents/openclaw/README.md index 145432f0d..8d46387c6 100644 --- a/examples/agents/openclaw/README.md +++ b/examples/agents/openclaw/README.md @@ -9,7 +9,7 @@ Run the bundled example like this: ```bash cp .env.example .env source .env -uv run p2m run --config examples/pipes/health_assistant_external.yaml +uv run assert-eval run --config examples/pipes/health_assistant_external.yaml ``` If you want to validate the Docker assets without running the full pipeline, build the image directly: diff --git a/examples/azure_doc_qa/IMPROVEMENT_JOURNEY.md b/examples/azure_doc_qa/IMPROVEMENT_JOURNEY.md index 3b0815c24..5c152d543 100644 --- a/examples/azure_doc_qa/IMPROVEMENT_JOURNEY.md +++ b/examples/azure_doc_qa/IMPROVEMENT_JOURNEY.md @@ -31,7 +31,7 @@ cases across different question types and adversarial pressures. ### Step 1 — Run the baseline eval ```bash -USE_MOCK_TOOLS=1 p2m run --config examples/azure_doc_qa/eval_config.yaml +USE_MOCK_TOOLS=1 assert-eval run --config examples/azure_doc_qa/eval_config.yaml ``` The initial run showed a **~80% policy_violation rate** — nearly every test case @@ -83,7 +83,7 @@ Each fix was a small, focused commit: ### Step 5 — Re-evaluate ```bash -USE_MOCK_TOOLS=1 p2m run --config examples/azure_doc_qa/eval_config.yaml +USE_MOCK_TOOLS=1 assert-eval run --config examples/azure_doc_qa/eval_config.yaml ``` Result: **34/56 passing (61%)**, up from ~20%. The routing JSON leak was @@ -422,7 +422,7 @@ pip install -e ".[otel,langgraph]" cp .env.example .env # configure AZURE_API_BASE, AZURE_API_KEY # Run eval -USE_MOCK_TOOLS=1 p2m run --config examples/azure_doc_qa/eval_config.yaml +USE_MOCK_TOOLS=1 assert-eval run --config examples/azure_doc_qa/eval_config.yaml # Check results cat artifacts/results/azure-doc-qa-v1/demo-1/metrics.json diff --git a/examples/azure_doc_qa/README.md b/examples/azure_doc_qa/README.md index e69d875f5..d955ba4b9 100644 --- a/examples/azure_doc_qa/README.md +++ b/examples/azure_doc_qa/README.md @@ -43,7 +43,7 @@ pip install -e ".[otel,langgraph]" cp .env.example .env # set AZURE_API_BASE, AZURE_API_KEY, P2M_AZURE_DEPLOYMENT # Run eval with mock tools (offline, no MCP servers needed) -USE_MOCK_TOOLS=1 p2m run --config examples/azure_doc_qa/eval_config.yaml +USE_MOCK_TOOLS=1 assert-eval run --config examples/azure_doc_qa/eval_config.yaml ``` ## Real MCP Mode @@ -57,7 +57,7 @@ export FOUNDRY_IQ_TOKEN="your-bearer-token" # Node.js required for Learn MCP (npx -y @microsoftdocs/mcp) # Run without USE_MOCK_TOOLS (real MCP tools used) -p2m run --config examples/azure_doc_qa/eval_config.yaml +assert-eval run --config examples/azure_doc_qa/eval_config.yaml ``` ## Environment Variables diff --git a/examples/azure_doc_qa/agent.py b/examples/azure_doc_qa/agent.py index 2c27ca38f..2b787603a 100644 --- a/examples/azure_doc_qa/agent.py +++ b/examples/azure_doc_qa/agent.py @@ -8,10 +8,10 @@ Usage: # Real MCP mode (requires Azure auth + Node.js): - p2m run --config examples/azure_doc_qa/eval_config.yaml + assert-eval run --config examples/azure_doc_qa/eval_config.yaml # Mock mode (offline, no auth needed): - USE_MOCK_TOOLS=1 p2m run --config examples/azure_doc_qa/eval_config.yaml + USE_MOCK_TOOLS=1 assert-eval run --config examples/azure_doc_qa/eval_config.yaml """ from __future__ import annotations diff --git a/examples/incident_triage_agent/README.md b/examples/incident_triage_agent/README.md index a280cafde..5ac9a4012 100644 --- a/examples/incident_triage_agent/README.md +++ b/examples/incident_triage_agent/README.md @@ -190,15 +190,15 @@ import DSPy at runtime. ### Run the demo path (A → C) ```powershell -p2m run --config examples\incident_triage_agent\eval_config_baseline.yaml -p2m run --config examples\incident_triage_agent\eval_config_guarded.yaml +assert-eval run --config examples\incident_triage_agent\eval_config_baseline.yaml +assert-eval run --config examples\incident_triage_agent\eval_config_guarded.yaml ``` ### Run the appendix experiments (B and D) ```powershell -p2m run --config examples\incident_triage_agent\eval_config_naive_prompt.yaml -p2m run --config examples\incident_triage_agent\eval_config_guarded_gepa.yaml +assert-eval run --config examples\incident_triage_agent\eval_config_naive_prompt.yaml +assert-eval run --config examples\incident_triage_agent\eval_config_guarded_gepa.yaml ``` Artifacts land in (`run:` value used directly as the directory name): @@ -232,7 +232,7 @@ still produces a sensible chart. directory (`artifacts/results/incident-triage-agent-v1/`) and reuse them across variants (per `CONFIG_REFERENCE.md`, "Suite-level stages write versioned artifacts under the suite directory and are shared - across runs"). In practice: the first `p2m run` (any variant) + across runs"). In practice: the first `assert-eval run` (any variant) generates `test_set.jsonl` once (n=200 prompt + n=200 scenario); the remaining runs detect the cached test set and only re-run `inference` and `judge` against the same 400 test cases. Cross-variant comparison @@ -1157,15 +1157,15 @@ From this folder: uv pip install agent-shield # 1. BEFORE — minimal-prompt baseline. -uv run p2m run --config ./eval_config_baseline.yaml +uv run assert-eval run --config ./eval_config_baseline.yaml # 2. AFTER — same test cases, runtime guardrails engaged. # (cached systematization/stratification/test_set; only inference + judge re-run) -uv run p2m run --config ./eval_config_guarded.yaml +uv run assert-eval run --config ./eval_config_guarded.yaml # 3. Compare. -uv run p2m results status incident-triage-agent-v1 baseline-weak-prompt -uv run p2m results status incident-triage-agent-v1 guarded-with-shield +uv run assert-eval results status incident-triage-agent-v1 baseline-weak-prompt +uv run assert-eval results status incident-triage-agent-v1 guarded-with-shield # 4. Browse inference outputs. cd ../../viewer && npm install && npm run dev @@ -1282,12 +1282,12 @@ uv run python ./agent.py uv run python ./agent_guarded.py # 4. BEFORE — generate systematization, stratification, test_set, inference, and judge outputs. -uv run p2m run --config ./eval_config_baseline.yaml -uv run p2m results status incident-triage-agent-v1 baseline-weak-prompt +uv run assert-eval run --config ./eval_config_baseline.yaml +uv run assert-eval results status incident-triage-agent-v1 baseline-weak-prompt # 5. AFTER — reuse the same test_set; rerun inference and judge against AgentShield. -uv run p2m run --config ./eval_config_guarded.yaml -uv run p2m results status incident-triage-agent-v1 guarded-with-shield +uv run assert-eval run --config ./eval_config_guarded.yaml +uv run assert-eval results status incident-triage-agent-v1 guarded-with-shield ``` Artifacts land under `artifacts/results/incident-triage-agent-v1/`. The suite-level files are `systematization.json`, `stratification.json`, and `test_set.jsonl`; each run writes `inference_set.jsonl`, `scores.jsonl`, and `metrics.json`. diff --git a/examples/incident_triage_simple/README.md b/examples/incident_triage_simple/README.md index 7f2224997..e6a93111e 100644 --- a/examples/incident_triage_simple/README.md +++ b/examples/incident_triage_simple/README.md @@ -22,12 +22,12 @@ uv sync cp ./.env.example ./.env # BEFORE — bare agent -uv run p2m run --config ./eval_config.yaml --run before +uv run assert-eval run --config ./eval_config.yaml --run before # AFTER — switch target.callable in eval_config.yaml to # examples.incident_triage_simple.agent_guarded:chat, then rerun uv pip install agent-shield # required for the AFTER run only -uv run p2m run --config ./eval_config.yaml --run after +uv run assert-eval run --config ./eval_config.yaml --run after ``` To disable the aux classifier (e.g. offline/no-Azure): `INCIDENT_TRIAGE_AUX_DISABLED=1`. diff --git a/examples/phoenix_auto_trace/travel_langgraph.py b/examples/phoenix_auto_trace/travel_langgraph.py index 2aebef28a..c31b77c13 100644 --- a/examples/phoenix_auto_trace/travel_langgraph.py +++ b/examples/phoenix_auto_trace/travel_langgraph.py @@ -4,7 +4,7 @@ LLM call, tool invocation, and routing decision via Phoenix auto-instrumentation. Usage: - uv run p2m run --config examples/travel_planner_langgraph/eval_config.yaml + uv run assert-eval run --config examples/travel_planner_langgraph/eval_config.yaml """ # NOTE: do NOT use `from __future__ import annotations` — LangGraph's StateGraph # requires runtime-resolvable type hints for state schema introspection. diff --git a/examples/pipes/README.md b/examples/pipes/README.md index deb854ecd..04e42962e 100644 --- a/examples/pipes/README.md +++ b/examples/pipes/README.md @@ -3,7 +3,7 @@ Run any config with: ```powershell -p2m run --config examples/pipes/.yaml +assert-eval run --config examples/pipes/.yaml ``` (Assumes your virtualenv is activated. See the [README](../../README.md#quickstart-langgraph-travel-planner-any-agent-works-the-same-way) for setup.) diff --git a/examples/travel_planner_langgraph/agent.py b/examples/travel_planner_langgraph/agent.py index 3b30684f0..20c6d6ec1 100644 --- a/examples/travel_planner_langgraph/agent.py +++ b/examples/travel_planner_langgraph/agent.py @@ -6,7 +6,7 @@ → safety_advisor → itinerary_optimizer Usage: - uv run p2m run --config examples/travel_planner_langgraph/eval_config.yaml + uv run assert-eval run --config examples/travel_planner_langgraph/eval_config.yaml """ from __future__ import annotations diff --git a/examples/travel_planner_neurosan/README.md b/examples/travel_planner_neurosan/README.md index 20bd57a24..12e7ac0c0 100644 --- a/examples/travel_planner_neurosan/README.md +++ b/examples/travel_planner_neurosan/README.md @@ -40,7 +40,7 @@ Each "agent" is a plain Python function. OTel spans are created manually with ## Running ```bash -uv run p2m run --config examples/travel_planner_neurosan/eval_config.yaml +uv run assert-eval run --config examples/travel_planner_neurosan/eval_config.yaml ``` ## What the judge sees diff --git a/p2m/cli.py b/p2m/cli.py index 99b6b724b..506f83a38 100644 --- a/p2m/cli.py +++ b/p2m/cli.py @@ -473,14 +473,14 @@ def _behavior_category_metric_map(rows: Iterable[dict[str, Any]], metric: str) - epilog=( "\b\n" "Examples:\n" - " p2m run --config examples/pipes/health_assistant.yaml\n" - " p2m run --config examples/pipes/health_assistant_external.yaml\n" - " p2m results list\n" - " p2m results compare health-assistant-v1 gpt54-eval gpt54-eval\n" - " p2m results compare-suites suite-a/run-1 suite-b/run-1 suite-c/run-1" + " assert-eval run --config examples/pipes/health_assistant.yaml\n" + " assert-eval run --config examples/pipes/health_assistant_external.yaml\n" + " assert-eval results list\n" + " assert-eval results compare health-assistant-v1 gpt54-eval gpt54-eval\n" + " assert-eval results compare-suites suite-a/run-1 suite-b/run-1 suite-c/run-1" ), ) -@click.version_option(version="0.1.0", prog_name="p2m") +@click.version_option(version="0.1.0", prog_name="assert-eval") @click.option("-v", "--verbose", is_flag=True, help="Enable debug-level logging.") @click.option("-q", "--quiet", is_flag=True, help="Suppress info-level output; show only warnings and errors.") @click.option( @@ -561,7 +561,7 @@ def run( ): """Run the evaluation pipeline.""" # Re-configure logging if flags were passed on the subcommand - # (e.g. `p2m run --verbose` instead of `p2m --verbose run`). + # (e.g. `assert-eval run --verbose` instead of `assert-eval --verbose run`). if verbose or quiet or log_file or output_format != "text": configure_logging( verbose=verbose, @@ -860,14 +860,14 @@ def results_compare( """Compare runs. Accepts two forms: \b - Within one suite: p2m results compare SUITE RUN1 RUN2 - Cross-suite: p2m results compare SUITE/RUN1 SUITE/RUN2 + Within one suite: assert-eval results compare SUITE RUN1 RUN2 + Cross-suite: assert-eval results compare SUITE/RUN1 SUITE/RUN2 """ if len(args) < 2: _error( "Provide at least two arguments.\n" - " Within suite: p2m results compare SUITE RUN1 RUN2\n" - " Cross-suite: p2m results compare SUITE/RUN1 SUITE/RUN2" + " Within suite: assert-eval results compare SUITE RUN1 RUN2\n" + " Cross-suite: assert-eval results compare SUITE/RUN1 SUITE/RUN2" ) # Detect cross-suite mode: any arg contains "/" @@ -892,7 +892,7 @@ def results_compare( _error( f"'{runs[0]}' looks like a suite name, not a run ID.\n" f"Use slash format for cross-suite:\n" - f" p2m results compare {suite}/run-1 {runs[0]}/run-1" + f" assert-eval results compare {suite}/run-1 {runs[0]}/run-1" ) _error("Provide at least two run IDs to compare.") @@ -1075,7 +1075,7 @@ def results_compare_suites( \b Examples: - p2m results compare-suites \\ + assert-eval results compare-suites \\ travel-planner-phoenix-otel-demo/run-1 \\ travel-planner-litellm-callable/run-1 \\ travel-planner-external-connector/run-1 diff --git a/p2m/core/artifact_cache.py b/p2m/core/artifact_cache.py index f4cff5d31..ecd8b95e4 100644 --- a/p2m/core/artifact_cache.py +++ b/p2m/core/artifact_cache.py @@ -53,7 +53,7 @@ # Bound on the version-allocation retry loop. Each retry rescans the stage # directory, so the only legitimate reason to exhaust this budget is a -# pathologically high concurrent allocation rate (hundreds of `p2m run` +# pathologically high concurrent allocation rate (hundreds of `assert-eval run` # invocations against the same suite hitting the same window). At that point # we'd rather raise loudly than silently misnumber. _MAX_VERSION_ALLOCATION_RETRIES = 100 @@ -412,7 +412,7 @@ def discard_artifact_plan(ctx: dict[str, Any], plan: ArtifactPlan) -> None: for a non-reused plan is always uniquely owned by this process: the slot was reserved by ``mkdir(exist_ok=False)`` in ``prepare_artifact_plan``. ``rmtree`` here therefore only removes content this process produced, - even when other ``p2m run`` invocations are racing on the same suite. + even when other ``assert-eval run`` invocations are racing on the same suite. """ if plan.reused: @@ -862,7 +862,7 @@ def _allocate_version_dir(stage_root: Path) -> tuple[str, Path]: Computes the next version number from the existing directory listing, then attempts ``mkdir(exist_ok=False)`` for the candidate path. If a - concurrent ``p2m run`` allocated the same number first (FileExistsError), + concurrent ``assert-eval run`` allocated the same number first (FileExistsError), we re-scan and retry with the new max. This closes the time-of-check/time-of-use window between the directory scan and the eventual on-disk write that previously allowed two concurrent pipelines diff --git a/p2m/core/runtime_safety.py b/p2m/core/runtime_safety.py index 0f4aaf6b0..ad6f2bbb7 100644 --- a/p2m/core/runtime_safety.py +++ b/p2m/core/runtime_safety.py @@ -13,7 +13,7 @@ * :class:`ManifestHeartbeat` — daemon thread that rewrites ``manifest.heartbeat_at`` (and an optional progress payload) every - ``interval_s`` seconds so external observers (``p2m results status``, + ``interval_s`` seconds so external observers (``assert-eval results status``, benchmark dashboards) get an honest liveness signal during long stages. * :class:`PipelineWatchdog` — daemon thread that dumps every Python thread's current stack to the log if the pipeline goes silent (no diff --git a/p2m/runner.py b/p2m/runner.py index a403898a0..6e2f5b4af 100644 --- a/p2m/runner.py +++ b/p2m/runner.py @@ -866,7 +866,7 @@ def _run_stages_inner( run_id = ctx.get('run_id', '') if suite_id and run_id: log.info("Inspect results:") - log.info(f" uv run p2m results status {suite_id} {run_id}") + log.info(f" uv run assert-eval results status {suite_id} {run_id}") else: log.error(f"Pipeline failed at {failed_stage} ({total_elapsed:.1f}s)") diff --git a/pyproject.toml b/pyproject.toml index 9334a01e7..878dcd7ea 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -91,8 +91,9 @@ dev = [ index-url = "https://pypi.org/simple" [project.scripts] -p2m = "p2m.cli:cli" +assert-eval = "p2m.cli:cli" assert = "p2m.cli:cli" +p2m = "p2m.cli:cli" [build-system] requires = ["setuptools>=61.0"] diff --git a/scripts/README.md b/scripts/README.md index 7df4b380f..6adc2c8b3 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -4,16 +4,16 @@ Run scripts in this directory with `uv run python ...` from the repo root so the ## Seed sampling -Test-set stratification and generation run through `p2m run` now. Start from +Test-set stratification and generation run through `assert-eval run` now. Start from `examples/pipes/health_assistant.yaml`, keep `pipeline.test_set.stratify` and `pipeline.test_set`, then run: ```bash source .env -uv run p2m run --config examples/pipes/health_assistant.yaml +uv run assert-eval run --config examples/pipes/health_assistant.yaml ``` -Use `uv run p2m --help` for CLI options. +Use `uv run assert-eval --help` for CLI options. ## `benchmark.py` diff --git a/scripts/benchmark.py b/scripts/benchmark.py index c10e7e483..bfd653765 100644 --- a/scripts/benchmark.py +++ b/scripts/benchmark.py @@ -538,7 +538,7 @@ def main(argv: list[str] | None = None) -> int: print(f" log file : {args.log_file}") print("=" * 72, flush=True) - # Configure logging the same way `p2m run` does so stage progress and + # Configure logging the same way `assert-eval run` does so stage progress and # any failure output actually reaches the terminal. Must run BEFORE # we attach the rate-limit counter, because configure_logging clears # existing handlers on the root logger. diff --git a/tests/test_artifact_cache.py b/tests/test_artifact_cache.py index 354248a49..3f0578116 100644 --- a/tests/test_artifact_cache.py +++ b/tests/test_artifact_cache.py @@ -636,7 +636,7 @@ def test_next_version_does_not_leak_after_discard(self) -> None: class AllocateVersionDirTest(unittest.TestCase): """Atomic ``vNNNN`` reservation under simulated concurrency. - Regression for the race in ``_next_version``: two ``p2m run`` + Regression for the race in ``_next_version``: two ``assert-eval run`` invocations on the same suite both read ``max(numbers) + 1``, both pick the same slot, and silently corrupt each other's outputs. ``_allocate_version_dir`` closes the time-of-check/time-of-use window diff --git a/tests/test_library_e2e.py b/tests/test_library_e2e.py index b9c847d84..756394440 100644 --- a/tests/test_library_e2e.py +++ b/tests/test_library_e2e.py @@ -174,7 +174,7 @@ def test_judge_tags_is_list_of_strings(self): # =================================================================== class CliLibraryListTest(unittest.TestCase): - """Test the ``p2m library list`` CLI command end-to-end.""" + """Test the ``assert-eval library list`` CLI command end-to-end.""" def setUp(self): self.runner = CliRunner() @@ -243,7 +243,7 @@ def test_list_json_filter_judge(self): # =================================================================== class CliLibraryShowTest(unittest.TestCase): - """Test the ``p2m library show`` CLI command end-to-end.""" + """Test the ``assert-eval library show`` CLI command end-to-end.""" def setUp(self): self.runner = CliRunner() diff --git a/viewer/README.md b/viewer/README.md index 8790cd65f..af55ef88e 100644 --- a/viewer/README.md +++ b/viewer/README.md @@ -5,7 +5,7 @@ Web app for browsing measurement results. It reads directly from `artifacts/resu ## Prerequisites - **Node.js 18+** -- Evaluation results in `artifacts/results/` (generated by `p2m run`) +- Evaluation results in `artifacts/results/` (generated by `assert-eval run`) ## Developing @@ -68,7 +68,7 @@ Completed judged runs are served from the run-level viewer read model, not by sc ```sh cd .. -uv run p2m run --config artifacts/results///config.yaml --resume --force-stage judge +uv run assert-eval run --config artifacts/results///config.yaml --resume --force-stage judge ``` The viewer expects each successful score row to use the strict judge verdict contract: diff --git a/viewer/src/lib/server/artifacts.ts b/viewer/src/lib/server/artifacts.ts index 65a023c5f..d512a65d7 100644 --- a/viewer/src/lib/server/artifacts.ts +++ b/viewer/src/lib/server/artifacts.ts @@ -238,7 +238,7 @@ function runSeedRows( function rebuildViewerInstruction(runDir: string): string { const configPath = path.resolve(runDir, RUN_CONFIG_FILE); - return `Rebuild it by re-running judge for this run: uv run p2m run --config ${configPath} --resume --force-stage judge`; + return `Rebuild it by re-running judge for this run: uv run assert-eval run --config ${configPath} --resume --force-stage judge`; } function validateViewerFileMetadata( diff --git a/viewer/src/lib/server/run-spawn.ts b/viewer/src/lib/server/run-spawn.ts index 77faee5b9..3be159790 100644 --- a/viewer/src/lib/server/run-spawn.ts +++ b/viewer/src/lib/server/run-spawn.ts @@ -19,7 +19,7 @@ * lives inline in behavior.description) * * spawnP2mRun(...) - * -> spawns `p2m run --config ` detached + * -> spawns `assert-eval run --config ` detached * -> waits for the OS spawn/error event before resolving so a missing * binary surfaces as HTTP 500 (not 200 then a forever-pending monitor) * @@ -228,7 +228,7 @@ export function normalizeWizardPayload(raw: unknown): NormalizedRun { errors.push( 'evaluationTarget "agent" is not yet supported by the UI submit path. ' + 'The wizard does not collect a Python callable target. ' + - 'For now, run agent evaluations via the CLI: `p2m run --config `.' + 'For now, run agent evaluations via the CLI: `assert-eval run --config `.' ); } @@ -563,40 +563,41 @@ interface ResolvedCommand { function resolveP2mCommand(configPath: string): ResolvedCommand { const cliArgs = ['run', '--config', configPath]; - const override = process.env.P2M_COMMAND; + const override = process.env.ASSERT_EVAL_COMMAND ?? process.env.P2M_COMMAND; if (override && override.trim()) { const parts = override.trim().split(/\s+/); return { command: parts[0], args: [...parts.slice(1), ...cliArgs], - source: 'P2M_COMMAND override' + source: process.env.ASSERT_EVAL_COMMAND ? 'ASSERT_EVAL_COMMAND override' : 'P2M_COMMAND override' }; } const venv = process.env.VIRTUAL_ENV; if (venv) { - const pythonBin = + const cliCandidates = os.platform() === 'win32' - ? path.join(venv, 'Scripts', 'python.exe') - : path.join(venv, 'bin', 'python'); - if (fs.existsSync(pythonBin)) { + ? [path.join(venv, 'Scripts', 'assert-eval.exe'), path.join(venv, 'Scripts', 'assert-eval')] + : [path.join(venv, 'bin', 'assert-eval')]; + const cliPath = cliCandidates.find((candidate) => fs.existsSync(candidate)); + if (cliPath) { return { - command: pythonBin, - args: ['-m', 'p2m.cli', ...cliArgs], - source: `VIRTUAL_ENV (${pythonBin})` + command: cliPath, + args: cliArgs, + source: `VIRTUAL_ENV (${cliPath})` }; } } // Fallback: PATH lookup. On Windows, .exe extension is resolved automatically // by spawn when shell:false because Node uses CreateProcess search behavior. - return { command: 'p2m', args: cliArgs, source: 'PATH' }; + return { command: 'assert-eval', args: cliArgs, source: 'PATH' }; } /** - * Spawn p2m detached, wait for the OS to confirm the spawn (or fail). Only - * after we hear back do we resolve — that way a missing `p2m` binary surfaces - * as a 500 instead of a 200 followed by a forever-pending monitor. + * Spawn assert-eval detached, wait for the OS to confirm the spawn (or fail). + * Only after we hear back do we resolve — that way a missing `assert-eval` + * binary surfaces as a 500 instead of a 200 followed by a forever-pending monitor. */ export function spawnP2mRun(written: WrittenRun): Promise { const resolved = resolveP2mCommand(written.configPath); @@ -609,7 +610,7 @@ export function spawnP2mRun(written: WrittenRun): Promise { } const preamble = - `# p2m run launched by viewer at ${new Date().toISOString()}\n` + + `# assert-eval run launched by viewer at ${new Date().toISOString()}\n` + `# command: ${resolved.command} ${resolved.args.join(' ')}\n` + `# resolved from: ${resolved.source}\n` + `# cwd: ${MEASUREMENTS_ROOT}\n` + @@ -638,7 +639,7 @@ export function spawnP2mRun(written: WrittenRun): Promise { } reject( new SpawnError( - `Failed to spawn p2m runner via ${resolved.source}: ${(err as Error).message ?? String(err)}`, + `Failed to spawn assert-eval runner via ${resolved.source}: ${(err as Error).message ?? String(err)}`, err ) ); @@ -680,9 +681,9 @@ export function spawnP2mRun(written: WrittenRun): Promise { } reject( new SpawnError( - `p2m runner failed to start via ${resolved.source}: ${err?.message ?? String(err)}. ` + + `assert-eval runner failed to start via ${resolved.source}: ${err?.message ?? String(err)}. ` + `Ensure the viewer was started in a shell with the project venv activated, ` + - `or set P2M_COMMAND to a working invocation (e.g. "python -m p2m.cli").`, + `or set ASSERT_EVAL_COMMAND to a working invocation (e.g. "assert-eval").`, err ) ); diff --git a/viewer/src/routes/api/runs/+server.ts b/viewer/src/routes/api/runs/+server.ts index 90c11ed44..fcde044a7 100644 --- a/viewer/src/routes/api/runs/+server.ts +++ b/viewer/src/routes/api/runs/+server.ts @@ -39,7 +39,7 @@ export const GET: RequestHandler = async () => { * 2. Refuse with 409 if a run with the same suite/run already exists. * 3. mkdir the run directory atomically; write eval_config.yaml (behavior * spec lives inline in behavior.description — no separate spec file). - * 4. Spawn `p2m run` detached; wait for the OS to acknowledge the spawn + * 4. Spawn `assert-eval run` detached; wait for the OS to acknowledge the spawn * so a missing binary surfaces as 500 (not 200 + forever-pending monitor). * 5. Return { suiteId, runId, pid, warnings } so the wizard can navigate * to /suite///monitor and start polling status. @@ -107,7 +107,7 @@ export const POST: RequestHandler = async ({ request }) => { const message = err instanceof SpawnError ? err.message : (err as Error).message ?? String(err); return json( { - error: 'Failed to start the p2m runner.', + error: 'Failed to start the assert-eval runner.', details: [message], // Surface the partially-written run dir so the user can inspect or clean up. runDir: written.runDir diff --git a/viewer/src/routes/new/+page.svelte b/viewer/src/routes/new/+page.svelte index 7dbf16ecd..03718b0ca 100644 --- a/viewer/src/routes/new/+page.svelte +++ b/viewer/src/routes/new/+page.svelte @@ -8,7 +8,7 @@ * - reserves artifacts/results/// atomically * - writes eval_config.yaml (single-YAML authoring; behavior description * lives inline in behavior.description) - * - spawns `p2m run --config ` detached + * - spawns `assert-eval run --config ` detached * On success the wizard navigates to /suite///monitor. */ import { onMount } from 'svelte'; diff --git a/viewer/src/routes/suite/[suite_id]/[run_id]/+page.svelte b/viewer/src/routes/suite/[suite_id]/[run_id]/+page.svelte index 76d16059f..1f77e0bca 100644 --- a/viewer/src/routes/suite/[suite_id]/[run_id]/+page.svelte +++ b/viewer/src/routes/suite/[suite_id]/[run_id]/+page.svelte @@ -862,7 +862,7 @@ {/if}

- uv run p2m run --config <config> + uv run assert-eval run --config <config>

{:else}