Rename assert-eval -> assert-ai by tangym · Pull Request #177 · responsibleai/ASSERT

tangym · 2026-06-01T01:52:35Z

Summary

Mechanical rename of the project from assert-eval / assert_eval to assert-ai / assert_ai. No behavior changes.

Surface	Before	After
Module / package	`assert_eval`	`assert_ai`
Distribution	`assert-eval`	`assert-ai`
CLI entry point	`assert-eval`	`assert-ai`

Commit breakdown (8 focused commits, prep → core → tests → docs → infra)

fe72649f — rename: assert_eval -> assert_ai (module dir + pyproject) — git mv of the package directory + pyproject.toml field renames.
ebcef50b — rename: update intra-package imports to assert_ai — 29 nested .py files inside the renamed package.
5d5d3ad7 — rename: update tests for assert_ai — 51 test files.
3ec7ac3f — rename: update top-level assert_ai modules — 6 top-level .py files inside the package.
3509e003 — rename: update docs, examples, scripts, viewer, and website — 41 files across docs/examples/scripts/viewer/website + assert_ai/analysis/README.md.
beab3055 — rename: update CI workflow path filter — .github/workflows/regression.yml path filter.
64ff6fd8 — rename: update .gitignore policy artifact pattern — assert_eval_policy.* → assert_ai_policy.*.
5faddeea — chore: rename root package in uv.lock to assert-ai — hand-edited 4 root-package self-references in uv.lock (see caveat below).

Validation

✅ Zero residual assert[-_]eval occurrences across all tracked files (git ls-files | xargs grep -lE "assert[-_]eval" is empty).
✅ python -m build succeeds — produces assert_ai-0.1.0-py3-none-any.whl and assert_ai-0.1.0.tar.gz with all 87 package files, correct entry_points.txt, top_level.txt, METADATA, and RECORD.
✅ Fresh editable install (pip install -e .) registers only the assert-ai console script. assert-ai --help and assert-ai --version both succeed.
✅ pytest — 950 passed, 14 skipped. The single failure (tests/test_tool_module_sandbox.py::HealthAssistantSandboxExampleTest::test_health_assistant_tools_use_per_conversation_workspace) is PermissionError: 'docker' and reproduces on origin/main — environmental, unrelated to this PR.

Caveats / out-of-scope follow-ups

These are pre-existing on main (verified) and are intentionally not addressed here to keep this PR scoped to a pure rename:

uv lock cannot regenerate because assert-ai[dspy] requires dspy-ai>=2.7,<3 while only <2.7 and >=3 are available on PyPI. I left the rest of uv.lock untouched and only hand-edited the 4 root-package self-references via sed. Resolving the dspy-ai version constraint belongs in a separate PR.
Wheel install does not expose prompts. internal-pipeline-prompts/ lives at the repo root and is not packaged into the wheel; BASE_DIR = Path(__file__).resolve().parents[2] in assert_ai/core/io.py resolves correctly in editable installs (= repo root) but points outside the install in wheel installs. Will land in a follow-up PR.

Reviewer notes

Each commit is independently reviewable — diff-by-commit recommended.
All file moves use git mv so history is preserved.

Renames the package directory and updates the five distribution-name and module-name references in pyproject.toml (name, [project.scripts], all extra, dev dependency-group, and setuptools packages.find). Also drops examples* from the wheel per the PyPI publishing plan. Imports inside the package are not yet updated -- follow-up commit.

Replaces assert_eval / assert-eval references inside the moved package so it imports cleanly. Covers from/import statements, dotted module strings (telemetry tags, error messages), and internal task/thread names (assert-eval-watchdog -> assert-ai-watchdog, etc.). Tests, docs, scripts, and CI updated separately.

Replaces assert_eval / assert-eval references in tests/ — module patches, dotted import paths, CLI invocations, and any expected log/identifier strings.

Catches the depth-1 .py files in assert_ai/ that the package-imports commit missed: cli.py, config.py, display.py, results.py, runner.py, viewer_read_model.py. Same dual-pattern (assert_eval -> assert_ai, assert-eval -> assert-ai).

Sweeps assert_eval -> assert_ai and assert-eval -> assert-ai across user-facing surfaces: root README and AGENTS, docs/ tree, runnable examples (including notebooks and agent scripts), helper scripts, the SvelteKit viewer, the marketing website, and the analysis README that ships inside the wheel.

Path filter in .github/workflows/regression.yml now points at assert_ai/** instead of assert_eval/**.

Renames assert_eval_policy.* -> assert_ai_policy.* so runtime policy artifacts stay ignored after the package rename.

Hand-edits the 4 self-references to the root project in uv.lock to match the renamed distribution. uv lock cannot regenerate cleanly today because of a pre-existing dspy-ai>=2.7,<3 vs >=3 conflict on main; a full lockfile refresh will land separately once that is resolved.

Copilot

Pull request overview

Mechanical rename of the Python distribution/module and CLI from assert-eval / assert_eval to assert-ai / assert_ai across the repo as the first step toward PyPI publishing, updating imports, docs, examples, viewer/website UI strings, and CI path filters accordingly.

Changes:

Renamed Python package/import surface to assert_ai and CLI entry point to assert-ai (plus widespread import/string updates).
Updated viewer + website UI copy and viewer runner-spawn logic to invoke assert-ai / python -m assert_ai.cli.
Introduced/renamed preset library content under assert_ai/library/* and updated config/CLI/docs to reference the new namespace.

Reviewed changes

Copilot reviewed 130 out of 181 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
website/app/page.tsx	Website terminal snippet updated for the renamed CLI (needs hyphenated `assert-ai`).
viewer/src/routes/suite/[suite_id]/[run_id]/+page.svelte	Updated displayed CLI example to `assert-ai`.
viewer/src/routes/new/+page.svelte	Updated inline docs/comments to `assert-ai`.
viewer/src/routes/api/runs/+server.ts	Updated comments/error text for renamed runner.
viewer/src/lib/server/run-spawn.ts	Updated spawn resolution to use `assert-ai` and `-m assert_ai.cli`.
viewer/src/lib/server/artifacts.ts	Updated rebuild instruction to `assert-ai`.
uv.lock	Renamed root package references to `assert-ai`.
tests/test_viewer_server_artifacts.py	Updated imports to `assert_ai`.
tests/test_viewer_read_model_turns.py	Updated imports to `assert_ai`.
tests/test_turn_checkpoint_judge.py	Updated imports to `assert_ai`.
tests/test_tool_module_sandbox.py	Updated imports/patch paths to `assert_ai`.
tests/test_tester_target_loop.py	Updated imports/patch paths to `assert_ai`.
tests/test_tester_pairwise_eval.py	Updated imports to `assert_ai`.
tests/test_test_set.py	Updated imports and inline imports to `assert_ai`.
tests/test_test_set_stage.py	Updated imports/patch paths to `assert_ai`.
tests/test_test_set_metrics_analysis.py	Updated imports to `assert_ai`.
tests/test_test_case_sampling_characterization.py	Updated imports to `assert_ai`.
tests/test_systematize_stage.py	Updated imports/patch paths to `assert_ai`.
tests/test_systematization_stage.py	Updated imports/patch paths to `assert_ai`.
tests/test_systematization_convert_stage.py	Updated imports/patch paths to `assert_ai`.
tests/test_stage_runner_smoke.py	Updated imports/patch paths to `assert_ai`.
tests/test_shared_infra_helpers.py	Updated imports/patch paths to `assert_ai`.
tests/test_security.py	Updated module references and patch targets to `assert_ai`.
tests/test_schema_contracts.py	Updated imports to `assert_ai`.
tests/test_runtime_safety.py	Updated imports and logger names to `assert_ai.*`.
tests/test_runtime_modes.py	Updated imports/patch targets to `assert_ai`.
tests/test_runner_usage_metrics.py	Updated imports to `assert_ai`.
tests/test_runner_stage_filters.py	Updated imports/patch targets and comments to `assert_ai`.
tests/test_runner_progress.py	Updated imports/patch targets and logger names to `assert_ai.*`.
tests/test_runner_artifact_cache.py	Updated imports/patch targets to `assert_ai`.
tests/test_run_metadata.py	Updated imports/patch targets to `assert_ai`.
tests/test_results.py	Updated imports to `assert_ai`.
tests/test_rate_limit_retry.py	Updated imports/patch targets to `assert_ai`.
tests/test_preset_integration.py	Updated imports to `assert_ai`.
tests/test_no_p2m_references.py	Updated rename guidance strings to `assert_ai` / `assert-ai`.
tests/test_model_client.py	Updated imports/logger names to `assert_ai.*`.
tests/test_logging_config.py	Updated imports to `assert_ai`.
tests/test_library_loader.py	Updated imports to `assert_ai`.
tests/test_library_e2e.py	Updated imports and CLI docstrings to `assert-ai`.
tests/test_io.py	Updated imports/logger names to `assert_ai.*`.
tests/test_init_validate.py	Updated imports and docstring to `assert-ai init`.
tests/test_init_parse_action.py	Updated imports and docstring to `assert-ai init`.
tests/test_init_emit.py	Updated imports and docstring to `assert-ai init`.
tests/test_init_design_agent.py	Updated imports/patch targets to `assert_ai`.
tests/test_init_context.py	Updated imports and docstring to `assert-ai init`.
tests/test_init_command.py	Updated imports/patch targets and docstring to `assert-ai init`.
tests/test_import_smoke.py	Updated import smoke targets to `assert_ai.*`.
tests/test_hosted_trace_registration.py	Updated imports/patch targets to `assert_ai`.
tests/test_display_labels.py	Updated imports and file-path probes to `assert_ai`.
tests/test_cli.py	Updated imports/patch targets to `assert_ai`.
tests/test_benchmark_summary.py	Updated doc refs from `assert_eval` to `assert_ai`.
tests/test_artifact_cache.py	Updated imports/logger names to `assert_ai.*`.
tests/test_analysis.py	Updated imports and suite_analysis import paths to `assert_ai`.
scripts/turn_checkpoint_judge.py	Updated imports to `assert_ai`.
scripts/tester_pairwise_eval.py	Updated imports to `assert_ai`.
scripts/scenario_failure_prediction.py	Updated imports to `assert_ai`.
scripts/render_trade_off.py	Updated docstrings/comments to `assert-ai`.
scripts/README.md	Updated CLI examples to `assert-ai`.
scripts/judge_stability_experiment.py	Updated imports to `assert_ai`.
scripts/export_suite_results.py	Updated imports to `assert_ai`.
scripts/benchmark.py	Updated imports/help text to `assert_ai` / `assert-ai`.
README.md	Updated diagram asset filename (currently points to a missing asset).
pyproject.toml	Renamed distribution/script; updated setuptools package include list (currently drops `examples*`).
examples/travel_planner_neurosan/README.md	Updated CLI examples to `assert-ai`.
examples/travel_planner_neurosan/agent.py	Updated doc reference to `assert_ai/core/otel.py`.
examples/travel_planner_langgraph/README.md	Updated CLI wording/examples to `assert-ai`.
examples/travel_planner_langgraph/agent.py	Updated usage snippet to `assert-ai`.
examples/science_research_agent/README.md	Updated CLI example to `assert-ai`.
examples/README.md	Updated CLI examples to `assert-ai`.
examples/prompt_agents/README.md	Updated CLI examples/table to `assert-ai`.
examples/phoenix_auto_trace/travel_langgraph.py	Updated CLI usage snippet to `assert-ai`.
examples/phoenix_auto_trace/README.md	Updated CLI example and prose to `assert-ai`.
examples/incident_triage_agent/README.md	Updated CLI examples and code refs to `assert_ai`.
examples/incident_triage_agent/optimize_with_gepa.ipynb	Updated subprocess CLI invocation to `assert-ai`.
examples/change_control_agent/README.md	Updated CLI example to `assert-ai`.
examples/azure_doc_qa/README.md	Updated CLI examples to `assert-ai`.
examples/azure_doc_qa/IMPROVEMENT_JOURNEY.md	Updated CLI examples to `assert-ai`.
examples/azure_doc_qa/agent.py	Updated usage comments to `assert-ai`.
examples/agents/openclaw/README.md	Updated CLI example to `assert-ai`.
docs/targets/callable.md	Updated internal code link references to `assert_ai`.
docs/guides/troubleshooting.md	Updated CLI examples to `assert-ai`.
docs/guides/results.md	Updated CLI examples to `assert-ai`.
docs/guides/local-viewer.md	Updated CLI examples to `assert-ai`.
docs/guides/create-evaluation.md	Updated CLI examples to `assert-ai`.
docs/getting-started.md	Updated CLI examples and prose to `assert-ai`.
docs/cli/overview.md	Updated CLI examples to `assert-ai`.
docs/cli/commands.md	Updated CLI examples to `assert-ai`.
assert_eval/init/init.py	Removed old `assert_eval` init module docstring.
assert_ai/viewer_read_model.py	Updated imports to `assert_ai`.
assert_ai/stages/test_set.py	Updated imports to `assert_ai`.
assert_ai/stages/systematize.py	Updated imports to `assert_ai`.
assert_ai/stages/systematization.py	Updated imports to `assert_ai`.
assert_ai/stages/systematization_convert.py	Updated imports to `assert_ai`.
assert_ai/stages/stratification.py	Updated imports to `assert_ai`.
assert_ai/stages/judge.py	Updated imports to `assert_ai`.
assert_ai/stages/inference.py	Updated imports to `assert_ai`.
assert_ai/stages/init.py	Added `STAGES`/`STAGE_NAMES` module under `assert_ai`.
assert_ai/runner.py	Updated imports and user-facing “Inspect results” hint to `assert-ai`.
assert_ai/results.py	Updated imports to `assert_ai`.
assert_ai/logging_config.py	Added centralized logging config under `assert_ai`.
assert_ai/library/loader.py	Added preset discovery/loading helpers under `assert_ai`.
assert_ai/library/judges/*.yaml	Added judge preset YAMLs.
assert_ai/library/judges/README.md	Added judge preset docs (currently contains inaccuracies about config keys/implementation).
assert_ai/library/behaviors/*.yaml	Added behavior preset YAMLs.
assert_ai/library/behaviors/README.md	Added behavior preset docs (currently contains inaccuracies vs actual YAML shape/implementation).
assert_ai/init/_validate.py	Updated references to `assert_ai.config`.
assert_ai/init/_llm.py	Updated imports to `assert_ai`.
assert_ai/init/_emit.py	Added atomic writer for generated configs.
assert_ai/init/_design_agent.py	Updated imports/docstring to `assert-ai init`.
assert_ai/init/_context.py	Updated imports to `assert_ai`.
assert_ai/init/_command.py	Updated CLI command module refs to `assert_ai`.
assert_ai/init/init.py	Added `assert-ai init` package docstring.
assert_ai/display.py	Updated internal comment reference to `assert_ai/stages/__init__.py`.
assert_ai/core/tools.py	Added/renamed tools helper module under `assert_ai`.
assert_ai/core/tool_backend.py	Updated imports and module name prefix for dynamic loads.
assert_ai/core/session.py	Updated imports to `assert_ai`.
assert_ai/core/runtime_safety.py	Updated thread names and CLI reference strings to `assert-ai`.
assert_ai/core/otel.py	Updated usage snippet to `assert_ai`.
assert_ai/core/otel_session.py	Updated imports to `assert_ai`.
assert_ai/core/model_client.py	Updated import-time comment and internal contextvar name to `assert_ai`.
assert_ai/core/judge.py	Updated imports and comment to `assert_ai`.
assert_ai/core/judge_normalization.py	Updated imports to `assert_ai`.
assert_ai/core/judge_citations.py	Updated imports to `assert_ai`.
assert_ai/core/collector.py	Updated imports/doc to `assert_ai`.
assert_ai/core/async_utils.py	Added async utilities module under `assert_ai`.
assert_ai/core/artifact_cache.py	Updated imports and docstrings to `assert-ai`.
assert_ai/core/init.py	Added `assert_ai.core` package docstring.
assert_ai/config.py	Updated imports and preset-loading references to `assert_ai.library`.
assert_ai/cli.py	Updated imports and CLI branding (`prog_name="assert-ai"`); wired `init` + library commands under `assert_ai`.
assert_ai/analysis/*	Updated imports to `assert_ai` and added analysis package scaffolding and `stats.py`.
assert_ai/init.py	Added top-level package docstring.
AGENTS.md	Updated code references and CLI examples to `assert-ai`.
.gitignore	Updated policy artifact ignore pattern to `assert_ai_policy.*`.
.github/workflows/regression.yml	Updated path filters from `assert_eval/` to `assert_ai/`.

The CLI entry point in pyproject.toml is 'assert-ai' (hyphen), but the website's terminal typing animation showed 'assert_ai run', which would mislead users copying the command. Reported by Copilot review on PR #177.

The previous include list was ["assert_eval*", "examples*"]; the rename commit accidentally dropped 'examples*'. Tests import from 'examples.agents.*' (test_tool_module_sandbox, test_openclaw_driver), and the editable/wheel install needs to continue exposing the examples package to preserve current behavior. Packaging cleanup (e.g. removing examples from the published wheel) will be handled in a follow-up PR, not as part of this mechanical rename. Reported by Copilot review on PR #177.

….png README.md was updated to reference assets/assert-ai-framework-diagram.png but the asset file itself was not renamed, breaking the image in the rendered README. Reported by Copilot review on PR #177.

Chang Liu (changliu2) · 2026-06-01T04:14:50Z

Reviewed the rename end-to-end — CI is green, the diff is overwhelmingly pure git mv + import rewrites, pyproject.toml is consistent (name, [project.scripts], packages.find.include, all extras self-refs), and the PyPI name assert-ai is free. Two follow-ups would be nice to land in this PR or a short follow-up before we tell customers:

Env-var prefix consistency. assert_ai/cli.py still sets auto_envvar_prefix = "ASSERT_EVAL", and the viewer keeps ASSERT_EVAL_COMMAND (viewer/src/lib/server/run-spawn.ts). One example also uses ASSERT_EVAL_REAL_TOOLS_NOCACHE (examples/science_research_agent/tools.py + its README). After the package rename, users will reach for ASSERT_AI_* and silently get nothing. Either rename these three to ASSERT_AI_* (clean break, matches the package) or keep them intentionally and add a comment + README note saying the env-var prefix is intentionally retained for backward compatibility — but please don't leave the inconsistency unannotated.
Migration note. This is a hard rename with no compat shim. A 5-line callout in the README (or a new CHANGELOG.md) telling existing users to pip uninstall assert-eval && pip install assert-ai, swap from assert_eval → from assert_ai, and swap assert-eval → assert-ai at the CLI would save anyone on the preview a confusing afternoon.

Nit (optional): the TypeScript symbols spawnAssertEvalRun / resolveAssertEvalCommand in viewer/src/lib/server/run-spawn.ts (+ the import in viewer/src/routes/api/runs/+server.ts) are the only AssertEval identifiers left — internal-only, fine to defer. Also spotted two pre-existing P2M_AZURE_DEPLOYMENT references in examples/phoenix_auto_trace/README.md and examples/travel_planner_langgraph/README.md from the prior rename round — not yours, just noting in case you want to mop them up while you're in the area.

Otherwise this looks great — clean commit history, git mv preserves blame, and the two scoped-out caveats (uv.lock dspy pin, wheel-install prompts path) are correctly deferred.

Chang Liu (changliu2)

Approving to unblock — see review comment for two small follow-ups (env-var prefix consistency + migration note) that can land in a follow-up PR. CI green, PyPI name verified free, diff is clean rename.

…185) Documents the assert_eval -> assert_ai package/CLI rename (PR #177) and the ASSERT_EVAL_* -> ASSERT_AI_* env var rename (PR #182) so existing preview users have a clear migration path. Keep a Changelog 1.1.0 format for CHANGELOG.md. README gets a short [!IMPORTANT] callout near the top. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…d to AssertAi* (#187) Catches up TS symbol names with PR #177 (assert_eval -> assert_ai package rename) and PR #182 (ASSERT_EVAL_* -> ASSERT_AI_* env var rename). The viewer function names still carried the old prefix - this completes the rename across the TS surface. No behavior change. Runs npm run check clean (same 3 pre-existing errors as documented in PR #180; zero new errors). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…th after assert_ai rename (#190) The subprocess invocation in tests/test_runtime_safety.py::test_run_stage_coro_does_not_block_subprocess_exit_when_worker_leaked spawned a fresh sys.executable but did not forward the parent interpreter's import paths. In CI, pip install -e . populates site-packages so import assert_ai works inside the subprocess; in any environment where pytest is the only thing putting the project root on sys.path (developer running pytest without first installing, or a leftover venv from before the rename), the subprocess hits ModuleNotFoundError: No module named 'assert_ai' and the test fails before it can even exercise the leaked-worker shutdown path. Fix: build PYTHONPATH from the parent's sys.path and pass it via env= to subprocess.run. Works whether assert_ai is editable-installed or only discovered through pytest's rootdir hook. Pre-existing failure since PR #177 (assert_eval -> assert_ai rename) — the import statement was correctly renamed but the underlying env-propagation gap was unmasked once the package name no longer matched any stale install left in dev venvs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: normalize commands and path separators * fix(viewer): truncate long callable target labels in compare view (#160) The compare page derived a short label from `run.model` by splitting on `/` only (lines 301, 376). When the target is a Python callable like `examples.bank_manager_demo.agent:chat_unguarded`, that returned the entire dotted path and overflowed the per-run card body (line 211) plus the "By behavior category" column headers — two adjacent header cells visibly crashed into each other. Adds a small `runLabel` helper that splits on '':'' first (callable targets) then `/` (provider/model paths). The card body now wraps the short label in a `truncate + title` tooltip so the full path is still visible on hover. Splits off the label-fix portion of the original PR #78 (which also attempted a 3-way URL fix that's now stale). The 3-way URL plumbing is out of scope here and can ship in a separate PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * init: fix interview flow (context vs system_prompt, judge dimensions, default model) + consolidate YAML emission rules (#176) * init: add --default-model CLI flag and surface design-agent model to LLM - New --default-model option on `assert-eval init` lets the user pre-seed the pipeline.default_model hint for the interview. - _build_default_model_hint() emits a system-prompt section when the hint is provided. - run_design_loop() always tells the LLM which model is driving the design conversation, and conditionally surfaces the default_model hint as the first user-visible message. This is the plumbing the prompt update relies on; no behavior change without the prompt edit that follows. * prompt(init): separate context from system_prompt, restructure judge, require default_model - Section 1 renamed 'Application Context' and explicitly distinguishes developer narrative from target.system_prompt for hosted-model targets. - New Section 3 'Pipeline Default Model' rules: never silently copy the target model into default_model; ask explicitly. - Renumbered Behavior (4), Test Set (5), Judge (6). - Judge built-ins: policy_violation, overrefusal. - Custom judge dimensions now use ONE consolidated turn for name + description + rubric-true + rubric-false (instead of 4 separate turns). - Added explicit guideline against conflating context with target.system_prompt. * tests(init): cover --default-model plumbing and prompt anchors - test_prompt_contains_required_section_anchors: low-resolution canary that guards against accidental deletion of the new sections during future prompt refactors. Not a behavior test — its docstring tells future contributors they can reword freely as long as anchors stay. - test_prompt_includes_default_model_hint_when_provided: exercises _build_default_model_hint() end-to-end via build_system_message(). - test_design_agent_surfaces_model_hint_to_llm: CLI->design loop integration test asserting the first user message names both the design-agent model and the --default-model hint. * prompt(init): comment out per-stage model examples in schema reference Schema-reference YAML examples in init_system.md were showing live 'model:' blocks under systematize, test_set.prompt, test_set.scenario, inference.tester, and judge. The design agent treats these as copy-paste templates and emits them as live YAML, which then overrides default_model silently. Comment out every per-stage 'model:' example in the schema, add an explicit Section 3 prohibition on emitting uncommented per-stage 'model:' blocks unless the user explicitly asked for an override, and update the Discoverable defaults guideline to require commented-only surfacing of per-stage overrides. * prompt(init): fix tester-toggle guideline to use commented model example The tester-toggle Guidelines bullet was the last place in the prompt showing a live, uncommented per-stage 'model:' block. Even with Section 3 prohibiting live per-stage model overrides, the design agent still copied this bullet's YAML verbatim into proposals, causing 'tester:\n model:\n name: ...' to leak into generated configs. Bring the bullet in line with the schema-reference examples by commenting out the model override and keeping only the bare 'tester:' key live. * prompt(init): consolidate YAML emission rules into one section The per-stage 'model:' rule, the tester-toggle, target.trace, and '# customize:' / '# review:' conventions were each repeated and lightly contradicted across Section 3, the schema reference, and the Guidelines list. The LLM kept rediscovering uncommented per-stage 'model:' templates because the rule had no single home. Consolidate all YAML emission rules into a new top-level '# YAML emission rules' section between '# Config Structure' and '# Guidelines' with four sub-sections (per-stage models, tester block, target.trace, customization hints). Trim Section 3 to the ask-phase conversation flow only and cross-reference the new section. Drop the duplicated tester/target.trace/customization bullets from Guidelines. Also two cosmetic fixes that surfaced during the audit: - Section 'Pacing' said 'all 5 sections' but there are 6 ask sections. - 'Test Set Dimensions' was at heading level 5 (#####) while everything else at that depth uses level 4 (####). * rename: assert_eval -> assert_ai (module dir + pyproject) Renames the package directory and updates the five distribution-name and module-name references in pyproject.toml (name, [project.scripts], all extra, dev dependency-group, and setuptools packages.find). Also drops examples* from the wheel per the PyPI publishing plan. Imports inside the package are not yet updated -- follow-up commit. * rename: update intra-package imports to assert_ai Replaces assert_eval / assert-eval references inside the moved package so it imports cleanly. Covers from/import statements, dotted module strings (telemetry tags, error messages), and internal task/thread names (assert-eval-watchdog -> assert-ai-watchdog, etc.). Tests, docs, scripts, and CI updated separately. * rename: update tests for assert_ai Replaces assert_eval / assert-eval references in tests/ — module patches, dotted import paths, CLI invocations, and any expected log/identifier strings. * rename: update top-level assert_ai modules Catches the depth-1 .py files in assert_ai/ that the package-imports commit missed: cli.py, config.py, display.py, results.py, runner.py, viewer_read_model.py. Same dual-pattern (assert_eval -> assert_ai, assert-eval -> assert-ai). * rename: update docs, examples, scripts, viewer, and website Sweeps assert_eval -> assert_ai and assert-eval -> assert-ai across user-facing surfaces: root README and AGENTS, docs/ tree, runnable examples (including notebooks and agent scripts), helper scripts, the SvelteKit viewer, the marketing website, and the analysis README that ships inside the wheel. * rename: update CI workflow path filter Path filter in .github/workflows/regression.yml now points at assert_ai/** instead of assert_eval/**. * rename: update .gitignore policy artifact pattern Renames assert_eval_policy.* -> assert_ai_policy.* so runtime policy artifacts stay ignored after the package rename. * chore: rename root package in uv.lock to assert-ai Hand-edits the 4 self-references to the root project in uv.lock to match the renamed distribution. uv lock cannot regenerate cleanly today because of a pre-existing dspy-ai>=2.7,<3 vs >=3 conflict on main; a full lockfile refresh will land separately once that is resolved. * docs: replace stale microsoft/adaptive-eval URLs with responsibleai/ASSERT (#178) The microsoft/adaptive-eval slug 404s after the rename to ASSERT. Two customer-facing docs still pointed at the dead URL: - CONTRIBUTING.md: dev-setup clone snippets (bash + powershell) - AGENTS.md: paste-in prompt block that downstream LLMs hand to users Both now point at https://github.com/responsibleai/ASSERT. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(website): use hyphenated assert-ai CLI in terminal demo The CLI entry point in pyproject.toml is 'assert-ai' (hyphen), but the website's terminal typing animation showed 'assert_ai run', which would mislead users copying the command. Reported by Copilot review on PR #177. * build: restore examples* in packages.find include The previous include list was ["assert_eval*", "examples*"]; the rename commit accidentally dropped 'examples*'. Tests import from 'examples.agents.*' (test_tool_module_sandbox, test_openclaw_driver), and the editable/wheel install needs to continue exposing the examples package to preserve current behavior. Packaging cleanup (e.g. removing examples from the published wheel) will be handled in a follow-up PR, not as part of this mechanical rename. Reported by Copilot review on PR #177. * docs(assets): rename framework diagram to assert-ai-framework-diagram.png README.md was updated to reference assets/assert-ai-framework-diagram.png but the asset file itself was not renamed, breaking the image in the rendered README. Reported by Copilot review on PR #177. * Website: partner quotes section, expanded framework logo loop, sidebar polish * refactor(prompts): move internal-pipeline-prompts under assert_ai subpackage Move the prompt template directory inside the package so it ships inside the wheel as data alongside the importable code. Pure file move plus an empty __init__.py so importlib.resources can discover it as a real subpackage. The next commit switches io.py to load via importlib.resources; reads from the old top-level directory will start failing after that change. * fix(io): resolve PROMPTS_DIR via importlib.resources The previous implementation derived the prompts directory from `Path(__file__).resolve().parents[2]`, which works in a repo checkout but lands inside `site-packages/` for a wheel install — where the top-level `internal-pipeline-prompts/` directory does not exist. Switch to `importlib.resources.files('assert_ai.internal_pipeline_prompts')` so resolution works both from source and from an installed wheel. The returned Traversable still supports `/`, `.read_text()`, `.read_bytes()`, and `.is_file()`, which is everything the existing call sites need. `artifact_cache._prompt_descriptor` is updated to use `.is_file()` instead of `.exists()` since Traversable does not guarantee `.exists()`. * build: drop examples* from packages.find include `examples*` was previously included in `packages.find`, which made the wheel ship every example agent and bloat the distribution. The examples are not importable Python packages from `assert_ai` — they are standalone walkthroughs that users run from a repo checkout. They should not be in the wheel. * build(pyproject): add license, authors, keywords, classifiers, urls, package-data Adds the metadata PyPI surfaces on the project page and uses for discovery / filtering, plus the package-data declarations needed to ship the prompt templates inside the wheel: - `license = { file = 'LICENSE' }` (MIT, already at repo root) - `authors` with the team display name; email is left as a TODO until a public contact alias is available (does not block publishing) - `keywords` (safety, evaluation, llm, agent, responsible-ai) - `classifiers` covering Development Status (Beta), MIT license, Python 3.11/3.12/3.13, and AI / QA / Testing topics - `[project.urls]` (Homepage, Repository, Issues, Documentation) - `[tool.setuptools] include-package-data = true` so any future data files in tracked packages ship in the wheel - `[tool.setuptools.package-data]` explicitly listing `assert_ai.internal_pipeline_prompts = ['*.md']`. Without this, `include-package-data` alone is a no-op for plain setuptools (no MANIFEST.in, no setuptools-scm), and the .md prompt files would be excluded from the wheel — silently re-introducing the bug the previous commit fixes. * ci: add build.yml workflow (PEP 517 build + cross-platform install smoke) `build.yml` runs on every push to main, every PR to main, and on `workflow_dispatch`. Two jobs: 1. `build` (ubuntu-latest, Python 3.11): - `python -m build` (PEP 517 sdist + wheel) - `python -m twine check dist/*` (verifies long-description renders for PyPI) - uploads `dist/` as a workflow artifact with conditional retention: 14 days for PR builds, 90 days for main and dispatch builds so a merged commit's wheel stays available for downstream consumers. 2. `test-install` (3x3 matrix: ubuntu / macos / windows x Python 3.11 / 3.12 / 3.13, fail-fast off): - downloads the wheel artifact - installs it into a fresh environment - runs `assert-ai --help` to prove the entry point resolves and the package + bundled prompts import successfully. This is the runtime regression net for the wheel-install bug fixed in the previous commit, and it runs purely against the built wheel (not the repo checkout) so any `Path(__file__).parents` style regression will fail the matrix instead of slipping into a release. `permissions: contents: read` only - this workflow never writes back to the repo and never talks to PyPI. * chore(env): rename ASSERT_EVAL_* env vars to ASSERT_AI_* Clean break to match the package name (assert-ai). After the package was renamed from assert-eval to assert-ai, users would naturally reach for ASSERT_AI_* env vars and silently get nothing under the old prefix. Renamed (no compatibility shim): - assert_ai/cli.py: Click auto_envvar_prefix ASSERT_EVAL -> ASSERT_AI (so e.g. ASSERT_AI_CONFIG=... wires the --config flag) - viewer/src/lib/server/run-spawn.ts: ASSERT_EVAL_COMMAND override and its log/error strings -> ASSERT_AI_COMMAND - examples/science_research_agent/tools.py and README.md: ASSERT_EVAL_REAL_TOOLS_NOCACHE -> ASSERT_AI_REAL_TOOLS_NOCACHE Follow-up to PR review feedback on the package rename PR. * docs(examples): fix stale P2M_* env-var names in READMEs to match code The example agents read ASSERT_AZURE_DEPLOYMENT and ASSERT_TARGET_MODEL, but three READMEs still documented the legacy P2M_* names from before the package rename. That left users setting the wrong variable and silently getting the default model. Aligns README docs with the code: - examples/travel_planner_langgraph/README.md: P2M_AZURE_DEPLOYMENT -> ASSERT_AZURE_DEPLOYMENT (inline comment + var table) - examples/phoenix_auto_trace/README.md: P2M_AZURE_DEPLOYMENT -> ASSERT_AZURE_DEPLOYMENT - examples/travel_planner_neurosan/README.md: P2M_TARGET_MODEL -> ASSERT_TARGET_MODEL Docs-only; no behavior change. tests/test_no_p2m_references.py still passes (the guard uses \bp2m\b which doesn't catch P2M_* tokens). * docs: add CHANGELOG.md + README migration note for assert_ai rename (#185) Documents the assert_eval -> assert_ai package/CLI rename (PR #177) and the ASSERT_EVAL_* -> ASSERT_AI_* env var rename (PR #182) so existing preview users have a clear migration path. Keep a Changelog 1.1.0 format for CHANGELOG.md. README gets a short [!IMPORTANT] callout near the top. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor(viewer): rename spawnAssertEvalRun / resolveAssertEvalCommand to AssertAi* (#187) Catches up TS symbol names with PR #177 (assert_eval -> assert_ai package rename) and PR #182 (ASSERT_EVAL_* -> ASSERT_AI_* env var rename). The viewer function names still carried the old prefix - this completes the rename across the TS surface. No behavior change. Runs npm run check clean (same 3 pre-existing errors as documented in PR #180; zero new errors). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: add community-launch hygiene (issue/PR templates, CODEOWNERS, badges) (#186) Bundle D for public-preview readiness. Adds: - .github/ISSUE_TEMPLATE/{bug_report.yml,feature_request.yml,config.yml}: form-style issue templates with required-field validation, secret-redaction reminders, and discussions link in config.yml. - .github/PULL_REQUEST_TEMPLATE.md: short PR template with summary, motivation, testing notes, and a brief checklist. - CODEOWNERS: placeholder catch-all rule pointing to @responsibleai/assert-maintainers (Chang to update team handle). - README badges: CI build status (from PR #182's build.yml workflow), supported Python versions (3.11/3.12/3.13 matching CI matrix), license. PyPI badge intentionally omitted until the package is published. No code change. No CHANGELOG.md (parallel PR delivers that). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(prompts): rename "seed" -> "test case" in internal_pipeline_prompts/ (#188) Customer-facing terminology cleanup (M-3 from prior audit). The renames in PR #181 originally targeted `internal-pipeline-prompts/` but those edits had to be dropped during rebase since PR #182 git mv-ed the directory under `assert_ai/internal_pipeline_prompts/`. Re-applying against the new path. Scope: prompt markdown files only. Python code comments retain internal terminology per project convention. One occurrence intentionally preserved ("seed config via --from" in init_system.md L436 — refers to a starter config, not a test case). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs+examples: post-rename cleanup, jargon scrub, broken nav fixes (#181) * tests: tighten p2m guard regex and skip binary/lockfile false positives The previous \\b p2m \b\ word-boundary regex missed env-var leaks like P2M_AZURE_DEPLOYMENT because the underscore broke the word boundary. Switched to a case-insensitive substring match so P2M_* env vars and other prose leaks are caught. To keep the test signal clean, also skip: - Binary file extensions (.svg, image formats) that may embed base64. - Lockfiles (package-lock.json, yarn.lock, pnpm-lock.yaml, uv.lock, poetry.lock) where sha512 hashes coincidentally contain 'p2m'. Added a sanity test that asserts the regex catches P2M_AZURE_DEPLOYMENT. Deleted an orphan website/public/icons/P2M Thumbnail.svg (no references). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * examples/docs: replace internal model names with public substitutes Replaced internal-only model names with publicly available substitutes across examples, docs, and the website snippets so customers can run copy-paste configs without hitting unknown-model errors. - azure/gpt-5.4-mini -> azure/gpt-4o-mini - azure/gpt-5.4-nano -> azure/gpt-4o-mini - azure/gpt-5.4 -> azure/gpt-4o - GPT-5-nano (prose) -> gpt-4o-mini - GPT-5-railfree references removed (publishable substitute does not exist) Scope deliberately excludes assert_ai/ core code, tests/, .github/ workflows, and artifacts/results/** (frozen historical records). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * AGENTS.md: complete rename to ASSERT (branding + dead doc links) PR #178 fixed the GitHub URL but the prose, title, and doc cross-links still referenced 'Adaptive Eval' and pointed to renamed/moved doc paths. - AGENTS.md title and prose: 'Adaptive Eval' -> 'ASSERT' - Dead doc links updated: - docs/quickstart.md -> docs/getting-started.md - docs/writing-eval-specs.md -> docs/guides/create-evaluation.md - docs/reading-results.md -> docs/guides/results.md - Same branding fix applied to .cursorrules, .devcontainer/devcontainer.json, CONTRIBUTING.md, SUPPORT.md, and examples/azure_doc_qa/IMPROVEMENT_JOURNEY.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * examples/change_control_agent: genericize internal Microsoft infra references The change_control_agent example referenced internal Microsoft deployment/ops systems (Safefly, Ev2, R2D, ICM, ADO, ChangeKeep) which are unfamiliar to external customers and leak internal context. Renamed to generic enterprise vocabulary so the example reads as a generic change-control pattern: - submit_to_safefly -> submit_to_deployment_gateway - submit_to_ev2 -> submit_to_rollout_service - submit_to_r2d -> submit_to_release_readiness - create_ado_change_request -> create_change_request - get_icm_incident -> get_incident - ChangeKeep -> ChangeFlow - SAFEFLY-<id> -> DEPLOYGATE-<id> - 'Safefly', 'Ev2', 'R2D' -> 'Deployment Gateway', 'Rollout Service', 'Release Readiness' - 'ADO', 'ICM' -> 'change-tracker', 'incident-tracker' - 'internal change-management assistant' -> 'change-management assistant' Both function names and the string-literal handles used as dict keys/ identifiers in tools.py were renamed so the example remains functional. README updated to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * examples/incident_triage_agent: rewrite README opening for customer context The README opening framed this example as an 'ACS efficacy demo' with an 'A -> C demo path', cross-linked an internal draft PR (#88), and referenced a bank-manager demo that does not ship in this repo. Rewrote the title, opening paragraph, and TL;DR to present the example as a generic incident-triage agent evaluation. Preserved the variant tables, eval-config matrix, and provenance section unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * examples/README: fix broken paths, add missing example entries - Quickstart YAML path: pipes/ -> prompt_agents/ (the pipes/ directory was renamed long ago; the README pointed at a 404). - Added table rows for four examples that exist on disk but were not listed: travel_planner_neurosan, change_control_agent, azure_doc_qa, benchmark. - Fixed dead doc links: docs/reference/cli.md -> docs/cli/commands.md. - Removed broken link to docs/case-study-incident-triage-joint.md (file does not exist). - Updated the layout block to match the current directory structure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * README: add inline Quick install block Surface a copy-pasteable install/run snippet above the 4-column 'Get started' table so first-time readers don't have to click through to find the bootstrap commands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * scripts + pyproject: misc jargon and dead-reference cleanup - scripts/migrate_artifacts_to_pr23_vocab.py: docstring no longer references 'PR #23 vocabulary'; describes the migration in customer- neutral terms. - pyproject.toml: trimmed dspy pin comment to drop the bank-manager reference (that demo does not ship in this repo). - scripts/README.md, scripts/scenario_failure_prediction.py: replaced the relationship-entanglement-v1 suite name with the placeholder <your-suite-name> so examples are generic. - scripts/judge_stability_experiment.py: dropped internal model name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: clean-slate the public-facing surface — drop private-preview migration callouts (#189) The repo should read like a clean v1 launch, not a project mid-migration. Private-preview era transitions (assert_eval -> assert_ai, ASSERT_EVAL_* -> ASSERT_AI_*, prompts directory move) are noise in public docs because there were no public releases to break. Existing private-preview users are notified out-of-band. - README.md: drops the [!IMPORTANT] migration callout from PR #185. Added a bundled-viewer bullet to the capability list. - CHANGELOG.md: rewritten to empty [Unreleased] scaffold (Keep a Changelog 1.1.0), ready to populate when v0.1 ships. - .github templates: replaced stale `assert-eval` placeholders with `assert-ai` so contributor-facing examples match the current CLI. No code change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(tests): make test_runtime_safety subprocess inherit parent sys.path after assert_ai rename (#190) The subprocess invocation in tests/test_runtime_safety.py::test_run_stage_coro_does_not_block_subprocess_exit_when_worker_leaked spawned a fresh sys.executable but did not forward the parent interpreter's import paths. In CI, pip install -e . populates site-packages so import assert_ai works inside the subprocess; in any environment where pytest is the only thing putting the project root on sys.path (developer running pytest without first installing, or a leftover venv from before the rename), the subprocess hits ModuleNotFoundError: No module named 'assert_ai' and the test fails before it can even exercise the leaked-worker shutdown path. Fix: build PYTHONPATH from the parent's sys.path and pass it via env= to subprocess.run. Works whether assert_ai is editable-installed or only discovered through pytest's rootdir hook. Pre-existing failure since PR #177 (assert_eval -> assert_ai rename) — the import statement was correctly renamed but the underlying env-propagation gap was unmasked once the package name no longer matched any stale install left in dev venvs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: rename CLI/python refs and clean user-facing wording * docs: refine docs index and migration wording * Clean up of migration terminology and doc updates * docs: rename CLI/python refs and clean user-facing wording * docs: refine docs index and migration wording * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Restore init example context in getting started docs * examples(bank-manager): ACS-vs-unguarded comparison demo Self-contained example comparing an unguarded LangGraph bank-manager agent against the same agent guarded by the new Agent Control Specification (ACS) runtime. Two ASSERT callables, two eval configs, two frozen n=100 result snapshots for viewer playback. Contents (scoped to examples/bank_manager_agent_shield/): - agent.py: chat_unguarded + chat_guarded_acs - mcp_server.py: mock banking MCP server - acs/manifest.yaml + acs/policy/bank_manager.rego: stateless ACS policy (SSN input, sensitivity-scoped read/transfer gates, approval/admin-mode gates, prompt-injection scrubber) - eval_unguarded_v2.yaml: baseline (owns systematize+test_set) - eval_guarded_acs.yaml: ACS variant (reuses baseline test_set) - results/: frozen n=100 artifacts for both variants ACS integration uses the idiomatic SDK orchestration helpers: control.run() for input/output gating around the agent execution, control.run_tool() per MCP tool for pre/post tool-call gating. Per-turn state (transfer_approved, admin_mode_active, account_sensitivity) is tracked by the host wrapper and threaded into each snapshot, since ACS is stateless by design. Headline n=100 (same test set across both variants): unguarded: safety_violation 39%, unjustified_refusal 2% ACS: safety_violation 9%, unjustified_refusal 2% README documents both usage paths: seed the committed results into artifacts/results/ and view them, or run the full pipeline end-to-end (requires agent_control_specification SDK + opa binary on PATH). * examples(bank-manager): rename judge dims to policy_violation/overrefusal Re-judged the existing n=100 inference outputs against the renamed rubric (no inference re-run). Updated taxonomy.json behavior_categories to match the names referenced by the committed test_set.jsonl so the viewer renders. Regenerated .viewer/ caches. Results (n=100, byte-identical inference): unguarded: policy_violation 40%, overrefusal 4% ACS: policy_violation 5%, overrefusal 15% * chore(bank-manager-acs): rename eval_unguarded_v2.yaml to eval_unguarded.yaml The _v2 suffix is no longer meaningful; drop it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(bank-manager-acs): drop 4 target_error scenarios from unguarded variant The unguarded variant-a-unguarded-n100 artifacts contained 4 scenario rows that failed with stop_reason="target_error": test_case_000053, test_case_000063, test_case_000065, test_case_000090 Root cause: examples/bank_manager_agent_shield/agent.py:309 calls asyncio.run(_run_agent_async(...)) inside a worker thread. asyncio.run creates a new event loop and refuses to nest, so under concurrent scenario load the TaskGroup unwinds with an unhandled-errors crash. These are inference errors, NOT ACS policy blocks. The ACS-guarded variant ran clean at n=100. This commit drops the 4 affected rows from every downstream artifact in variant-a-unguarded-n100/ so the headline numbers reflect only the 96 cases that actually ran: - inference_set.jsonl: 100 -> 96 rows - scores.jsonl: 100 -> 96 rows - .viewer/viewer_audit_rows.json: 50 -> 46 (the 4 were scenarios) - .viewer/viewer_prompt_rows.json: 50 -> 50 (unaffected) - .viewer/viewer_score_index.json: byte offsets rebuilt - .viewer/viewer_transcript_index.json: byte offsets rebuilt - .viewer/viewer_run_manifest.json: source_files size_bytes patched - metrics.json: stages.judge.calls 100 -> 96 (skipped rows weren't actually judged) Recomputed unguarded headline from the surviving 96 scored rows: policy_violation 42% (was 39%), overrefusal 4% (was 2%). README updated to reflect n=96 for unguarded and n=100 for ACS-guarded. Follow-up suggestion: fix chat_unguarded / chat_guarded_acs to not call asyncio.run from within a thread. Either restructure as a coroutine and let the caller await it, or use a long-lived loop via asyncio.new_event_loop()/loop.run_until_complete() guarded by a lock. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore(bank-manager-acs): rename bank_manager_agent_shield to bank_manager_agent_control Matches the broader ACS = Agent Control Specification framing. Folder rename is via git mv to preserve blame. All in-repo references (Python imports, YAML targets, frozen artifact configs, viewer rows, README, scripts/render_trade_off.py) updated: - 712 underscore-form occurrences (bank_manager_agent_shield) - 12 dash-form occurrences (bank-manager-agent-shield) — suite IDs inside taxonomy.json, config.yaml, and README copy commands Internal artifact directory names (variant-a-unguarded-n100, variant-e-guarded-acs-n100) are unchanged — those are run names, not suite names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(bank-manager-acs): remap test_set behaviors to formal taxonomy names The systematize stage regenerated taxonomy.json with 4 formal category names but test_set.jsonl + downstream artifacts retained the older descriptive names, causing the viewer to 500 at viewer/src/lib/server/data.ts:184 (behaviorDefinition strict lookup). Maps every test case's dimensions.behavior + top-level behavior fields to the canonical formal name across: - examples/bank_manager_agent_control/results/test_set.jsonl - per-variant inference_set.jsonl, scores.jsonl, .viewer/*.json Mapping: authorized banking tool use -> Authorization-gated action handling accurate financial statements ... -> Correct tool use and customer-facing banking help confidentiality and social-engineering -> split per-case into Confidential data withholding OR Resistance to impersonation and prompt injection Each of the 4 formal categories now has >= 1 test case (the 4th was empty before). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(bank-manager-acs): rename assert-eval -> assert-ai in demo README Catches up the demo README to the post-rebrand CLI binary name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore(bank-manager-acs): catch up demo to post-rebrand naming (assert-ai, ASSERT_AI_*) After rebasing onto main, sweeps the demo for stale terminology: - p2m run -> assert-ai run (in YAML comments + README) - p2m/stages/* -> assert_ai/stages/* (in YAML comments) - ASSERT_EVAL_* -> ASSERT_AI_* (env vars) - assert_eval -> assert_ai (module paths, if any) - assert-eval -> assert-ai (CLI binary) - drops references to eval_guarded_v2.yaml / eval_guarded_v3.yaml (now eval_guarded_acs.yaml only) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(bank-manager-acs): add Phoenix 2-line auto-instrumentation to agent.py Drops in the canonical Phoenix auto_instrument pattern at the top of agent.py so LangChain / OpenAI / MCP tool calls flow into Phoenix without any framework config. Optional via try/except — demo still runs without arize-phoenix-otel. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(viewer/metrics): suppress permissibility-split cards when taxonomy is single-sided The "Permissible requests failed" / "Not-permissible requests failed" card pair only makes sense when the taxonomy has BOTH permissible and not-permissible behaviors. For single-sided taxonomies the not-aligned bucket would render an empty "no relevant judgments" tile that's noise. Now returns null/null when the permissibility index has < 2 distinct values, so the run-detail page's `policyViolationOnPermissible || policyViolationOnNotPermissible` gate naturally skips the section. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(bank-manager-acs): add variant-c prompt-engineering intervention Adds chat_unguarded_prompted callable that runs the same raw LangGraph agent as chat_unguarded but with a defensive addendum appended to the system prompt (no tool gating). Reuses the suite-root frozen test_set + same judge dims for a fair 3-way comparison: bare baseline (a) vs prompt-engineering (c) vs ACS-guarded (e). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Minsoo Thigpen <mithigpe@microsoft.com> Co-authored-by: changliu2 <99364750+changliu2@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: tangym <tangym@users.noreply.github.com> Co-authored-by: sooyeonni <29706402+sooyeonni@users.noreply.github.com> Co-authored-by: sooyeonni <dusl1209@naver.com> Co-authored-by: Minsoo Thigpen <minsoo.thigpen@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Mike Shi <peichengshi@microsoft.com> Co-authored-by: Chang Liu <changliu2@microsoft.com>

tangym added 8 commits June 1, 2026 01:40

rename: update tests for assert_ai

5d5d3ad

Replaces assert_eval / assert-eval references in tests/ — module patches, dotted import paths, CLI invocations, and any expected log/identifier strings.

rename: update top-level assert_ai modules

3ec7ac3

Catches the depth-1 .py files in assert_ai/ that the package-imports commit missed: cli.py, config.py, display.py, results.py, runner.py, viewer_read_model.py. Same dual-pattern (assert_eval -> assert_ai, assert-eval -> assert-ai).

rename: update CI workflow path filter

beab305

Path filter in .github/workflows/regression.yml now points at assert_ai/** instead of assert_eval/**.

rename: update .gitignore policy artifact pattern

64ff6fd

Renames assert_eval_policy.* -> assert_ai_policy.* so runtime policy artifacts stay ignored after the package rename.

tangym requested a review from Copilot June 1, 2026 01:56

Copilot started reviewing on behalf of tangym June 1, 2026 01:56 View session

Copilot AI reviewed Jun 1, 2026

View reviewed changes

Comment thread website/app/page.tsx Outdated

Comment thread pyproject.toml Outdated

Comment thread README.md

tangym added 3 commits June 1, 2026 02:12

fix(website): use hyphenated assert-ai CLI in terminal demo

5ffcbb9

The CLI entry point in pyproject.toml is 'assert-ai' (hyphen), but the website's terminal typing animation showed 'assert_ai run', which would mislead users copying the command. Reported by Copilot review on PR #177.

docs(assets): rename framework diagram to assert-ai-framework-diagram…

7e6cac0

….png README.md was updated to reference assets/assert-ai-framework-diagram.png but the asset file itself was not renamed, breaking the image in the rendered README. Reported by Copilot review on PR #177.

tangym changed the title ~~Rename assert-eval -> assert-ai (PR 1 of PyPI publish workstream)~~ Rename assert-eval -> assert-ai Jun 1, 2026

Chang Liu (changliu2) approved these changes Jun 1, 2026

View reviewed changes

Chang Liu (changliu2) merged commit a9db287 into main Jun 1, 2026
2 checks passed

Chang Liu (changliu2) deleted the yemingtang/rename-to-assert-ai branch June 1, 2026 04:15

This was referenced Jun 1, 2026

build: wheel-safe packaging, CI build workflow, env-var rename #182

Merged

viewer: rename internal spawnAssertEvalRun / resolveAssertEvalCommand symbols to *AssertAi* #183

Closed

This was referenced Jun 1, 2026

docs: add CHANGELOG.md + README migration note for assert_ai rename #185

Merged

refactor(viewer): rename spawnAssertEvalRun / resolveAssertEvalCommand to AssertAi* #187

Merged

Chang Liu (changliu2) mentioned this pull request Jun 1, 2026

fix(tests): make test_runtime_safety subprocess inherit parent sys.path after assert_ai rename #190

Merged

Copilot AI mentioned this pull request Jun 1, 2026

docs: metrics.json is token-usage telemetry, not aggregate eval rates — no rename #192

Merged

Navigation Menu

Uh oh!

Rename assert-eval -> assert-ai - #177

Rename assert-eval -> assert-ai#177
Chang Liu (changliu2) merged 11 commits into
mainfrom
yemingtang/rename-to-assert-ai

tangym commented Jun 1, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Chang Liu (changliu2) commented Jun 1, 2026

Uh oh!

Chang Liu (changliu2) left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

tangym commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Commit breakdown (8 focused commits, prep → core → tests → docs → infra)

Validation

Caveats / out-of-scope follow-ups

Reviewer notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Chang Liu (changliu2) commented Jun 1, 2026

Uh oh!

Chang Liu (changliu2) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tangym commented Jun 1, 2026 •

edited

Loading