examples(bank-support): replace AIEWF demo with two realistic-baseline behaviors

[Chang Liu (changliu2)]

Why

The original AIEWF bank demo began from an agent with no authorization logic. That made it easy to dismiss the result as a missing-check bug rather than evidence for a reusable evaluation/control loop.

This PR replaces it with two independently actionable bank-support behaviors whose baselines are reasonable first versions:

1. a deterministic sensitivity-tier authorization policy that is correct where deployed but has shallow cross-domain coverage;
2. a semantic coercion failure where a claimed authority signal can be confused with legitimate work.

ASSERT discovers and measures the runtime failure. ACS applies the matching control: property-based Rego for the deterministic case and a classifier-backed policy for the semantic case.

Both behaviors use the same workflow: one behavior -> one YAML -> three target arms.

Current viewer headline

The public README, CI guide, talk index, Pareto image, and AIEWF PDF use the current viewer headline values. The committed scored artifacts are prompt rows; this PR does not claim a committed scenario dataset.

Behavior 1 — sensitivity-tier authorization

Total 72 per arm:

Arm	Impermissible behavior violated	Permissible behavior violated
Baseline gate	8%	0%
Defensive prompt	6%	0%
ACS Rego	0%	0%

The defensive prompt improves the displayed aggregate by two percentage points. ACS Rego eliminates every observed impermissible authorization violation without adding permissible violations.

The deterministic generalization proof remains separate and reproducible:

• deposit-only gate: 2/13 protected records blocked;
• property-based Rego: 13/13;
• false positives: 0/11;
• protected records allowed after authorization: 13/13;
• additional policy lines for two later domains: 0.

Behavior 2 — resist coercion without blocking legitimate work

Total 120 per arm:

Arm	Impermissible behavior violated	Permissible behavior violated
Baseline prompt + keyword tripwire	8%	27%
Hardened prompt	0%	47%
ACS classifier	0%	27%

Both controls eliminate observed impermissible violations. The ACS classifier preserves 20 percentage points more legitimate work than the hardened prompt and matches the baseline permissible-violation rate.

The branch retains the reviewed 120-prompt fixture, labels, per-case arm outcomes, and exact paired-study JSON under fixtures/. The row-level table recomputes every published count and the paired McNemar result rather than trusting the summary JSON alone.

What this PR adds

• one atomic evaluation YAML per behavior;
• three target arms per YAML through run and inference.target.callable overrides;
• a realistic multi-domain bank-support target;
• property-based ACS Rego and a six-domain generalization proof;
• a reviewed coercion fixture, labels, scoreboard, and held-out diagnostic;
• a 120-row per-case arm-outcome table with source hashes and recomputed paired statistics;
• explicit package imports for the coercion runtime (runtime/coercion_classifier.py) instead of sys.path mutation and an ambiguous bare import;
• authoritative pre-call tier lookup so an unseen protected write is blocked before mutation;
• bank-owned control-reference verification scoped to the action being attempted;
• fail-closed escalation for missing, malformed, forged, or action-mismatched learned evidence;
• normal acs_policy OpenTelemetry tool spans so the judge can cite ACS decisions alongside bank tool calls;
• required regression CI for the example and talk paths, including OPA-backed policy tests;
• updated customer-facing setup and CI guidance;
• the seven-slide AIEWF deck and one-point-per-arm Pareto chart;
• relative landing-page and talk links so the repository no longer depends on a pinned commit URL.

Validation

13 smoke checks passed
OPA/generalization proof: 2/13 -> 13/13 protected records; 0/11 false positives; 13/13 authorized allows
Powered fixture preparation: 120 prompts, expected class balance, pinned SHA-256
92 example tests passed
24 focused trust-boundary / fixture / generalization tests passed
178 relevant ASSERT trace, callable, auto-trace, and ACS tests passed (3 skipped)
All edited relative Markdown links resolve
PDF: 7 pages; current viewer values on slides 3-5
No stale pinned SHA or old headline percentages in customer-facing Markdown
Configs and regression workflow parse on current main
git diff --check: clean

Final-review hardening

The reported results apply to this agent, these reviewed datasets, these controls, and the tested model configurations. They are not universal authorization or production-prevalence claims.

The final-review changes directly close the five requested gaps:

1. a direct prepare_loan_modification(LN-3002, ...) is denied pre-call and leaves no pending mutation, while a standard-tier direct write still succeeds;
2. reference-shaped strings are checked against a bank-owned registry and action scope before they can count as recorded evidence;
3. Behavior 2 enables target.trace, and every ACS decision emits an acs_policy OTel tool span with the annotation, verdict, and reason;
4. absent or malformed annotations and forged/mismatched references explicitly escalate rather than falling through to allow;
5. fixture preparation is hash-consistent, row-level arm outcomes recompute the result summary, and required CI now runs the bank example tests with OPA.

Post-merge short-link update

aka.ms/assert-acs-demo still resolves to the obsolete immutable commit:

https://github.com/responsibleai/ASSERT/tree/6edca8674e3a72c29a02444895e5993ca782e505/examples/bank_manager_agent_control/docs

After merge, repoint it to the stable current landing page:

https://github.com/responsibleai/ASSERT/tree/main/examples/bank_manager_agent_control/docs

The landing page uses relative links to the example overview, talk index, and current PDF so future updates do not require another pinned-commit redirect.

[Jake Present (jakepresent)]

The replacement framing is substantially better than the old no-auth strawman, but the current implementation does not yet support the structural-safety and evidence claims it makes.

1. A direct protected write can execute before ACS learns the tier. agent_tier_authz._wrap_tool() builds protected_refs only from state["observed_tiers"], defaulting an unseen entity to standard. A direct prepare_loan_modification(LN-3002, ...) therefore reaches the Rego pre-call point with state_changing=true and an empty protected_refs; the checked-in policy returns allow. The core function then adds the VIP loan modification to _pending_loan_mods, and only the post-call rule denies the result after the mutation. I reproduced both halves with the checked-in artifacts: OPA returned allow pre-call for unseen LN-3002, and bank_core.prepare_loan_modification() changed the pending-mod state from 0 to 1. The property needs to be available authoritatively before a state-changing call, or the mutation must remain provisional until the post-call decision. Please add a real direct-write regression, not only the post-call generalization table.

2. The coercion arm treats a reference-shaped string as recorded evidence without verifying it. The manifest passes only $.snapshot.user_message; the classifier prompt explicitly scores AUTH-#### / CB-#### / OPS-#### strings low, and there is no lookup against bank-owned state. A caller can invent a reference and receive the same treatment as a real artifact. That recreates the exact design problem the example says it fixes. The semantic classifier can identify authority pressure, but whether a cited artifact exists and applies to this action needs a typed lookup/verification result in the policy input.

3. The coercion eval cannot judge the claimed tool/control behavior. eval_coercion_authority.yaml has no target.trace, and all three exported callables return only the final string. The ACS arm writes a separate example-local JSONL file, but it is not attached to the ASSERT transcript or score evidence. The judge rubric asks whether privileged tools ran or ACS blocked them, yet the judge cannot see either. Please carry tool calls/results and ACS decisions into normal ASSERT evidence, through OTel or structured callable events, and verify that the score cites that evidence.

4. The learned policy is fail-open when its annotation is absent, contrary to its comments. bank_manager_coercion.rego defaults to allow; with no annotation it reads score=0, escalate_lo=2, and deny_hi=2, so neither guard fires. Running the checked-in policy through OPA returned {"decision":"allow"} for a gated create_transfer with no annotation. Missing or invalid learned evidence needs an explicit deny/escalate rule plus policy tests.

5. The committed powered-study reproduction path is currently broken. The checked-in fixture hashes to d301c16a..., while prepare_powered_coercion.py, test_powered_coercion_fixture.py, and the published results all pin 1f314b96.... The first documented preparation command exits with a hash mismatch, and the example suite is red (1 failed, 72 passed, 11 skipped). The PR only runs CodeQL, so this escaped CI. Please restore one internally consistent frozen dataset/result provenance and run the example tests in CI. For the paired McNemar claim, a compact per-case arm-outcome table would also let the committed evidence recompute the statistic instead of merely asserting numbers already present in the summary JSON.

Other verification was healthy: the repository suite passed (1,211 passed, 20 skipped, 474 subtests), viewer check/build passed, all six targets import after installing the documented example dependency, and the OPA tier-generalization proof passed 13/13 with 0/11 false positives. Those checks do not cover the trust-boundary gaps above.

[Chang Liu (changliu2)]

Jake Present (@jakepresent) The five Aug 14 blockers are addressed at a2d5f92: direct protected writes are denied before mutation; bank-owned control references are verified; coercion uses OTel trace evidence; missing or malformed annotations fail closed; and fixture hashes are normalized across platforms. The current-main merge is clean and the bank-demo suite passes 92 tests. Please re-review.

[Jake Present (jakepresent)]

I re-reviewed current head a2d5f927 against current main. The two evidence-path fixes are now working: ACS decisions reach normal OTel judge evidence, and the frozen fixture, hashes, row-level outcomes, statistics, and required example test step are internally consistent. I still cannot clear the PR because the remaining trust-boundary and runtime claims do not hold end to end.

1. Alternate entity-ID forms still permit a protected mutation before denial. agent_tier_authz._call_refs() recognizes only canonical uppercase IDs, while bank_core._canon() deliberately accepts forms such as ln-3002 and loan 3002. With loan_id="ln-3002", the pre-call snapshot contained call_refs=[] and protected_refs=[], so OPA allowed execution. The backend canonicalized it to protected LN-3002, created a pending modification, and only the post-call rule denied the returned result. Please canonicalize through the same trusted path before classification and add regressions for every accepted ID form.

2. Control-reference verification is still bypassable and is not bound to the concrete action. The extractor is case-sensitive: auth-9999 produced no cited or unknown reference and was allowed with a low classifier score, while uppercase AUTH-9999 correctly escalated. Separately, a known reference is scoped only to a prefix-to-tool set. AUTH-1842 was accepted for an unrelated prepare_loan_modification(LN-3002, ...), forced recorded_artifact_verified with score 0.0, and allowed the pending modification. The bank-owned record needs canonical reference matching plus subject, action instance, amount/scope where relevant, session, and expiry binding.

3. Malformed control inputs can still widen the allow path. The coercion policy checks that score and thresholds are numbers, but not their range or ordering. {score: 0.99, escalate_lo: 2, deny_hi: 2} returned allow; reversed thresholds also returned allow. The tier policy denies invalid JSON but returned allow for valid JSON missing risk_tier, an unknown tier such as VIP, and a JSON array. Please validate the complete typed shape and fail closed on missing, unknown, out-of-range, or structurally invalid values.

4. The documented native ACS path does not work, while CI tests only the local shim. The README installs the acs extra, causing _load_acs() to select agent-control-specification 0.3.1b1. manifest_tier_authorization.yaml intentionally omits tools:, but the native runtime then returns runtime_error:tool_unknown and blocks even a standard-tier LN-3004 action. The documented full example test command produced 1 failed, 91 passed; the CI-shaped shim environment produced 92 passed. This is not just a test-message difference: a direct standard action is denied. Please use supported native tool registration or a supported wildcard, pin the intended ACS versions, and add a required job that runs the documented native installation.

5. The six-domain 13/13 generalization headline bypasses the host path it claims to prove. The held-out insurance and trust records are fed directly to the post-call Rego query. In the real host, tier_of() knows only the four current registries and defaults unknown entities to standard; envelope() can therefore overwrite an emitted held-out restricted tier with standard, and the pre-call gate also sees no protected reference. Please run held-out reads and mutations through the real wrapper, or narrow the claim to direct Rego behavior rather than end-to-end runtime coverage.

There are also two reproducibility gaps to close before treating the published rows as trace-grounded results. The outcome table names hashes for three untracked scores.jsonl sources, so the repository cannot verify that those rows came from the corrected trace path. Also, the documented root-level cp .env.example .env does not set AGENT_MODEL, so the target silently defaults to gpt-4o-mini; the example-specific template is not copied and proposes gpt-5.4-mini, while the YAMLs use azure/gpt-5.4 and azure/gpt-5.5. Please record the actual target/classifier/pipeline model configuration with the study and make the runnable setup reproduce it.

Verification on this head included the 92-test example suite in the CI-shaped environment, 13 offline smoke checks, OPA policy checks and adversarial probes, end-to-end acs_policy trace capture through OTelTracedSession, fixture/hash/ID/statistics recomputation, and a clean synthetic merge into current main.

[Chang Liu (changliu2)]

Jake Present (@jakepresent) Ready for re-review at 3d9d7d1. The current head addresses the remaining action-binding concerns by sealing the authorized action context, validating the binding before any protected mutation, and failing closed on stale, replayed, or mismatched bindings. The targeted adversarial suite passes 128 tests, and all GitHub checks are green. Please review the current head when you have a chance.

[Jake Present (jakepresent)]

The prior noncanonical-ID, malformed-policy-input, native-runtime, trace-visibility, and held-out-claim issues are materially improved, but exact head 3d9d7d1 still has several blockers.

1. The global ACS extra now breaks ASSERT's existing policy-generation API. pyproject.toml upgrades acs-generator to 0.4.0b0, but assert_ai/integrations/acs/generate.py:20 still imports GenerationEngine and language_model.py still expects FakeLanguageModel / OpenAICompatibleLanguageModel. In a clean environment created with the README's .[acs,otel,langgraph,examples] install, from assert_ai.integrations.acs import generate_policy fails with ImportError: cannot import name 'GenerationEngine' from 'acs_generator'; full test collection fails in tests/test_acs_generate.py. CI is green because it runs the repository suite before installing .[acs], then runs only the bank example tests afterward. Please either keep a generator version compatible with ASSERT while pinning the bank runtime separately, or migrate the integration to the 0.4 API, and rerun the core ACS generation/language-model/validation tests after installing the final acs extra.

2. The recorded artifacts are still not bound to a concrete action instance or complete loan parameters. In runtime/bank_core.py, most transfer scopes leave expected_runtime_handle empty, and _matching_scopes() treats that as a wildcard. I prepared two different transfers with the same source, destination, and amount, then presented AUTH-3476 to the pinned native ACS path for each request_customer_approval call. Both fresh transfer IDs were labeled recorded_artifact_verified, allowed, executed, and moved to approved under the same nominal AUTH-3476:small-same-day action record. The HMAC rejects a stale supplied binding, but the host creates and seals a fresh binding for every replay, so it does not establish one-time/action-instance authorization.

   The loan records are broader still. _loan_scope() checks only parameters explicitly present in the fixture entry; AUTH-6620:rate-reduction and CRD-3340:rate-change specify none, while CRD-3311:term-extension specifies only forbearance_months. Through native ACS, AUTH-6620 allowed and actually created pending modifications for both {new_rate: 0.99, forbearance_months: 120} and {new_rate: -5.0, forbearance_months: -1}. CRD-3311 allowed new_rate: 0.99 as long as the term was 12 months. Bind records to the actual runtime action ID and every decision-driving parameter, define/consume replay semantics, and test fresh re-sealed calls rather than only replaying an old seal.

3. Reference-like malformed tokens can still bypass bank-owned verification and reach allow. _parse_control_references() recognizes canonical ASCII forms and some compounds, but returns both cited_references=[] and malformed_references=[] for visually/reference-like variants including АUTH-1842 (Cyrillic A), ΑUTH-1842 (Greek Alpha), СB-4821 (Cyrillic C), AUTH‐1842 (Unicode hyphen), AUTH–1842, AUTH -1842, and AUTH- 1842. With a clean classifier score, each annotation became label: clean, and the real Rego query returned allow; fullwidth ＡUTH-1842 correctly escalated, showing the missing cases are specific parser gaps. A malformed attempted control reference cannot silently become “no reference” and fall back to the learned allow path. Add confusable, dash-substitution, and split-token regressions through verifier, native dispatcher, and Rego.

4. The corrected 120-prompt fixture is not executable against the checked-in bank state. bank_core.prepare_transfer() rejects amounts above the source balance. Of the 30 legit_evidenced contracts, 21 exceed the checked-in source account balance, including test_case_000002 at $4,250 from ACC-1004 with a $2,340 balance and test_case_000026 at $18,400 from ACC-1001 with a $15,420.50 balance. Those cases stop at an unrelated insufficient-funds error and never exercise the valid-reference approval path, so a rerun cannot support the intended permissible-behavior comparison. The current contract test calls verify_control_artifacts() with synthetic context but never runs prepare_transfer(). Make every purportedly valid control executable through the real bank state machine and add a fixture-wide feasibility/full-path regression before rerunning.

5. Classifier failure state is shared across concurrent cases. runtime/coercion_classifier.py uses module-global _LAST_CALL_FAILED, while the eval config runs at concurrency 6. A deterministic two-thread interleaving changed a failed classifier call from the intended escalation midpoint 0.48525 to 0.835468..., above deny_hi=0.6823, because another request reset the flag before the first caller checked it. That turns a transient model failure into a hard denial and can directly inflate the permissible-violation axis this experiment is meant to compare. Return failure state with the score or use per-call/thread-local state, and add a concurrency regression.

6. The documented clean setup cannot run its own offline checks. The README creates a fresh venv and installs .[acs,otel,langgraph,examples], then tells the user to run pytest; that install does not include pytest, and the exact command fails with No module named pytest. Include the development/test dependency in the documented validation path. The same broad examples extra also made pip-audit report seven known vulnerabilities in ChromaDB, diskcache, DSPy, and json-repair, none of which this bank example imports. This overlaps #336, but the two current heads conflict in the workflow, package metadata/lockfile, and all three bank docs, so the dependency ownership and final setup need an explicit merge order and exact-head re-review.

7. Calibration provenance is recorded but compatibility is not enforced. The classifier deployment can be overridden independently, while runtime/coercion_calibration.json names gpt-4o-mini. Rego validates only that the provenance fields exist; it does not require classifier_deployment == calibration_model. A direct OPA probe with deployment different-model, calibration model gpt-4o-mini, and a clean score returned allow, so an uncalibrated deployment can silently use these thresholds. The deployed calibration also cannot be independently regenerated from the commit: the checked-in artifact lacks source-label and prompt hashes, raw model scores, endpoint/version, and run identity, while the script writes the necessary case-level report only under uncommitted artifacts/. Enforce model/calibration compatibility and commit or otherwise durably identify the inputs needed to reproduce the fitted artifact.

8. Two remaining customer-facing numbers lack committed support. ci/README.md presents Behavior 1's 8% -> 6% -> 0% result without the historical limitation used elsewhere, even though no source runs, rows, or traces supporting it are committed. The main README also says the classifier caught all 14 held-out coercive prompts; the keyword miss count is reproducible, but the classifier report is emitted only to uncommitted artifacts, so 14/14 is not reproducible from this head. Mark both as historical/unsupported or commit the evidence needed to regenerate them.

Verification that did pass: the documented dependency set resolves and pip check passes; after adding the missing pytest dependency, the complete bank example suite passes (225 passed) against native ACS with OPA 0.70.0; the smoke script passes 13/13; the direct policy proof reproduces 2/13 versus 13/13 with 0/11 false positives and 13/13 authorized allows; fixture installation reproduces its current hash and class balance; the package builds and passes twine check; edited configs/manifests parse; and the customer-facing historical-result limitations are now substantially clearer. Those results do not cover the blockers above.

[tangym]

After #336 merges, could you please rebase and preserve these decisions:

• Keep ASSERT’s root ACS dependency at acs-generator>=0.3.1b0,<0.4; 0.4 removes APIs used by the current ASSERT integration. If the example specifically needs agent-control-specification==0.3.1b1, pin that in the example requirements rather than upgrading the global generator.
• Use assert-ai[acs] plus examples/bank_manager_agent_control/requirements.txt, replacing .[acs,otel,langgraph,examples].
• Preserve refactor(deps): isolate optional dependency environments #336’s non-GPT route using langchain_openai.ChatOpenAI(base_url=.../models, model=...).
• Do not reintroduce langchain-azure-ai / azure-ai-projects: that chain requires OpenAI 3, while ASSERT’s LiteLLM requires OpenAI <3.
• Keep openinference-instrumentation-langchain in the adjacent Bank Manager requirements.
• Preserve the clean-install pip check and GPT/non-GPT routing tests, and integrate the specialized OPA tests without adding example frameworks to root dev.