Skip to content

MAINT: migrate AdversarialBenchmark off deprecated PromptChatTarget#1746

Merged
romanlutz merged 1 commit into
microsoft:mainfrom
romanlutz:romanlutz/promptchattarget-removal-check
May 18, 2026
Merged

MAINT: migrate AdversarialBenchmark off deprecated PromptChatTarget#1746
romanlutz merged 1 commit into
microsoft:mainfrom
romanlutz:romanlutz/promptchattarget-removal-check

Conversation

@romanlutz
Copy link
Copy Markdown
Contributor

Description

pyrit/scenario/scenarios/benchmark/adversarial.py still typed its public adversarial_models parameter as list[PromptChatTarget]. PromptChatTarget was deprecated in #1678 with removal scheduled for v0.16.0, so without this migration the public AdversarialBenchmark(adversarial_models=...) API will break at import time once v0.16 ships.

Why it slipped

Merge race between #1662 (Adversarial Benchmark Scenario, merged 2026-05-07) and the surrounding deprecation work:

No follow-up PR addresses it; this is the only remaining PromptChatTarget reference in pyrit/ outside the deprecated shim itself.

Changes

  • Type adversarial_models as list[PromptTarget].
  • Validate each supplied target against CHAT_TARGET_REQUIREMENTS in __init__ so callers get a clear, early error if a non-chat target sneaks in. Per-technique stricter requirements (e.g. SYSTEM_PROMPT for RedTeamingAttack) continue to be enforced at factory.create() time, unchanged.
  • Update docstrings and error messages to describe the capability requirements rather than the deprecated class.
  • Update test mocks to stamp a real TargetConfiguration (multi-turn, multi-message-pieces, system-prompt, editable-history — matching what PromptChatTarget._DEFAULT_CONFIGURATION advertised) so the existing test suite still exercises the techniques that demand these capabilities.
  • Add a unit test asserting that an adversarial_models entry missing chat-target capabilities is rejected at construction.
  • Sweep the two doc/scanner/benchmark.{py,ipynb} comments that referenced "PromptChatTargets".

Tests and Documentation

  • uv run pytest tests/unit/scenario/test_adversarial.py — 34 passed
  • uv run pytest tests/unit/scenario/ — 579 passed
  • uv run ruff format --check + ruff check + ty (type check) — pass via pre-commit
  • Doc references in doc/scanner/benchmark.{py,ipynb} updated to the new wording.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hannahwestra25
Copy link
Copy Markdown
Contributor

thanks ! was this caught by our integration tests ? I could be missing something but it seems like we don't have any integration tests running the doc/scanner notebooks

@hannahwestra25 hannahwestra25 self-assigned this May 18, 2026
@romanlutz
Copy link
Copy Markdown
Contributor Author

@hannahwestra25 copilot found this while doing a general assessment of the repo. It said something like it's a result of your PR (which removed it) merging around the same time as the adversarial benchmark PR got merged. I don't think the integration test was run because those happen outside of our PR gates.

@romanlutz romanlutz added this pull request to the merge queue May 18, 2026
Merged via the queue into microsoft:main with commit 6055d23 May 18, 2026
48 checks passed
@romanlutz romanlutz deleted the romanlutz/promptchattarget-removal-check branch May 18, 2026 20:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants