FEAT: Adversarial Benchmark Scenario Refactor#1765

Merged

ValbuenaVC merged 62 commits into

microsoft:mainfrom

ValbuenaVC:adversarial_benchmark_refactor

Jun 1, 2026

Commits on May 20, 2026

new .env_example with adversrial target models.
Victor Valbuena
committed
.env_example formatting
Victor Valbuena
committed
More .env_example formatting
Victor Valbuena
committed
Reorganized unit test directory
Victor Valbuena
committed
Renaming unit tests for consistency
Victor Valbuena
committed

Commits on May 21, 2026

FIX: TargetInitializer propagates config.tags to registry entries
Victor Valbuena
committed
FEAT: Add TargetRegistry.get_by_tag_query for TagQuery-based lookup
Victor Valbuena
committed
FEAT: Register ADVERSARIAL_CHAT singleturn/multiturn/reasoning variants
Victor Valbuena
committed
FEAT: Add BenchmarkInitializer for adversarial-target fan-out
Victor Valbuena
committed
Merge remote-tracking branch 'origin/main' into adversarial_benchmark_refactor
Victor Valbuena
committed
REFACTOR: Collapse AdversarialBenchmark to RapidResponse pattern
Victor Valbuena
committed
FEAT: Add skip_cached cross-run caching to AdversarialBenchmark
Victor Valbuena
committed
Merge branch 'main' into adversarial_benchmark_refactor
ValbuenaVC
authored
FEAT: Widen AdversarialBenchmark objective_scorer to Scorer (stage 1)
Victor Valbuena
committed
TEST: e2e per-scenario initializer override for benchmark.adversarial
Victor Valbuena
committed
DOCS: Rewrite benchmark scanner notebook for registry-driven flow
Victor Valbuena
committed
STYLE: Move TYPE_CHECKING-only imports in target_registry

Victor Valbuena
and
Copilot
committed
FIX: TargetInitializer initialized adversarial chats as OpenAIChatTarget. Changed to AzureMLChatTarget.
Victor Valbuena
committed
Merge branch 'main' into adversarial_benchmark_refactor
ValbuenaVC
authored

Commits on May 22, 2026

Merge branch 'main' into adversarial_benchmark_refactor
ValbuenaVC
authored

Commits on May 26, 2026

Merge branch 'main' into adversarial_benchmark_refactor
ValbuenaVC
authored
Removed _cached_strategy_class and scorer type validation.
Victor Valbuena
committed
REFACTOR: Remove BenchmarkInitializer and source adversarial targets via scenario parameter

Victor Valbuena
and
Copilot
committed
Merge branch 'main' into adversarial_benchmark_refactor
ValbuenaVC
authored
Merge branch 'main' into adversarial_benchmark_refactor
ValbuenaVC
authored
DOCS: Trim adversarial benchmark scanner doc to basic-usage shape

Victor Valbuena
and
Copilot
committed
DOCS: Add executed benchmark scanner notebook outputs

Victor Valbuena
and
Copilot
committed
Add mcp Python SDK dependency

Victor Valbuena
and
Copilot
committed
Merge branch 'main' into MCP
ValbuenaVC
authored
Add tools/ package with tool_loop decorator and CallableToolBackend

Victor Valbuena
and
Copilot
committed
Addition of pyrit/tools package and unit tests. Introduces base models for tool calling.
Victor Valbuena
committed
Addition of MCP components including the MCP client and tool backend.
Victor Valbuena
committed

Commits on May 27, 2026

Add supports_tool_use capability + ToolEventPolicy and wire tool_loop into PromptTarget.send_prompt_async

Victor Valbuena
and
Copilot
committed
Merge branch 'main' into MCP
ValbuenaVC
authored

Commits on May 28, 2026

Drop C5 (Chat target tool calling): defer to follow-up, redesign around Response target

Victor Valbuena
and
Copilot
committed
Migrate OpenAIResponseTarget onto @tool_loop and LocalToolBackend

Victor Valbuena
and
Copilot
committed
DOCS: Migrate reST roles, drop dead spec-selector and stale scorer-flex copy

Victor Valbuena
and
Copilot
committed
Add integration tests for RedTeamingAttack with real MCP tool dispatch

Victor Valbuena
and
Copilot
committed
REFACTOR: Delete unused `TargetInitializerTags.ADVERSARIAL`

Victor Valbuena
and
Copilot
committed
DOCS: Compare singleturn vs multiturn adversarial targets in benchmark notebook

Victor Valbuena
and
Copilot
committed
Merge branch 'main' into adversarial_benchmark_refactor
Victor Valbuena
committed
Merge branch 'main' into adversarial_benchmark_refactor
ValbuenaVC
authored
FEAT: Add ObjectiveTargetEvaluationIdentifier + analytics cache lookup

Victor Valbuena
and
Copilot
committed
Merge branch 'vvalbuena-microsoft/mcp-target-decorator-design' into adversarial_benchmark_refactor
Victor Valbuena
committed
Merge branch 'main' into adversarial_benchmark_refactor
ValbuenaVC
authored
FEAT: Wire adversarial benchmark cache to analytics primitives

Victor Valbuena
and
Copilot
committed
TEST: Add real-memory coverage for adversarial cache wiring

Victor Valbuena
and
Copilot
committed
Merge remote-tracking branch 'origin/main' into vvalbuena-microsoft/fluffy-potato
Victor Valbuena
committed

Commits on May 29, 2026

Merge branch 'main' into adversarial_benchmark_refactor
ValbuenaVC
authored
Merge branch 'main' into adversarial_benchmark_refactor
ValbuenaVC
authored
Removing tool components from this branch. Tool-calling is a feature also in PR (1811), but its components were incorrectly added to this PR.
Victor Valbuena
committed
Remove remaining tool-calling leakage from PR #1811

Victor Valbuena
and
Copilot
committed
Merge resolution.
Victor Valbuena
committed
Fix merge conflicts with main (PR #1785/#1784): adopt factory registry API

Victor Valbuena
and
Copilot
committed
Merge branch 'main' into adversarial_benchmark_refactor
ValbuenaVC
authored

Commits on May 30, 2026

Merge branch 'main' into adversarial_benchmark_refactor
ValbuenaVC
authored

Commits on Jun 1, 2026

Merge branch 'main' into adversarial_benchmark_refactor

Victor Valbuena
and
Copilot
committed
FEAT: inject cached attack results into AdversarialBenchmark ScenarioResult

Victor Valbuena
and
Copilot
committed
feat: add dataset-level scoping to _collect_cached_completion_pairs

Victor Valbuena
and
Copilot
committed
Merge remote-tracking branch 'origin/main' into adversarial_benchmark_refactor
Victor Valbuena
committed
fix: use uuid4 for conversation_id in _persist_attack_result to prevent CI collision

Victor Valbuena
and
Copilot
committed
Merge branch 'main' into adversarial_benchmark_refactor
ValbuenaVC
authored