FEAT: Run atomic attacks in parallel (better) within a scenario#1783
Merged
rlundeen2 merged 7 commits intoMay 28, 2026
Merged
Conversation
rlundeen2
force-pushed
the
users/rlundeen/05_22_scenario_parallel
branch
from
c9926be to
22fb8b5
Compare
Allow multiple atomic attacks in a scenario to run concurrently, driven by the existing max_concurrency parameter. All in-flight objectives across all atomic attacks share a single asyncio.Semaphore(max_concurrency), so the global concurrent-objective budget is bounded by max_concurrency regardless of how work is distributed across atomic attacks. A long-running attack can elastically use slots freed by short-running siblings. Changes: - AttackExecutor now accepts an optional external semaphore kwarg. When provided, it gates both seed-group parameter building and per-objective execution, letting a parent (e.g. Scenario) share one budget across many executors. - AtomicAttack.run_async forwards the optional semaphore to its executor. - Scenario._execute_scenario_async: when max_concurrency > 1 and more than one atomic attack remains, creates one shared semaphore and launches every remaining atomic attack via asyncio.gather, all sharing that semaphore. When max_concurrency == 1 (or only one attack remains), keeps the existing sequential loop verbatim, preserving abort-on-first-failure semantics. - Parallel failure mode uses gather(return_exceptions=True) so in-flight siblings finish before the first error is re-raised (preserves partial work for resume). - No new user-facing parameters or CLI flags. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
rlundeen2
force-pushed
the
users/rlundeen/05_22_scenario_parallel
branch
from
22fb8b5 to
b3bd470
Compare
…emaphore Per code-review feedback, replace the `semaphore` parameter threaded through the executor and atomic-attack APIs with a shared `AttackExecutor` instance. The semaphore becomes a pure implementation detail of `AttackExecutor`. Changes: - `AttackExecutor.__init__` no longer accepts `semaphore`; it always builds its own internal `asyncio.Semaphore(max_concurrency)`. - `AtomicAttack.run_async` now accepts `executor=` (optional). When omitted, a fresh `AttackExecutor(max_concurrency=max_concurrency)` is built. The `max_concurrency` kwarg is deprecated and emits a `DeprecationWarning`; when `executor` is also passed, the deprecated value is ignored (warning surfaces this). - `Scenario._execute_atomic_attacks_parallel_async` builds one `AttackExecutor(max_concurrency=self._max_concurrency)` and passes it to every `atomic_attack.run_async` via the new `executor=` kwarg, sharing the single concurrency budget across all in-flight atomic attacks. Tests updated to assert on the shared `executor` instance and to verify the deprecated `max_concurrency=` path still works and warns. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jsong468
reviewed
May 27, 2026
jsong468
reviewed
May 27, 2026
jsong468
reviewed
May 27, 2026
jsong468
reviewed
May 27, 2026
jsong468
reviewed
May 27, 2026
- Switch deprecated atomic_attack.max_concurrency warning to print_deprecation_message (was raw warnings.warn). Default arg flipped to a None sentinel so passing explicit max_concurrency=1 also warns. - Reword Scenario.initialize_async max_concurrency docstring to make the "unit of work" definition explicit (objective build OR objective run) and call out the shared AttackExecutor. - Surface every in-flight failure in parallel mode: collect errors and wrap multiples in ExceptionGroup; single failures re-raise as-is to keep the common case readable. Add exceptiongroup>=1.2.0 backport for Python 3.10. - Make AttackExecutor's semaphore loop-aware: build it lazily inside _get_semaphore() and rebuild when the running event loop changes, so the executor can be safely reused across asyncio.run() calls. - Strengthen the deprecation style-guide section: forbid raw warnings.warn explicitly, document the sentinel pattern for deprecating argument values, and add an INCORRECT example. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
rlundeen2
force-pushed
the
users/rlundeen/05_22_scenario_parallel
branch
from
5ab4a40 to
b5d6a9f
Compare
hannahwestra25
approved these changes
May 28, 2026
jsong468
reviewed
May 28, 2026
jsong468
reviewed
May 28, 2026
jsong468
approved these changes
May 28, 2026
True parallel execution across atomic attacks (added earlier in this PR) makes a default budget of 10 too aggressive for the typical case where both objective scoring and target calls share that budget. Drop to 4 and keep the rest of the documented semantics intact. - Scenario.initialize_async default 10 -> 4; matching docstring update. - FoundryRedTeamAgentScenario override default 10 -> 4 to stay in sync with the base default (the override exists only for type-widening). - Scenario.__init__: drop the bogus self._max_concurrency = 1 sentinel (initialize_async always overwrites it). Type is now Optional[int], matching other "set in initialize_async" attributes, and the parallel executor narrows it once at the top of the function. - Tests: update the two assertions that pinned the pre-init value to 1, plus the comment that referenced max_concurrency=10. Note: the backend Pydantic model (ScenarioRunRequest.max_concurrency) still defaults to 10 -- that's a separate API surface and out of scope for this change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- atomic_attack.py: use 'unknown' instead of '?' in the executor max_concurrency log fallback (review microsoft#7). - scenario.py: extract the post-gather error-collection loop into a new _collect_errors_from_outcomes helper for readability (review microsoft#10). - scenario.py: rewrite the comment block above the parallel-execution branch per Justin's suggestion to name the AttackExecutor-level Semaphore and the parameter-build + attack-execution units of work (review microsoft#11). - test_attack_executor.py: add four regression tests for AttackExecutor._get_semaphore covering lazy construction, in-loop reuse, and rebuild across asyncio.run() invocations (review microsoft#12). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
hannahwestra25
pushed a commit
to hannahwestra25/PyRIT
that referenced
this pull request
Jun 4, 2026
- Drop 'two people running the same pass...' sentence; switch opener to 'Enter scenarios.' - Add a 'Strategies - the runtime knob' bullet so the runtime-selection concept is introduced before it's used in RapidResponse / adaptive sections. - Add adaptive scenarios as a fifth flavor in the catalog with a link to its section; mark RapidResponse and AdversarialBenchmark as new in v0.14. - Rework RapidResponse closer-look: replace 'where it's soft' phrasing, drop the 'crosses two axes' framing (every scenario does that), keep the grouping detail as the actually-RR-specific bit. - Apply the 'less about building out our scenario library' wording, then explicitly surface the two new v0.14 scenarios so the section isn't misleading about scope. - Strip every github.com PR link. - Add two new paragraphs: 'Configuration from the CLI and from YAML' and 'Parallel execution within a scenario' to cover microsoft#1680 and microsoft#1783. - Add a mermaid diagram to the Better Scenario Tracking paragraph showing scenario_run_id flowing into memory and out to resume / analytics / printer / adaptive selector. - Drop the 'Smaller polish worth knowing about' paragraph. - Pull ASR mention into the adaptive section's opening paragraph. - Point the 'Where to go next' bullet at the real adaptive notebook path (doc/code/scenarios/3_adaptive_scenarios.ipynb). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously, scenarios could somewhat run in parallel within an atomic attack.
This allows multiple atomic attacks in a scenario to run concurrently, driven by the existing max_concurrency parameter. All in-flight objectives across all atomic attacks share a single asyncio.Semaphore(max_concurrency), so the global concurrent-objective budget is bounded by max_concurrency regardless of how work is distributed across atomic attacks. A long-running attack can elastically use slots freed by short-running siblings.