FIX: Integration test fixes#1897
Merged
Merged
Conversation
labels required to be non-None in order for MessagePiece not to raise and interrupt test_retry_timing_integration. Fix was to apply labels or {} on line 72 of tests/integration/mocks.py.
…ntegration test ScenarioRegistry.list_metadata() instantiates every registered scenario class to build metadata. Cyber.__init__ calls _build_cyber_strategy(), which calls AttackTechniqueRegistry.get_factories_or_raise() — raising RuntimeError when the registry is empty. The integration test was missing the ScenarioTechniqueInitializer step that populates the registry. Add it before LoadDefaultDatasets.initialize_async(). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
romanlutz
approved these changes
Jun 2, 2026
…C/PyRIT into integration_test_fixes Merging changes from main.
…ast changed in harmbench_dataset in PR 1780
…attack registry failures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes to failing integration tests.
Description
test_translation_converter_exponential_backoff_timing— MockPromptTarget crash before retry loop (tests/integration/mocks.py): A recent refactor tightenedMessagePiece.labelsto non-optionaldict[str, Any]. The integration mock'sset_system_promptwas passinglabels=Nonedirectly toMessagePiece(...), causing Pydantic validation to fail beforesend_prompt_asyncwas ever called. Fixed with a one-linelabels=labels or {}guard, matching what the production base class already does.test_initialize_loads_datasets_into_memory— emptyAttackTechniqueRegistry(tests/integration/datasets/test_load_default_datasets_integration.py):LoadDefaultDatasets.initialize_async()callsScenarioRegistry.list_metadata(), which instantiates every registered scenario class includingCyber.Cyber.__init__builds its strategy enum fromAttackTechniqueRegistry, raisingRuntimeErrorwhen the registry is empty. The test was callinginitialize_pyrit_asyncbut notScenarioTechniqueInitializer, which is the initializer responsible for populating the registry. Addedawait ScenarioTechniqueInitializer().initialize_async()before theLoadDefaultDatasetscall.test_harmbench_metadata_parses_correctly— wrong size assertion (tests/integration/datasets/test_seed_dataset_provider_integration.py): The test assertedmetadata.size == {"large"}but_HarmBenchDataset._parse_metadata()returns{"medium"}. Updated the assertion to match the actual metadata.test_execute_notebooks[0_scenarios.ipynb]and related scenario notebooks — emptyAttackTechniqueRegistry(doc/code/scenarios/0_scenarios.py,1_common_scenario_parameters.py,2_custom_scenario_parameters.py): Same root cause as theLoadDefaultDatasetstest above. Each notebook callsinitialize_pyrit_async(orinitialize_from_config_async) but notScenarioTechniqueInitializer, so the registry is empty whenlist_scenarios_async()or scenario constructors are invoked. Added a singleawait ScenarioTechniqueInitializer().initialize_async()at the top of each notebook's setup cell. Also added missinginitialize_pyrit_asyncto2_custom_scenario_parameters.pywhich had no memory setup at all.Tests and Documentation
Updates listed above.