FIX: unblock main CI — Test GUI (PyPI), Crescendo parser, CoCoNot empty prompt#1862
Merged
ValbuenaVC merged 2 commits intoJun 1, 2026
Merged
Conversation
…ty prompt Three independent fixes for failures observed on main HEAD 5eab2f8: 1. docker_build / Test GUI (PyPI) — `ModuleNotFoundError: No module named 'alembic'`. `WORKDIR=/app` plus a `COPY pyrit/ /app/pyrit/` meant `python -m pyrit.*` imported the local source instead of the installed PyPI wheel. Local source pulls in `alembic` (added after 0.13.0), so the GUI container crashed in lifespan. The Dockerfile now removes the local source for `PYRIT_SOURCE=pypi` after install. The PyPI 0.13.0 launcher also lives at `pyrit.cli.pyrit_backend` (PR microsoft#1753 moved it to `pyrit.backend.pyrit_backend`), so start.sh now falls back to the legacy module name when the new one is missing — the fallback becomes dead code after the next release. 2. Integration Tests / 4_sequential_attack.ipynb — Crescendo's `_parse_adversarial_response` required snake_case keys but the adversarial chat returned `generatedQuestion` / `rationaleBehindJailbreak` / `lastResponseSummary` for three retries straight. Normalize incoming keys to snake_case before validation; extra-key rejection is preserved. 3. End to End Tests / test_fetch_dataset[_CoCoNotRefusalDataset] — `original.train` (wildchats subcategory) contains rows with empty `prompt`, producing `SeedObjective(value='')` that trips the `seed.value` invariant. Skip empty/whitespace prompts in the loader and log a warning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jsong468
reviewed
Jun 1, 2026
jsong468
reviewed
Jun 1, 2026
jsong468
approved these changes
Jun 1, 2026
Add /app/README.md and /app/LICENSE to the rm in the PYRIT_SOURCE=pypi branch so the cleanup mirrors the COPY block one-to-one. /app/doc is intentionally retained because the later RUN block copies it into /app/notebooks/ for Jupyter mode; documented that in the comment block above the RUN. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three independent CI failures on
main(HEAD5eab2f8c), one fix each. Investigation notes are in the commit message; the short version:1.
docker_build→Test GUI (PyPI)—ModuleNotFoundError: No module named 'alembic'docker/DockerfileunconditionallyCOPY pyrit/ /app/pyrit/withWORKDIR=/app, sopython -m pyrit.*imported the local source instead of the PyPI-installed wheel. Local source now usesalembic(PR #1631), but PyPI 0.13.0''s metadata doesn''t list it → lifespan crash. Fix: forPYRIT_SOURCE=pypi,rm -rfthe local source after install so the installed 0.13.0 wheel actually wins.PR #1753 also moved the launcher from
pyrit.cli.pyrit_backendtopyrit.backend.pyrit_backend; PyPI 0.13.0 only has the old path.docker/start.shnow tries the new module first and falls back to the legacy module for older PyPI versions. The fallback becomes dead code after the next release ships.2.
Integration Tests→4_sequential_attack.ipynb— Crescendo retries exhausted on camelCaseCrescendoAttack._parse_adversarial_responserequired snake_case keys, but the adversarial chat returnedgeneratedQuestion/rationaleBehindJailbreak/lastResponseSummaryfor three retries straight and burned the whole budget. Normalize keys to snake_case before validation; the strict extra-key check is preserved.3.
End to End Tests→_CoCoNotRefusalDataset—Seed in _CoCoNotRefusalDataset has no valueoriginal.train(wildchatssubcategory) contains rows with emptyprompt, producingSeedObjective(value='''')which tripstests/end_to_end/test_all_datasets.py''sassert seed.valueinvariant. Skip empty/whitespace prompts in the loader with a warning.Out of scope (separate sessions)
The other failing items on this run aren''t included here and will be handled in follow-ups:
_ComicJailbreakDatasettimeout (>300s) and_VLGuardDataset401 gated-repoairt.cyberscenario exception +Partner Integration Tests(need AzDO logs)Verification
pytest tests/unit/datasets/ tests/unit/executor/attack/multi_turn/→ 865 passedpre-commit run --files <changed>→ all hooks green (ruff, ruff-format, ty)bash -nondocker/start.shand on the Dockerfile RUN block → clean