Skip to content

ci(sandbox): run real Docker containment tests - #328

Merged
Jake Present (jakepresent) merged 8 commits into
mainfrom
jake/ci-sandbox-docker-tests
Aug 28, 2026
Merged

ci(sandbox): run real Docker containment tests#328
Jake Present (jakepresent) merged 8 commits into
mainfrom
jake/ci-sandbox-docker-tests

Conversation

@jakepresent

@jakepresent Jake Present (jakepresent) commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Problem

ASSERT's real Docker sandbox tests were opt-in, and the ordinary unit workflow could collect and skip all of them while still reporting success. A path-filtered workflow also could not serve as a reliable required check.

Fix

Add an always-present GitHub Actions workflow that:

  • runs on every pull request and every push to main, avoiding path-filter omissions and pending required checks;
  • explicitly fails if the Docker daemon is unavailable;
  • makes the test module fail instead of skip when ASSERT_RUN_DOCKER_TESTS=1 but Docker cannot run;
  • executes every tests/test_sandbox_*docker*.py module rather than one hard-coded file;
  • uploads a JUnit receipt for the exact tests that ran; and
  • always checks for leftover target/relay containers and networks, including after failure.

Verification

Remaining repository gate

The current main ruleset requires review but does not require status checks. After this workflow merges, Sandbox: real Docker containment should be added as a required status check with strict base-branch freshness. The workflow now runs on every PR, so making it required will not leave unrelated PRs waiting for a path-filtered check that never starts.

A clean local image rebuild remains blocked by the local Docker builder's external PyPI TLS failure; the clean build and test path passed in GitHub Actions.

Comment thread .github/workflows/sandbox-docker.yml Fixed

@changliu2 Chang Liu (changliu2) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two CI blockers on exact head 0db98e0:

  1. When Docker is unavailable, tests/test_sandbox_runtime_docker.py skips all four tests and the workflow still succeeds. Add an explicit failing Docker preflight so an outage cannot produce a false green.
  2. The workflow path filters omit direct sandbox dependencies such as assert_ai/core/security.py. Prefer assert_ai/** or include the complete dependency surface.

Until these are fixed, green CI does not reliably prove real Docker containment.

@changliu2 Chang Liu (changliu2) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two previous blockers are resolved: Docker unavailability now fails explicitly, xfail/skip cannot mask the proof, the receipt confirms four executed tests, and removing path filters covers all relevant changes. One required-check reliability blocker remains: the concurrency group uses only github.head_ref, so PRs from different forks with the same branch name can cancel each other’s required check. Please key PR runs by github.event.pull_request.number, with github.ref as the non-PR fallback.

@jakepresent

Copy link
Copy Markdown
Collaborator Author

Fixed the concurrency collision in 3fa87f9: PR runs now key the Docker job by pull-request number, with github.ref as the push and manual-run fallback. I added a workflow regression, and all checks including real Docker containment are green. Chang Liu (@changliu2) ready for re-review.

@changliu2 Chang Liu (changliu2) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR-number concurrency key fixes the cross-fork cancellation blocker, and the earlier Docker preflight, path coverage, xfail/skip receipt, and cleanup fixes remain sound. One new false-green path was introduced on this head: tests/test_sandbox_docker_workflow.py matches the workflow selector tests/test_sandbox_*docker*.py. Because the receipt only requires a positive test count, deleting or renaming all real container tests would still run this static YAML test and let the "real Docker containment" gate pass with one test. Please keep the static workflow test outside that selector (for example, rename it) or explicitly select/mark the real Docker containment tests, so the receipt cannot be satisfied without executing one.

@jakepresent

Copy link
Copy Markdown
Collaborator Author

Fixed the false-green selector on exact head e80bdef: the Docker workflow now explicitly runs only tests/test_sandbox_runtime_docker.py, and the static workflow regression locks that exact selector. The GitHub receipt contains the four real container tests with zero failures, errors, or skips, and all checks are green. Chang Liu (@changliu2) ready for re-review.

@changliu2 Chang Liu (changliu2) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed exact head e80bdef. The static-test false-green blocker is resolved: the workflow now selects only tests/test_sandbox_runtime_docker.py, the regression test pins that selector, and the hosted receipt contains four executed tests with no failures, errors, or skips. The earlier Docker-daemon preflight, all-change path coverage, xfail/skip protection, cleanup proof, and PR-scoped concurrency fixes remain sound. Approved.

@jakepresent
Jake Present (jakepresent) merged commit ec453d6 into main Aug 28, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants