test: add copilot network-isolation egress smoke test by lpcox · Pull Request #6330 · github/gh-aw-firewall

lpcox · 2026-07-17T17:14:10Z

Summary

Closes the network-isolation CI coverage gap tracked in #6326 (Gap 2).

Today every Copilot smoke test already runs under the AWF network-isolation topology (isolation: true), but the only egress check runs on the runner (a pre-step), not inside the sandbox — and nothing asserts that a non-allowlisted host is actually blocked. So the egress-enforcement path has no automated regression coverage.

This adds a dedicated CI-integrated smoke test, smoke-copilot-network-isolation, that runs the Copilot engine under network isolation (sandbox.agent.sudo: false, isolation: true) and asserts the egress allowlist is enforced from inside the sandbox:

egress ALLOW — curl https://api.github.com/zen succeeds (allowlisted)
egress DENY — curl https://example.com is blocked (not allowlisted)
agent completion — the agent must call the add_comment safe-output, validated by post-steps

Both curl checks are executed by the agent via the bash tool, so they exercise the real Docker-network egress model rather than the unfiltered runner.

Wiring

Triggers mirror the other copilot smoke tests: ready-for-aw label_command on PRs + schedule (every 12h) + workflow_dispatch, so it runs automatically in CI.
Registered in ready-for-aw-workflows.test.ts (gating) and smoke-copilot-workflow.test.ts (output-requirement) test lists.
Lock file compiled with the latest gh-aw pre-release (v0.82.12) and post-processed via scripts/ci/postprocess-smoke-workflows.ts, so CI dogfoods the repo own AWF source (--build-local).

Validation

gh aw compile — 0 errors
npx tsx scripts/ci/postprocess-smoke-workflows.ts — lock rewritten to local build
jest ready-for-aw + smoke-copilot + postprocess suites — all pass
npm run type-check, markdownlint — clean

Refs #6326

Adds a CI-integrated smoke test that runs the Copilot engine under the AWF network-isolation topology (sandbox.agent.sudo: false, isolation: true) and asserts the egress allowlist is enforced from inside the sandbox: - egress ALLOW: api.github.com is reachable - egress DENY: example.com is blocked - agent completion: add_comment safe-output validated by post-steps Wires the new workflow into the ready-for-aw label matrix and the copilot smoke output-requirement tests. Closes the network-isolation CI coverage gap tracked in #6326 (Gap 2). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b593fd45-255a-49a2-9d21-fd78108650a2

github-actions · 2026-07-17T17:16:35Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	99.01%	99.04%	📈 +0.03%
Statements	98.95%	98.98%	📈 +0.03%
Functions	99.35%	99.35%	➡️ +0.00%
Branches	95.17%	95.17%	➡️ +0.00%

📁 Per-file Coverage Changes (1 files)

File	Lines (Before → After)	Statements (Before → After)
`src/log-directory-setup.ts`	96.2% → 100.0% (+3.78%)	96.3% → 100.0% (+3.71%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot

Pull request overview

Adds a CI-integrated Copilot smoke test for AWF network-isolation egress enforcement.

Changes:

Adds allow/deny egress checks inside the sandbox.
Validates required safe-output usage.
Registers and compiles the new workflow.

Show a summary per file

File	Description
`.github/workflows/smoke-copilot-network-isolation.md`	Defines the smoke test.
`.github/workflows/smoke-copilot-network-isolation.lock.yml`	Compiled workflow using local AWF.
`scripts/ci/smoke-copilot-workflow.test.ts`	Registers output-requirement checks.
`scripts/ci/ready-for-aw-workflows.test.ts`	Registers activation-gating checks.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 4/4 changed files
Comments generated: 3
Review effort level: Medium

lpcox · 2026-07-17T18:34:29Z

+  - name: Validate safe outputs were invoked
+    run: |


Fixed in d8d1df9. The post-step now requires a machine-readable EGRESS_RESULT allow=<pass|fail> deny=<pass|fail> marker in the safe-output body and fails the job unless it's exactly allow=pass deny=pass. The prompt instructs the agent to emit the real observed outcome of both curl checks, so a broken allowlist (allowed host unreachable) or a leak (denied host reachable) now turns the workflow red instead of leaving it green.

lpcox · 2026-07-17T18:34:30Z

+  add-comment:
+    hide-older-comments: true
+  add-labels:
+    allowed: [smoke-copilot-network-isolation]


Created the smoke-copilot-network-isolation label in the repo (color ededed, matching the per-workflow label convention used by smoke-copilot-byok, smoke-copilot-pat, etc.), so add_labels no longer 422s.

+          fi
+          # shellcheck disable=SC1003,SC2016,SC2086
+          awf --config "${RUNNER_TEMP}/gh-aw/awf-config.json" --container-workdir "${GITHUB_WORKSPACE}" --mount "${RUNNER_TEMP}/gh-aw:${RUNNER_TEMP}/gh-aw:ro" --mount "${RUNNER_TEMP}/gh-aw:/host${RUNNER_TEMP}/gh-aw:ro" ${GH_AW_TOOL_CACHE_MOUNT:+--mount "$GH_AW_TOOL_CACHE_MOUNT"} ${GH_AW_DOCKER_HOST:+--docker-host "$GH_AW_DOCKER_HOST"} --env-all --exclude-env COPILOT_GITHUB_TOKEN --exclude-env GITHUB_MCP_SERVER_TOKEN --exclude-env MCP_GATEWAY_API_KEY --log-level info --build-local \
+            -- /bin/bash -c 'set +o histexpand; export PATH="${RUNNER_TEMP}/gh-aw/mcp-cli/bin:$PATH" && : "${RUNNER_TOOL_CACHE:?RUNNER_TOOL_CACHE must be set}"; GH_AW_TOOL_CACHE="$RUNNER_TOOL_CACHE"; export PATH="$(find "$GH_AW_TOOL_CACHE" -maxdepth 5 -type d -name bin 2>/dev/null | tr '\''\n'\'' '\'':'\'')$PATH"; [ -n "$GOROOT" ] && export PATH="$GOROOT/bin:$PATH" || true; [ -n "$ERLANG_HOME" ] && export PATH="$ERLANG_HOME/bin:$PATH" || true && GH_AW_NODE_EXEC="${GH_AW_NODE_BIN:-}"; if [ -z "$GH_AW_NODE_EXEC" ] || [ ! -x "$GH_AW_NODE_EXEC" ]; then GH_AW_NODE_EXEC="$(command -v node 2>/dev/null || true)"; fi; if [ -z "$GH_AW_NODE_EXEC" ]; then echo "node runtime missing on this runner — check runtimes.node in workflow YAML" >&2; exit 127; fi; GH_AW_NPM_GLOBAL_ROOT="$(npm root -g 2>/dev/null || true)"; if [ -n "$GH_AW_NPM_GLOBAL_ROOT" ]; then export NODE_PATH="${GH_AW_NPM_GLOBAL_ROOT}${NODE_PATH:+:${NODE_PATH}}"; fi; "$GH_AW_NODE_EXEC" ${RUNNER_TEMP}/gh-aw/actions/copilot_harness.cjs /usr/local/bin/copilot --add-dir /tmp/gh-aw/ --log-level all --log-dir /tmp/gh-aw/sandbox/agent/logs/ --disable-builtin-mcps --no-ask-user --allow-all-tools --allow-all-paths --add-dir "${GITHUB_WORKSPACE}" --prompt-file /tmp/gh-aw/aw-prompts/prompt.txt' 2>&1 | tee -a /tmp/gh-aw/agent-stdio.log


Explain what gVisor is (application kernel, Sentry, netstack, Systrap/KVM platforms) and how AWF runs the agent under the runsc OCI runtime: the compose executionModel, runtime field wiring, the netstack DNS workaround (extra_hosts + chroot /host/etc/hosts patching), iptables DNAT compat, and the Bun JIT shim. Includes guidance for adding other compose-model runtimes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 23717692-af7a-4e03-a156-5b696c3f01bd

Address PR review feedback on the workflow .md: - Post-step now parses a machine-readable EGRESS_RESULT marker from the safe-output body and fails the workflow unless both the allow and deny egress checks passed, instead of only asserting a safe-output tool was called. Prompt updated to require the agent to emit the marker. - The smoke-copilot-network-isolation label is created in the repo (matching the per-workflow label convention) so add_labels no longer 422s. Recompiled the lock with gh-aw v0.82.12 and re-ran the smoke postprocess. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e846bbd4-4c40-4e7b-869d-c3c4d7e6fa0f

github-actions · 2026-07-17T18:34:33Z

✅ Copilot review passed with no inline comments.

@lpcox Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

github-actions · 2026-07-17T18:34:43Z

Documentation Preview

Documentation build failed for this PR. View logs.

Built from commit b58ea42

…n-smoke # Conflicts: # docs/gvisor-integration.md

Copilot AI review requested due to automatic review settings July 17, 2026 17:14

lpcox mentioned this pull request Jul 17, 2026

ARC/DinD implementation gaps identified from community feedback (gh-aw#40107) #6326

Closed

Copilot started reviewing on behalf of lpcox July 17, 2026 17:14 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

lpcox and others added 2 commits July 17, 2026 11:33

Merge remote-tracking branch 'origin/main' into test/network-isolatio…

c724d15

…n-smoke # Conflicts: # docs/gvisor-integration.md

lpcox merged commit dc81a63 into main Jul 17, 2026
24 checks passed

lpcox deleted the test/network-isolation-smoke branch July 17, 2026 23:22

github-actions Bot mentioned this pull request Jul 19, 2026

🩺 Runner Doctor UpdateRunner Doctor: 4 new/updated failure-mode entries (2026-07-10 to 2026-07-19 scan) #6405

Closed

Copilot AI mentioned this pull request Jul 19, 2026

Update runner doctor catalog for ARC/DinD DNS, gVisor MCP 403s, and sbx home exposure #6409

Merged

Navigation Menu

Uh oh!

test: add copilot network-isolation egress smoke test - #6330

test: add copilot network-isolation egress smoke test#6330
lpcox merged 4 commits into
mainfrom
test/network-isolation-smoke

lpcox commented Jul 17, 2026

Uh oh!

github-actions Bot commented Jul 17, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

lpcox Jul 17, 2026

Uh oh!

lpcox Jul 17, 2026

Uh oh!

github-actions Bot commented Jul 17, 2026

Uh oh!

github-actions Bot commented Jul 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

lpcox commented Jul 17, 2026

Summary

Wiring

Validation

Uh oh!

github-actions Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Coverage Check Passed

Overall Coverage

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

lpcox Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

lpcox Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 17, 2026

Uh oh!

github-actions Bot commented Jul 17, 2026

Documentation Preview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jul 17, 2026 •

edited

Loading