Repository navigation
fix: purge orphaned VPCs after stack removal to prevent VPC limit errors - #130
Merged
Merged
Conversation
Synthetic Performance Results — Graviton (run)Commit: ✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01) Synthetic UDP Performance ResultsMeasures framework overhead: sync IPv4 Baseline
IPv6
IPv6 vs IPv4 Comparison (sync path)
IPv4 avg sync/async ratio: 0.9x, worst: 1.1x | IPv6 vs IPv4 worst ratio: 1.48x (OK)
|
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: DeployInfrastructure ready.
|
Integration Tests — ❌ 2 real failures
❌ Real failures (2) — code/test issues
details |
Integration Tests — ❌ 3 real failures
❌ Real failures (3) — code/test issues
detailsdetails |
|
Integration Test Failure — Graviton (Run 39541970880)Branch: receiver-user-data.log (7309 bytes, last 80 lines)sender-user-data.log (7305 bytes, last 80 lines)receiver-console-output.log (45222 bytes, last 80 lines)sender-console-output.log (47970 bytes, last 80 lines) |
❌ Integration Tests Failed — Graviton (run)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log |
TCP Synthetic Performance Results (run)Commit: TCP Synthetic Performance ResultsMeasures TCP engine framework overhead using a mock PacketBackend.
conn_est=1976ns, throughput=4487.5Mbps, tick=734ns |
Synthetic Performance Results (run)Commit: ✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01) Synthetic UDP Performance ResultsMeasures framework overhead: sync IPv4 Baseline
IPv6
IPv6 vs IPv4 Comparison (sync path)
IPv4 avg sync/async ratio: 1.0x, worst: 1.1x | IPv6 vs IPv4 worst ratio: 1.25x (OK)
|
[CI] Stage: DeployInfrastructure ready.
|
Integration Tests — ❌ 3 real failures
❌ Real failures (3) — code/test issues
detailsdetails |
Integration Test Failure (Run 39541971040)Branch: No failure-summary.json found receiver-user-data.log (6943 bytes, last 80 lines)sender-user-data.log (6906 bytes, last 80 lines)receiver-console-output.log (42570 bytes, last 80 lines)sender-console-output.log (43511 bytes, last 80 lines)All instance-logs files |
❌ Integration Tests Failed (Run 39541971040)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log receiver-test-client-iperf.log sender-test-client-iperf.log Full Application Logs (last 200 lines each)receiver-echo-server.logsender-echo-server.logsender-test-client.logreceiver-test-client-iperf.logsender-test-client-iperf.log
|
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: DeployInfrastructure ready.
|
|
Integration Tests — ❌ 3 real failures
❌ Real failures (3) — code/test issues
detailsdetails |
Integration Test Failure — Graviton (Run 39541970880)Branch: receiver-user-data.log (7298 bytes, last 80 lines)sender-user-data.log (7298 bytes, last 80 lines)receiver-console-output.log (44895 bytes, last 80 lines)sender-console-output.log (47889 bytes, last 80 lines) |
❌ Integration Tests Failed — Graviton (run)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log |
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: DeployInfrastructure ready.
|
Integration Tests — ❌ 3 real failures
❌ Real failures (3) — code/test issues
detailsdetails |
Integration Tests — ❌ 3 real failures
❌ Real failures (3) — code/test issues
detailsdetails |
Integration Test Failure (Run 39541971040)Branch: No failure-summary.json found receiver-user-data.log (6927 bytes, last 80 lines)sender-user-data.log (6925 bytes, last 80 lines)receiver-console-output.log (42789 bytes, last 80 lines)sender-console-output.log (43277 bytes, last 80 lines)All instance-logs files |
❌ Integration Tests Failed (Run 39541971040)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log receiver-test-client-iperf.log sender-test-client-iperf.log Full Application Logs (last 200 lines each)receiver-echo-server.logsender-echo-server.logsender-test-client.logreceiver-test-client-iperf.logsender-test-client-iperf.log
|
Integration Test Failure — Graviton (Run 39541970880)Branch: receiver-user-data.log (7300 bytes, last 80 lines)sender-user-data.log (7305 bytes, last 80 lines)receiver-console-output.log (44967 bytes, last 80 lines)sender-console-output.log (47976 bytes, last 80 lines) |
❌ Integration Tests Failed — Graviton (run)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log |
[CI] Stage: DeployInfrastructure ready.
|
|
|
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: DeployInfrastructure ready.
|
Integration Tests — ❌ 2 real failures
❌ Real failures (2) — code/test issues
details |
Integration Tests — ❌ 3 real failures
❌ Real failures (3) — code/test issues
detailsdetails |
Integration Test Failure — Graviton (Run 39606189512)Branch: receiver-user-data.log (7291 bytes, last 80 lines)sender-user-data.log (7317 bytes, last 80 lines)receiver-console-output.log (44985 bytes, last 80 lines)sender-console-output.log (47876 bytes, last 80 lines) |
❌ Integration Tests Failed — Graviton (run)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log |
|
…llision The quic-integration-tests.yml workflow was sharing DpdkTestStack with integration-tests.yml. When both workflows triggered on the same PR event, validate-cdk (fast, no AWS) completed quickly and released quic-integration-tests to start. By that point integration-tests.yml had already deployed DpdkTestStack (CREATE_COMPLETE). The QUIC pre-deploy cleanup issued a delete-stack, but the x86 job's own teardown was also running concurrently, so the stack entered DELETE_IN_PROGRESS from two deleters simultaneously. The QUIC waiter timed out after 17 minutes, failing the workflow. Fix: add DpdkTestStackQuic construct to app.ts (same DpdkTestStack class, different logical ID / CFN stack name) and update quic-integration-tests.yml to: - Use a dedicated concurrency group (quic-integration-tests) instead of sharing integration-tests, since the two workflows no longer touch the same stack - Clean up DpdkTestStackQuic in pre-deploy and safety-net teardown steps - Pass CDK_STACK_NAME=DpdkTestStackQuic to run-integration-tests.sh
The safety-net teardown was gated on failure(), which is false when a job is cancelled or hits timeout-minutes. The TCP job has been hitting its 55-min job timeout (Oct 21 on this PR, Oct 22 on #129) mid log-collection, before run-integration-tests.sh reaches --teardown, leaving DpdkTestStack (2x c5n.large + NAT) running until the next run's pre-deploy cleanup. For all four EC2 jobs (x86, x86 TCP, Graviton, QUIC): - Safety net teardown: if: always(). cleanup-cfn-stack.sh is a no-op plus orphaned-VPC purge when nothing exists, and each workflow's concurrency group (cancel-in-progress: false) means no other run shares its stack. - Re-assume the AWS role right before it, since the first OIDC session lasts 1h and a long run can outlive it. - Step timeout on the test step (45m; 55m for TCP) under a raised job timeout (65m; 75m for TCP), so ~15m always remains for teardown. Healthy test steps take 31-36m. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gspivey
force-pushed
the
fix/purge-orphaned-vpcs-after-force-delete
branch
2 times, most recently
from
October 22, 2026 13:49
098d9ce to
ebb01fa
Compare
TCP Synthetic Performance Results (run)Commit: TCP Synthetic Performance ResultsMeasures TCP engine framework overhead using a mock PacketBackend.
conn_est=1285ns, throughput=5978.8Mbps, tick=494ns |
Synthetic Performance Results (run)Commit: ✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01) Synthetic UDP Performance ResultsMeasures framework overhead: sync IPv4 Baseline
IPv6
IPv6 vs IPv4 Comparison (sync path)
IPv4 avg sync/async ratio: 1.0x, worst: 1.1x | IPv6 vs IPv4 worst ratio: 1.26x (OK)
|
Synthetic Performance Results — Graviton (run)Commit: ✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01) Synthetic UDP Performance ResultsMeasures framework overhead: sync IPv4 Baseline
IPv6
IPv6 vs IPv4 Comparison (sync path)
IPv4 avg sync/async ratio: 0.9x, worst: 1.0x | IPv6 vs IPv4 worst ratio: 1.42x (OK)
|
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: DeployInfrastructure ready.
|
Integration Tests — ❌ 2 real failures
❌ Real failures (2) — code/test issues
details |
Integration Tests — ✅ all passed
✅ All 17 tests passed |
|
✅ Integration Tests Passed — Graviton (run)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log |
Integration Tests — ✅ all passed
✅ All 17 tests passed |
✅ Integration Tests Passed (Run 39790246329)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log receiver-test-client-iperf.log sender-test-client-iperf.log Full Application Logs (last 200 lines each)receiver-echo-server.logsender-echo-server.logsender-test-client.logreceiver-test-client-iperf.logsender-test-client-iperf.log
|
[CI] Stage: DeployInfrastructure ready.
|
|
Integration Tests —
|
| Tier | Test | Result | Time | Reason |
|---|---|---|---|---|
| tier1-tcp-echo-full | execution | 0.000s | ENI bind failed on receiver instance | |
| tier1-tcp-handshake | single_handshake | ✅ pass | 0.927s | |
| tier1-tcp-handshake | repeated_handshake | ✅ pass | 0.719s | |
| tier1-tcp-handshake | handshake_rate | ✅ pass | 0.740s | |
| tier1-tcp-shutdown | execution | 0.000s | ENI bind failed on receiver instance | |
| tier2-tcp-flow-control | execution | 0.000s | ENI bind failed on receiver instance | |
| tier2-tcp-retransmit | execution | 0.000s | ENI bind failed on receiver instance | |
| tier3-tcp-kernel-interop | execution | 0.000s | ENI bind failed on receiver instance | |
| tier3-tcp-std-parity | execution | 0.000s | ENI bind failed on receiver instance |
⚠️ Infra flakes (6) — SSM/ENI setup, not code (task #10)
- tier1-tcp-echo-full / execution — ENI bind failed on receiver instance
- tier1-tcp-shutdown / execution — ENI bind failed on receiver instance
- tier2-tcp-flow-control / execution — ENI bind failed on receiver instance
- tier2-tcp-retransmit / execution — ENI bind failed on receiver instance
- tier3-tcp-kernel-interop / execution — ENI bind failed on receiver instance
- tier3-tcp-std-parity / execution — ENI bind failed on receiver instance
gspivey
pushed a commit
that referenced
this pull request
Oct 22, 2026
PR #131's x86 and Graviton runs both skipped every Phase B and C test: the sender's bind command stayed InProgress past the 120s poll, so configure_eni reported "bind failed", although dmesg shows vfio-pci took the device on time. The SSM agent goes silent for minutes right after a driver change while it reconnects. #130's passing run hit the same 120s silence on three configure-eni calls and survived only because none was the fatal Phase B bind (which took 97s there). configure_eni now waits up to 180s, then asks configure-eni.sh --action status up to four times (60s each) while the agent recovers, and accepts the step when the driver is already the wanted one (re-running the idempotent assign-ip after an unbind). An agent that answers with the wrong state still fails the step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.