Repository navigation
fix: use IMDSv2 tokens and explicit IP assignment after ENI unbind - #16
Merged
Merged
Conversation
Three root causes fixed:
1. Tier 2 ENI bind failure ("Device or resource busy"):
- DPDK processes from Tier 1 hold vfio-pci file descriptors open
- configure-eni.sh now kills DPDK processes before bind/unbind
- Added retry logic for transient bind failures
- Check driver symlink exists before attempting unbind
2. Tier 3 iperf3 incompatibility (60s timeout both directions):
- iperf3 requires TCP control channel which DPDK doesn't support
- Replaced iperf3 with Python UDP echo server (Direction 1) and
kernel test-client (Direction 2) for true cross-stack testing
- Direction 1: DPDK sends → kernel receives (with --bind-ip)
- Direction 2: kernel sends → DPDK receives (tokio fallback)
3. Missing ARP warming between tiers:
- Extracted ARP cache warming into reusable warm_arp_cache()
- Called before every tier (was only in Tier 1)
- Also pings subnet gateway to ensure gateway MAC is cached
- Added cleanup_dpdk_state() for robust inter-tier cleanup
https://claude.ai/code/session_01NM1kfsCAYe1BCzufhG9SZx
After rebinding the secondary ENI to the ena kernel driver, the interface needs its IP configured via DHCP. Without this, the kernel interface exists but has no IP, causing Tier 3 Direction 1 (kernel echo server) to not receive packets on the DPDK ENI IP. https://claude.ai/code/session_01NM1kfsCAYe1BCzufhG9SZx
The tier3-our-app-sends test fails because the receiver's kernel interface doesn't have its IP configured after unbinding from vfio-pci. Root causes: 1. IMDS calls used IMDSv1 which is blocked on AL2023 (IMDSv2 required) 2. NetworkManager/dhclient may not auto-configure the interface after a vfio-pci round-trip Fixes: - configure-eni.sh: Use IMDSv2 session tokens for IMDS metadata queries - run-integration-tests.sh: configure_eni() now accepts an optional expected_ip parameter; after unbinding, it directly assigns the IP to the kernel interface as a safety net - All configure_eni unbind calls now pass the known ENI IP - Added pre-flight diagnostics to tier3 test for better debugging https://claude.ai/code/session_01NM1kfsCAYe1BCzufhG9SZx
[CI] Stage: DeployInfrastructure ready.
|
- CI workflow now always posts instance logs and test summaries as PR comments (not just on failure), using resolved PR number that works for both pull_request and workflow_dispatch events - Tier 3 ENI bind failure now generates failure XML (not skip) and captures detailed diagnostics (driver state, vfio modules, dmesg) - configure-eni.sh bind path logs driver state at each step and captures vfio-pci bind errors for diagnosis https://claude.ai/code/session_01NM1kfsCAYe1BCzufhG9SZx
[CI] Stage: SummaryAll tests PASSED. ARP seeding: kernel /proc/net/arp (automatic)
|
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: SummarySome tests FAILED (exit code: 1). ARP seeding: kernel /proc/net/arp (automatic)
|
Integration Test Failure (Run 22666885814)Branch: No failure-summary.json found receiver-user-data.log (10248 bytes, last 80 lines)sender-user-data.log (10248 bytes, last 80 lines)receiver-console-output.log (32112 bytes, last 80 lines)sender-console-output.log (31985 bytes, last 80 lines)All instance-logs files |
❌ Integration Tests Failed (Run 22666885814)Branch: Test Results
Instance Logsreceiver-user-data.log (10248 bytes, last 50 lines)sender-user-data.log (10248 bytes, last 50 lines)receiver-console-output.log (32112 bytes, last 50 lines)sender-console-output.log (31985 bytes, last 50 lines) |
Root cause: do_unbind() attempted IP configuration (NetworkManager, dhclient, IMDS) which took up to 34 seconds, exceeding the 30-second SSM command timeout. This caused spurious "ENI bind failed" errors that cascaded into Tier 2 and Tier 3 test failures. Changes: - Remove all IP configuration from do_unbind() — the orchestrator handles IP assignment via a separate SSM command with the known IP - Increase ENI_BIND_TIMEOUT from 30 to 45 seconds for safety margin - Rewrite PR log posting to include application logs, crash diagnostics, network/PCI state, and filtered kernel dmesg instead of duplicated user-data/console-output logs https://claude.ai/code/session_01NM1kfsCAYe1BCzufhG9SZx
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: SummarySome tests FAILED (exit code: 1). ARP seeding: kernel /proc/net/arp (automatic)
|
Integration Test Failure (Run 22668412859)Branch: No failure-summary.json found receiver-user-data.log (10248 bytes, last 80 lines)sender-user-data.log (10321 bytes, last 80 lines)receiver-console-output.log (32199 bytes, last 80 lines)sender-console-output.log (32019 bytes, last 80 lines)All instance-logs files |
❌ Integration Tests Failed (Run 22668412859)Branch: Test Results
Application Logsreceiver-echo-server.logsender-echo-server.logsender-test-client.logreceiver-test-client-iperf.logsender-test-client-iperf.logreceiver-iperf3-server.logNetwork & PCI Statereceiver-network-interfaces.logsender-network-interfaces.logreceiver-networking-diag-baseline.txtsender-networking-diag-baseline.txt
|
Root cause: After unbinding from vfio-pci, NetworkManager detects the new interface and reconfigures it, removing the manually-assigned IP. The receiver's DPDK ENI had no IPv4 address, so Tier 3 packets never reached the kernel echo server. Changes: - Tell NM to not manage the interface before assigning IP - Wait for net interface to appear after ena bind (async) - Flush stale addresses before assigning new IP - Fail Tier 3 early if receiver ENI IP can't be assigned - Replace generate_skip_xml with generate_failure_xml for tier3 https://claude.ai/code/session_01NM1kfsCAYe1BCzufhG9SZx
❌ Integration Tests Failed (Run 22668418969)Branch: Test ResultsApplication LogsNetwork & PCI StateKernel Console (dmesg) |
Integration Test Failure (Run 22669292275)Branch: failure-summary.json{
"failed_step": "deploy_infrastructure",
"error": "Infrastructure deployment failed",
"exit_code": 2,
"timestamp": "2026-03-04T12:31:29.575915Z",
"sender_instance_id": "",
"receiver_instance_id": "",
"commit": "6ba3a487f905db796592662ffea7f019c0ec5986",
"run_url": "https://github.com/gspivey/dpdk-stdlib-rust/actions/runs/22669292275"
}```
### instance-0-user-data.log (2847 bytes, last 80 lines)=== User-data starting at Wed Mar 4 12:26:25 UTC 2026 === Compiling cexpr v0.6.0 warning: warning: variable does not need to be mutable warning: [ 7.678337] cloud-init[1643]: The key's randomart image is: Amazon Linux 2023.10.20260216 ip-10-0-1-71 login: [ 29.317147] cloud-init[1748]: Amazon Linux 2023 repository 2.7 MB/s | 54 MB 00:20 [ 133.343155] cloud-init[1764]: Compiling smallvec v1.15.1 total 200 |
❌ Integration Tests Failed (Run 22669292275)Branch: Test ResultsApplication LogsNetwork & PCI Stateinstance-0-network-interfaces.loginstance-1-network-interfaces.log
|
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: SummarySome tests FAILED (exit code: 1). ARP seeding: kernel /proc/net/arp (automatic)
|
Integration Test Failure (Run 22669622827)Branch: No failure-summary.json found receiver-user-data.log (10321 bytes, last 80 lines)sender-user-data.log (10321 bytes, last 80 lines)receiver-console-output.log (32235 bytes, last 80 lines)sender-console-output.log (31578 bytes, last 80 lines)All instance-logs files |
❌ Integration Tests Failed (Run 22669622827)Branch: Test Results
Application Logsreceiver-echo-server.logsender-test-client.logNetwork & PCI Statereceiver-network-interfaces.logsender-network-interfaces.logreceiver-networking-diag-baseline.txtsender-networking-diag-baseline.txt
|
The IP assignment SSM command was failing due to: 1. set -euo pipefail causing early exit on benign failures 2. ip addr add failing with "File exists" on repeat calls 3. No verification that the IP was actually assigned Changes: - Use quoted heredoc to prevent variable expansion issues - Use ip addr replace instead of ip addr add (idempotent) - Use ip route replace instead of ip route add (idempotent) - Remove set -euo pipefail, handle errors explicitly - Add verification step that confirms IP is configured - Add detailed logging for debugging https://claude.ai/code/session_01NM1kfsCAYe1BCzufhG9SZx
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: SummarySome tests FAILED (exit code: 1). ARP seeding: kernel /proc/net/arp (automatic)
|
Integration Test Failure (Run 22670435921)Branch: No failure-summary.json found receiver-user-data.log (10248 bytes, last 80 lines)sender-user-data.log (10248 bytes, last 80 lines)receiver-console-output.log (32072 bytes, last 80 lines)sender-console-output.log (31375 bytes, last 80 lines)All instance-logs files |
❌ Integration Tests Failed (Run 22670435921)Branch: Test Results
Application Logsreceiver-echo-server.logsender-test-client.logNetwork & PCI Statereceiver-network-interfaces.logsender-network-interfaces.logreceiver-networking-diag-baseline.txtsender-networking-diag-baseline.txt
|
The IP assignment was failing because ssm_run_command embeds the command string in JSON: commands=["..."], and multi-line scripts with quotes break the JSON parsing. Fix: add 'assign-ip' action to configure-eni.sh so the SSM command is a simple single-line: bash configure-eni.sh --action assign-ip --ip X.X.X.X The assign-ip action: - Disables NetworkManager on the interface - Uses ip addr replace for idempotency - Verifies the IP is actually configured https://claude.ai/code/session_01NM1kfsCAYe1BCzufhG9SZx
- Increase ENI_BIND_TIMEOUT from 45s to 90s (bind can take 20s+ for driver transitions plus module loading) - Increase ssm_wait_command timeouts from 75s to 120s for bind/unbind - Add ssm_save_failure_log() shared helper — captures stdout/stderr from failed SSM commands for PR comment visibility - ssm_wait_command now saves failure diagnostics (was silent before) - Deduplicate SSM failure logging in ssm_run_command https://claude.ai/code/session_01NM1kfsCAYe1BCzufhG9SZx
[CI] Stage: DeployInfrastructure ready.
|
Integration Test Failure (Run 22700258483)Branch: No failure-summary.json found All instance-logs files |
❌ Integration Tests Failed (Run 22700258483)Branch: Test Results
Application LogsNetwork & PCI Statereceiver-networking-diag-baseline.txtsender-networking-diag-baseline.txtKernel Console (dmesg) |
[CI] Stage: DeployInfrastructure ready.
|
Integration Test Failure (Run 22700893046)Branch: No failure-summary.json found All instance-logs files |
❌ Integration Tests Failed (Run 22700893046)Branch: Test Results
Application Logs
|
Root cause analysis from run 22700893046: SSM send-command API calls were being throttled (empty command ID returned), causing cascading failures. The SSM agent wasn't saturated — the API itself was rejecting commands sent too rapidly. Changes: - unbind_all_enis: switch from parallel to sequential with 3s pause between instances and 15s cooldown after - Tier 2/3 ENI setup: use sequential configure_eni() calls instead of parallel async, with 3s pause between instances - warm_arp_cache: sequential with 2s pause - ssm_run_command_async: add retry with exponential backoff (3 attempts) - ssm_save_failure_log: handle empty command IDs gracefully https://claude.ai/code/session_01NM1kfsCAYe1BCzufhG9SZx
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: SummaryAll tests PASSED. ARP seeding: kernel /proc/net/arp (automatic) |
✅ Integration Tests Passed (Run 22701987602)Branch: Test ResultsApplication Logs
|
Critical bug: ssm_run_command_async retry log messages were written to stdout, polluting the command ID return value. This caused downstream ssm_wait_command to fail with garbage command IDs. Also fixes: - collect_results: remove duplicate ssm_run_command call, use 2 SSM commands per instance (list + cat with delimiters) instead of 1 batched command that hit SSM's 24KB output limit - collect_instance_logs: replace tar+base64 approach with 2 batched SSM commands per instance (app logs + system logs), staying under SSM output size limits - ssm_save_failure_log: handle empty command IDs https://claude.ai/code/session_01NM1kfsCAYe1BCzufhG9SZx
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: SummarySome tests FAILED (exit code: 1). ARP seeding: kernel /proc/net/arp (automatic)
|
Integration Test Failure (Run 22702940474)Branch: No failure-summary.json found receiver-user-data.log (8171 bytes, last 80 lines)sender-user-data.log (8171 bytes, last 80 lines)receiver-console-output.log (32174 bytes, last 80 lines)sender-console-output.log (31896 bytes, last 80 lines)All instance-logs files |
❌ Integration Tests Failed (Run 22702940474)Branch: Test Results
Application Logsreceiver-echo-server.logsender-echo-server.logsender-test-client.logreceiver-test-client-iperf.logsender-test-client-iperf.log
|
Two issues caused the Tier 3 our-app-sends test to fail: 1. Port 0 literal: DPDK sockets bound to port 0 used port 0 literally in UDP headers (unlike kernel sockets which assign ephemeral ports). The Python echo server couldn't respond to port 0. Added ephemeral port allocator using Linux range (32768-60999) for both bind() and bind_with_backend(). 2. Async recv deadlock: spawn_blocking(recv_from) held the socket lock indefinitely. When tokio::time::timeout fired, the blocking thread kept running with the lock, deadlocking all subsequent send/recv. Fixed by setting 1-second read_timeout in spawn_blocking, looping in the async wrapper with yield_now between iterations. Also added set_read_timeout/set_write_timeout to match std::net::UdpSocket API and docs/ephemeral-ports.md documenting both issues. https://claude.ai/code/session_01NM1kfsCAYe1BCzufhG9SZx
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: SummaryAll tests PASSED. ARP seeding: kernel /proc/net/arp (automatic)
|
✅ Integration Tests Passed (Run 22714907951)Branch: Test Results
Application Logsreceiver-echo-server.logsender-echo-server.logsender-test-client.logreceiver-test-client-iperf.logsender-test-client-iperf.log
|
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: SummaryAll tests PASSED. ARP seeding: kernel /proc/net/arp (automatic)
|
✅ Integration Tests Passed (Run 22714985882)Branch: Test Results
Application Logsreceiver-echo-server.logsender-echo-server.logsender-test-client.logreceiver-test-client-iperf.logsender-test-client-iperf.log
|
gspivey
pushed a commit
that referenced
this pull request
Apr 13, 2026
gspivey
pushed a commit
that referenced
this pull request
Jun 13, 2026
gspivey
added a commit
that referenced
this pull request
Jun 13, 2026
## Roadmap Item Implements **#16**: `dpdk-stdlib-tcp: Contract types, TcpState, Clock, and IsnGenerator` Spec tasks: 5.1, 5.2, 5.3, 5.4 ## Changes Four new source modules in `dpdk-stdlib-tcp/src/`: - **state.rs** — `TcpState` enum (11 states per RFC 9293) + `FourTuple` (Hash, Eq, serializable to bytes) - **clock.rs** — `Clock` trait + `SystemClock` + `MockClock` (with `advance()`/`set()` for deterministic testing) - **isn.rs** — `IsnGenerator` per RFC 6528 (128-bit per-boot secret via getrandom, SipHash-2-4 of FourTuple, M = elapsed µs / 4) - **contract.rs** — `ConnectionHandle` (SPSC rings, atomic state, condvar wake, AtomicWaker), `EngineCommand` enum, `SocketOption` enum, `CommandSender` (wraps mpsc + EngineWakeup signal), `OneshotSender`/`OneshotReceiver`, `EngineWakeup`, `AtomicWaker` Updated `lib.rs` with module declarations and `Cargo.toml` with `getrandom` + `siphasher` dependencies. ## Tests - 23 new unit tests covering all new types - All 65 TCP tests pass (55 unit + 10 property) - Full workspace build and test pass (800+ tests) ## Tradeoffs - `AtomicWaker` uses `Mutex<Option<Waker>>` for simplicity (no unsafe, no external dep). Sufficient for MVP; can be replaced with lock-free implementation if contention is measured. - `EngineWakeup` uses AtomicBool + Condvar (portable). Linux eventfd optimization deferred to engine loop implementation. - `OneshotSender`/`OneshotReceiver` avoid tokio dependency per spec (CI enforces no tokio dep in dpdk-stdlib-tcp). --------- Co-authored-by: Agent Router <agent@agent-router.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.