Repository navigation
Fix AMI build: harden AMI ID extraction and DPDK verification - #9
Merged
Merged
Conversation
The AMI build workflow succeeded at building the AMI but failed at the SSM PutParameter step (exit code 254). Two root causes: 1. AMI ID extraction used `cut -d: -f2` which is fragile with Packer's machine-readable format. Switched to awk-based parsing and added validation that the extracted ID matches the ami-* pattern. 2. SSM step passed empty value when extraction failed. Added guard to fail early with a clear error message. Additional hardening: - Packer verification provisioner now fails the build if fewer than 10 DPDK libraries are found or pkg-config can't find libdpdk - DPDK install script tolerates missing kernel modules (uio, vfio-pci) and hugetlbfs mount failures during AMI build context - Install script verifies DPDK libraries exist before declaring success https://claude.ai/code/session_01FpAkJ8kPt1CchVZ9kf1r7C
The second CI run showed Packer itself is failing (step 7), not just SSM. The 3m16s runtime is too short for a real DPDK build, suggesting Packer fails at instance launch or SSH connection. Fixes: - Disable errexit around packer call with set +e / PIPESTATUS so we can capture the exit code AND dump the last 80 lines of Packer output for diagnosis (previously set -e killed the script before any diagnostics could run) - Add associate_public_ip_address=true to Packer template to ensure SSH connectivity when default VPC doesn't auto-assign public IPs - Add ssh_timeout=10m to give Packer more time for SSH on slower instance types - Increase workflow timeout to 45m since DPDK compilation from source takes 15-20 minutes https://claude.ai/code/session_01FpAkJ8kPt1CchVZ9kf1r7C
…ostics Three fixes for the Packer build that failed after ~3 min of provisioning: 1. Remove `pip3 install --user pyelftools` from install script — redundant (python3-pyelftools already installed via dnf) and fails on AL2023 with PEP 668 externally-managed-environment protection. 2. Disable DPDK kernel module build (-Denable_kmods=false) — igb_uio frequently fails to compile against newer AL2023 kernels and vfio-pci (shipped with the kernel) is the preferred driver anyway. 3. Make aws-cfn-bootstrap install non-fatal in Packer provisioner — the package may not always be available and isn't required for DPDK tests. Also greatly improved error diagnostics in the workflow: parse Packer's machine-readable output to extract human-readable error/UI messages, and dump the last 100 lines of raw output on failure. https://claude.ai/code/session_01FpAkJ8kPt1CchVZ9kf1r7C
…ifact upload Root cause: `python3-pyelftools` RPM is not available in Amazon Linux 2023 default repositories, causing `dnf install` to fail with "No match for argument". With `set -euo pipefail`, this aborts the entire DPDK install script. Fix: Remove python3-pyelftools from the dnf install and install pyelftools via pip3 instead, with --break-system-packages fallback for PEP 668. Also adds `actions/upload-artifact` step to upload packer-output.log on failure, so we can always retrieve the full build log for debugging. https://claude.ai/code/session_01FpAkJ8kPt1CchVZ9kf1r7C
…stics DPDK 22.11.6 has known compilation failures on AL2023 with the GVE (Google Cloud) and ionic (AMD Pensando) drivers due to kernel header conflicts (__le64 type issues). These drivers are not needed on AWS where only the ENA driver is required. Add -Ddisable_drivers=net/gve,net/ionic to the meson setup command. Also write error details to GITHUB_STEP_SUMMARY so failure diagnostics are visible on the Actions run summary page without needing to download logs. https://claude.ai/code/session_01FpAkJ8kPt1CchVZ9kf1r7C
AL2023 (Fedora-based) has a real /usr/lib64 directory, so meson auto-detects lib64 as the default libdir. This causes DPDK libraries to install to /usr/local/lib64/ while all verification and downstream paths expect /usr/local/lib/. The build succeeds but verification fails with "No DPDK libraries found." Add --libdir=lib to the meson setup command to force consistent install to /usr/local/lib/. Also add a dedicated "Show build failure diagnostics" step that runs on failure and writes error details to both step output (expandable in the GitHub UI) and GITHUB_STEP_SUMMARY (visible on the run summary page). https://claude.ai/code/session_01FpAkJ8kPt1CchVZ9kf1r7C
gspivey
pushed a commit
that referenced
this pull request
Apr 11, 2026
Captures the perf-test run that validated the tokio-dpdk fix from Run #8 (100% drops -> real DPDK linkage) and exposes a new finding: the current dpdk-tokio compat shim caps single-socket throughput at ~40K pps because each recv_from/send_to round-trips through tokio::task::spawn_blocking + a Mutex acquisition. App Drops is 0 across the board, confirming the bottleneck is the polling rate (NIC RX ring drops) rather than the dpdk-udp socket buffer. rust-dpdk continues to track native-dpdk within 0.25% at all packet sizes and rates; the App Drops column proves the residual ~3% loss at 64B/700K is upstream of our stack (AWS network), not in the Rust user-space stack.
gspivey
pushed a commit
that referenced
this pull request
Apr 11, 2026
Run #9's "App Drops" column only captured software-layer drops (rx_drops_ring_full + rx_drops_buffer_full from dpdk-udp). The writeup overclaimed that a zero in that column meant "loss is on the wire, not in the Rust stack" — which is wrong. Packets dropped at the DPDK NIC descriptor ring (rte_eth_stats.imissed), at rx_nombuf refill, or at ierrors never reach the socket layer, so they can't appear in App Drops no matter how bad the hardware-level drop rate is. This commit does two things: 1. Plumbs real NIC counters into the perf reporter. - Adds NicStatsSnapshot + NicStatsFn to dpdk-udp::perf. - PerfReporter::start takes an optional NIC stats callback that it invokes once per reporting tick and reports as deltas. - UdpSocket::enable_perf_reporting builds the callback from SocketBackend::Dpdk(res) -> res.port.stats(). Non-DPDK backends pass None and the reporter emits "nic_imissed=-" so the harness can distinguish "no data" from "zero drops". - Adds rx_nombuf to PortStats (was missing from the wrapper). - [PERF] log line now carries nic_imissed / nic_ierrors / nic_rx_nombuf fields. - scripts/run-perf-tests.sh parses the new fields, attributes them to rate steps the same way as App Drops, and renders a new "NIC Drops" column in the markdown comparison table. 2. Rewrites the Run #9 analysis to: - Define exactly what App Drops measures (software-layer only). - Stop claiming a zero App Drops proves the loss is on the wire. At 64B/700K rust-dpdk, the 3.17% RX/TX gap could be NIC ring overflow, mempool exhaustion, or wire loss — App Drops alone can't tell us which. - Call out the instrumentation gap explicitly and note that Run #10 will be the first run with NIC Drops populated. New test perf_reporter_invokes_nic_stats_fn verifies the callback is called at baseline + at least once per tick. https://claude.ai/code/session_01AEEjCeNiuSA8MWU7BPy15G
gspivey
pushed a commit
that referenced
this pull request
Apr 13, 2026
Records hardware benchmark results from the spawn_blocking elimination. tokio-dpdk improved from ~40K pps (Run #9) to 343K pps at 700K/64B (8.6x). Matches sync DPDK at ≤140K pps; remaining gap at >350K pps is from Tokio yield_now() scheduling latency, not framework overhead. https://claude.ai/code/session_01EpC1AocdV7TkvTiMXmtCkm
gspivey
added a commit
that referenced
this pull request
Jun 9, 2026
## Roadmap Item Implements roadmap item #9: `dpdk-stdlib-quic`: Provider and builder. Spec: `.kiro/specs/s2n-quic-provider/` · tasks `10.1`, `10.2`, `10.3` ## Changes - **`with_backend_config()`** added to `ProviderBuilder` for backend selection (DPDK, AF_PACKET, or Auto) - **`io::Provider` implementation** for `DpdkProvider`: - `start(self, endpoint)` initializes the packet backend via `create_backend` - Resolves gateway MAC (explicit `with_gateway_mac` or kernel ARP cache fallback via `/proc/net/route` + `/proc/net/arp`) - Spawns `event_loop_with_tx_queue` on a dedicated `dpdk-quic-io` thread - Returns bound `SocketAddress` - **`SharedThread`** (`Arc<Mutex<Option<JoinHandle>>>`) enables `ProviderHandle::shutdown()` to join the event loop thread spawned by `start()` - **IPv6 rejection**: bind addresses with `IpAddr::V6` return `UnsupportedAddressFamily` - **`BackendConfig` re-export** from `lib.rs` for downstream consumers ## Tests Added | Test | Validates | |------|-----------| | `provider_start_stub_mode` | start() initializes and returns SocketAddress in stub mode | | `provider_ipv6_rejected` | IPv6 bind address returns UnsupportedAddressFamily | | `provider_handle_shutdown_joins_thread` | shutdown() sets flag, joins thread, stops counter increments | | `builder_with_all_options` | All builder methods including with_backend_config | | `resolve_gateway_mac_explicit` | Explicit MAC takes priority | | `resolve_gateway_mac_fallback_no_panic` | Kernel ARP fallback doesn't panic | ## Tradeoffs - Thread `JoinHandle` is shared via `Arc<Mutex<Option<JoinHandle>>>` between provider and handle — necessary because `start()` consumes `self` after `build()` returns the handle separately. - Gateway MAC resolution falls back to all-zeros in environments without `/proc` (stub/test/macOS) — expected for non-Linux testing. --------- Co-authored-by: Agent Router <agent@agent-router.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The AMI build workflow succeeded at building the AMI but failed at the
SSM PutParameter step (exit code 254). Two root causes:
AMI ID extraction used
cut -d: -f2which is fragile with Packer'smachine-readable format. Switched to awk-based parsing and added
validation that the extracted ID matches the ami-* pattern.
SSM step passed empty value when extraction failed. Added guard to
fail early with a clear error message.
Additional hardening:
DPDK libraries are found or pkg-config can't find libdpdk
and hugetlbfs mount failures during AMI build context
https://claude.ai/code/session_01FpAkJ8kPt1CchVZ9kf1r7C