Repository navigation
Conversation
gspivey
changed the title
The gh token can access API endpoints but not blob storage (logs and artifacts return Forbidden). Post the failure report as a comment on PR #11 so the failure details are readable via `gh pr view` or the API without needing blob storage access. https://claude.ai/code/session_01CMHbfazEcMsFVWph2bheWa
Integration Test Failure (Run 22264026597)Branch: failure-summary.json{
"failed_step": "deploy_infrastructure",
"error": "Infrastructure deployment failed",
"exit_code": 2,
"timestamp": "2026-02-21T20:52:59.372046Z",
"sender_instance_id": "",
"receiver_instance_id": "",
"commit": "874eddb2c5827889337ab41adb4bdd32b4e65507",
"run_url": "https://github.com/gspivey/dpdk-stdlib-rust/actions/runs/22264026597"
}```
### instance-0-user-data.log (12453 bytes, last 80 lines)Downloaded bytes v1.11.1 warning: For more information about this error, try Downloaded libc v0.2.182 warning: For more information about this error, try [ 144.513141] cloud-init[1746]: Downloaded errno v0.3.14 [ 153.740523] cloud-init[1742]: Compiling bindgen v0.70.1 total 208 |
Integration Test Failure (Run 22264260766)Branch: failure-summary.json{
"failed_step": "deploy_infrastructure",
"error": "Infrastructure deployment failed",
"exit_code": 2,
"timestamp": "2026-02-21T21:08:19.386621Z",
"sender_instance_id": "",
"receiver_instance_id": "",
"commit": "60c295e2570f9d1c0dd018652fa088d6088ad2a3",
"run_url": "https://github.com/gspivey/dpdk-stdlib-rust/actions/runs/22264260766"
}```
### instance-0-user-data.log (21814 bytes, last 80 lines)294 | let ptr = dpdk_sys::rte_pktmbuf_alloc(self.raw.as_ptr()); error[E0425]: cannot find function error[E0425]: cannot find function error[E0425]: cannot find function error[E0615]: attempted to take value of method error[E0615]: attempted to take value of method error[E0615]: attempted to take value of method Some errors have detailed explanations: E0425, E0615. 294 | let ptr = dpdk_sys::rte_pktmbuf_alloc(self.raw.as_ptr()); error[E0425]: cannot find function error[E0425]: cannot find function error[E0425]: cannot find function error[E0615]: attempted to take value of method error[E0615]: attempted to take value of method error[E0615]: attempted to take value of method Some errors have detailed explanations: E0425, E0615. [ 108.122102] cloud-init[1723]: Downloaded smallvec v1.15.1 [ 150.109183] cloud-init[1750]: 312 | dpdk_sys::rte_pktmbuf_alloc_bulk( total 248 |
Integration Test Failure (Run 22264897768)Branch: No failure-summary.json found receiver-user-data.log (12635 bytes, last 80 lines)sender-user-data.log (12635 bytes, last 80 lines)receiver-console-output.log (31695 bytes, last 80 lines)sender-console-output.log (31695 bytes, last 80 lines)All instance-logs files |
Integration Test Failure (Run 22265060891)Branch: No failure-summary.json found receiver-user-data.log (12635 bytes, last 80 lines)sender-user-data.log (12708 bytes, last 80 lines)receiver-console-output.log (31695 bytes, last 80 lines)sender-console-output.log (31786 bytes, last 80 lines)All instance-logs files |
|
Interesting the reports are posting as comments now from GitHub Actions bot. This should help with agent debugging. Real DPDK is compiling and the apps are running now. I'm going to merge this and continue debugging in next session. |
|
Closing this since another PR was merged getting to intended stage |
Integration Test Failure (Run 22267977574)Branch: failure-summary.json{
"failed_step": "wait_for_ssm_readiness",
"error": "Instances not ready within SSM timeout",
"exit_code": 2,
"timestamp": "2026-02-22T01:40:03.039745Z",
"sender_instance_id": "i-0e1522a3977ac695b",
"receiver_instance_id": "i-07197c74abecafcc9",
"commit": "dcbe72e43b8d593913949a72ab9306d7648bdac8",
"run_url": "https://github.com/gspivey/dpdk-stdlib-rust/actions/runs/22267977574"
}```
### receiver-console-output.log (23431 bytes, last 80 lines)[ 118.953374] cloud-init[1678]: Compiling thiserror-impl v1.0.69 [ 119.421210] cloud-init[1680]: Compiling thiserror-impl v1.0.69 total 60 |
- Add SSM agent enable/restart to EC2 user-data runtime config to prevent intermittent SSM readiness timeouts - Fix hardcoded PR #11 in failure diagnostics — now uses github.event.pull_request.number dynamically https://claude.ai/code/session_01PvJEBrDaQRSs6c6LxtBrhT
…ombuf Run #10 populated the NIC Drops column for the first time and surfaced two problems: 1. rust-dpdk shows ~30-40K NIC Drops per step, flat across rates, which looked like the Rust stack was dropping packets. It wasn't — the same perf run posted a testpmd diag log with RX-missed=0, RX-nombuf=0, RX-errors=403,626 over 8 steps. A C binary running testpmd (which we did not write) sees the same ~50K ierrors/step on this NIC. Those are AWS ENA background events, not TRex test-traffic drops. Lumping ierrors into a single "NIC Drops" column hid the real signal. 2. tokio-dpdk shows 2-12K NIC Drops per step while TRex observes up to 20M lost packets at 64B/700K. That's a 4000x discrepancy that our instrumentation cannot currently explain. Flagged as a followup to investigate with kernel-side ENA counters (ethtool -S) in Run #11. This commit: - Writes up Run #10 in docs/perf-test-log.md with the testpmd evidence and explicit calls-out of what the 0.15% rust-dpdk drop at 64B/700K actually is (TRex kpps measurement rounding, not DUT loss). - Splits the markdown NIC Drops column into three sub-columns so the ierrors noise does not visually contaminate the imissed/rx_nombuf real-drop signal going forward. - Removes the combined nic_drops total from annotate_with_app_drops and documents why summing imissed+ierrors+rx_nombuf is misleading on ENA. No Rust-side code changes — the [PERF] log format is unchanged. Only the aggregator and the markdown table layout change. https://claude.ai/code/session_01AEEjCeNiuSA8MWU7BPy15G
Run #11 shipped [NIC-BASELINE]/[NIC-FINAL] one-shot lines but the self-check table reported "no FINAL (abnormal shutdown)" for both rust-dpdk and tokio-dpdk. Root cause is three independent bugs in the shutdown/log-collection path — not in the instrumentation itself: 1. **tokio-echo had no SIGTERM handler.** The main loop was an infinite `loop { socket.recv_from(..).await; .. }` with no break condition. pkill -TERM from the harness terminated the process without ever running destructors, so PerfReporter::drop() never fired and [NIC-FINAL] was never emitted. Added a tokio::signal SIGTERM/SIGINT race and an explicit AsyncUdpSocket:: disable_perf_reporting() call before main returns so the reporter is joined synchronously (flush is deterministic, not dependent on Arc-refcount or runtime-shutdown ordering). 2. **run-perf-tests.sh grepped logs before stopping the DUT.** For each config the sequence was: start DUT → run benchmark → grep [PERF]/[NIC-*] from the log → next iteration's dut_stop_all_apps. [NIC-FINAL] is only emitted when the reporter thread is joined (on SIGTERM + graceful shutdown), so the grep always sampled too early. Moved dut_stop_all_apps to run AFTER benchmarks finish but BEFORE log collection, and skip the defensive top-of-iteration stop on iterations 2+. 3. **plain-rust ethtool capture had a bash precedence bug.** The `[ -n "$IFACE" ] && ethtool -S $IFACE || echo unavailable` idiom silently wrote "ethtool unavailable" to the baseline/final files whenever ethtool had any hiccup, which the Python aggregator then counted as "no data". Replaced with an explicit retry loop that waits for the freshly-rebound interface to report numeric stats and writes an unambiguous ETHTOOL_*_FAILED marker on genuine failure. Also switched apps/echo/src/main.rs signal installation from libc::signal() fn-pointer cast to POSIX sigaction() for robustness. Local validation (no real DPDK needed): - cargo test: 282 tests pass including perf_reporter_final_snapshot_ runs_after_shutdown which exercises the Drop path - Spawned target/release/echo under stubs + SIGTERM → stderr contains [NIC-BASELINE], per-tick [PERF], and [NIC-FINAL] lines cleanly - Spawned target/release/tokio-echo under stubs + SIGTERM → clean exit, "Received shutdown signal" and "Shutting down gracefully" printed (DpdkUdpSocket path exercised on real hardware only) https://claude.ai/code/session_01AEEjCeNiuSA8MWU7BPy15G
Captures the Run #11 results after bcd83ca fixed the [NIC-FINAL] emission path. This is the run where the NIC Drops Instrumentation Self-Check table shows OK for both rust-dpdk and tokio-dpdk with zero drift, and where rust-dpdk's 403,492 ierrors cross-validates against upstream testpmd's 403,450 (delta 42 packets, 0.000037%). https://claude.ai/code/session_01AEEjCeNiuSA8MWU7BPy15G
## Summary Implements ROADMAP item #11: `dpdk-stdlib-quic`: Benchmark binary (spec tasks 14.1, 14.2). ## Changes - **`dpdk-stdlib-quic/src/bin/bench.rs`**: Two-way QUIC benchmark binary with CLI args: - `--provider=stock|native-dpdk` — select I/O provider - `--duration=<secs>` — benchmark duration (default 10s) - `--streams=<n>` — concurrent streams per connection (default 1) - `--payload-size=<bytes>` — payload per stream (default 1MB) - **`dpdk-stdlib-quic/Cargo.toml`**: Added `[[bin]]` entry for `quic-bench`. Moved `tokio`, `rcgen`, `bytes` from dev-dependencies to regular dependencies (required for bin targets). ## Design - Stock provider uses s2n-quic's default Tokio I/O on localhost - Native-dpdk provider uses `PairedLoopback` backend (same pattern as loopback_handshake test) - TLS configured at runtime via rcgen self-signed certs - Reports: throughput (Gbps), PPS (approx), handshake latency P50/P99, provider stats counters - Compiles in stub mode without DPDK installed ## Tests - `cargo build && cargo test` passes (757+ tests, 0 failures) - Binary compiles successfully in stub mode (task 14.2 verified) ## Tradeoffs - Moved tokio/rcgen/bytes to regular deps since Cargo doesn't expose dev-dependencies to `[[bin]]` targets. These are lightweight and already in the dep tree via s2n-quic. --------- Co-authored-by: Agent Router <agent@agent-router.dev>
Uh oh!
There was an error while loading. Please reload this page.