Repository navigation
perf: eliminate spawn_blocking overhead in Tokio async wrapper - #38
Merged
Merged
Conversation
…ughput Adds a new `synthetic-bench` binary that measures the pure framework overhead of the Tokio async wrapper (spawn_blocking + Mutex + buf.to_vec) vs the sync dpdk_udp::UdpSocket. Uses a mock PacketBackend with ARP auto-reply so no real NIC is needed — runs in ~20 seconds. Benchmarks: TX send_to and RX recv_from at 64B and 1400B payloads. Initial results show the async wrapper is 20-188x slower than sync, with ~20us/op overhead dominated by spawn_blocking. CI integration: adds a `synthetic-perf` job to integration-tests.yml that runs on every PR, posts results as a PR comment for fast regression detection. https://claude.ai/code/session_01EpC1AocdV7TkvTiMXmtCkm
The Tokio async wrapper (DpdkUdpSocket and compat::tokio::UdpSocket) previously used spawn_blocking + tokio::sync::Mutex + buf.to_vec() for every send/recv operation, capping throughput at ~50K ops/sec due to ~20μs dispatch overhead per call. Changes: - Add try_recv_from() to dpdk_udp::UdpSocket: non-blocking single-poll that returns Ok(None) immediately instead of sleeping in a loop. Includes both pipeline and inline paths mirroring recv_from. - Replace tokio::sync::Mutex with std::sync::Mutex in DpdkUdpSocket and compat::tokio::DpdkAsyncSocket (safe because critical sections are short CPU-only ops that never await). - Remove spawn_blocking from send_to, send, recv_from, recv hot paths. Only connect (ARP resolution) and disable_perf_reporting (thread join) still use spawn_blocking. - Remove per-call buf.to_vec() allocations. - Update synthetic bench to measure the new pattern. Synthetic benchmark results (sync vs async, same backend): TX 64B: 9.3M vs 8.3M pps (1.1x ratio) RX 64B: 2.4M vs 2.9M pps (0.8x — async faster) TX 1400B: 1.6M vs 1.6M pps (1.0x — identical) RX 1400B: 856K vs 911K pps (0.9x — async faster) Average ratio: 1.0x — framework overhead eliminated. https://claude.ai/code/session_01EpC1AocdV7TkvTiMXmtCkm
Synthetic Performance Results (run)Commit: ✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01) Synthetic UDP Performance ResultsMeasures framework overhead: sync
Avg sync/async ratio: 0.9x, worst: 1.1x
|
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: SummaryAll tests PASSED. ARP seeding: kernel /proc/net/arp (automatic)
|
✅ Integration Tests Passed (Run 24350601300)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log receiver-test-client-iperf.log sender-test-client-iperf.log Full Application Logs (last 200 lines each)receiver-echo-server.logsender-echo-server.logsender-test-client.logreceiver-test-client-iperf.logsender-test-client-iperf.log
|
[Perf] Stage: DeployDeploying |
[Perf] Stage: Instances Ready
|
[Perf] Stage: TRex ConfigStarting TRex configuration (MAC discovery + NIC binding)... |
[Perf] Stage: TRex Config OK
|
[Perf] Stage: TRex StartedTRex server running. Beginning benchmarks... |
[Perf] DUT ReadyDUT instance |
[Perf] Stage: Benchmark (1/4)Running |
[Perf] Benchmark Diag:
|
[Perf] Benchmark Diag:
|
[Perf] Stage: Benchmark (2/4)Running |
[Perf] Benchmark Diag:
|
[Perf] Benchmark Diag:
|
[Perf] Stage: Benchmark (3/4)Running |
[Perf] Benchmark Diag:
|
[Perf] Benchmark Diag:
|
[Perf] Stage: Benchmark (4/4)Running |
[Perf] Benchmark Diag:
|
[Perf] Benchmark Diag:
|
[Perf] Diag: testpmd logtestpmd output (last 30 lines) |
[Perf] Stage: Results[16:42:31] INFO Generating markdown summary... Performance Test Results — unknownCommit: 64B packets
512B packets
1400B packets
8500B packets
NIC Drops Instrumentation Self-CheckCompares
plain-rust Kernel NIC Drops (ethtool -S delta)ethtool snapshots not available — baseline or final file missing in |
Records hardware benchmark results from the spawn_blocking elimination. tokio-dpdk improved from ~40K pps (Run #9) to 343K pps at 700K/64B (8.6x). Matches sync DPDK at ≤140K pps; remaining gap at >350K pps is from Tokio yield_now() scheduling latency, not framework overhead. https://claude.ai/code/session_01EpC1AocdV7TkvTiMXmtCkm
Synthetic Performance Results (run)Commit: ✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01) Synthetic UDP Performance ResultsMeasures framework overhead: sync
Avg sync/async ratio: 0.9x, worst: 1.1x
|
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: SummaryAll tests PASSED. ARP seeding: kernel /proc/net/arp (automatic)
|
[CI] Stage: SummaryAll tests PASSED. ARP seeding: kernel /proc/net/arp (automatic)
|
[Perf] Stage: DeployDeploying |
✅ Integration Tests Passed (Run 24370003640)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log receiver-test-client-iperf.log sender-test-client-iperf.log Full Application Logs (last 200 lines each)receiver-echo-server.logsender-echo-server.logsender-test-client.logreceiver-test-client-iperf.logsender-test-client-iperf.log
|
[Perf] Stage: Instances Ready
|
[Perf] Stage: TRex ConfigStarting TRex configuration (MAC discovery + NIC binding)... |
[Perf] Stage: TRex Config OK
|
[Perf] Stage: TRex StartedTRex server running. Beginning benchmarks... |
[Perf] DUT ReadyDUT instance |
[Perf] Stage: Benchmark (1/4)Running |
[Perf] Benchmark Diag:
|
[Perf] Benchmark Diag:
|
[Perf] Stage: Benchmark (2/4)Running |
[Perf] Benchmark Diag:
|
[Perf] Benchmark Diag:
|
[Perf] Stage: Benchmark (3/4)Running |
[Perf] Benchmark Diag:
|
[Perf] Benchmark Diag:
|
[Perf] Stage: Benchmark (4/4)Running |
[Perf] Benchmark Diag:
|
[Perf] Benchmark Diag:
|
[Perf] Diag: testpmd logtestpmd output (last 30 lines) |
[Perf] Stage: Results[23:55:34] INFO Generating markdown summary... Performance Test Results — unknownCommit: 64B packets
512B packets
1400B packets
8500B packets
NIC Drops Instrumentation Self-CheckCompares
plain-rust Kernel NIC Drops (ethtool -S delta)ethtool snapshots not available — baseline or final file missing in |
The spin-poll optimization (commit a02b721) did not close the high-rate tokio-dpdk performance gap. Hardware perf tests showed essentially flat results vs Run #17, and the NIC counters (imissed=0) disprove the original hypothesis that yield_now() scheduler latency was causing ring overflow. The real bottleneck is per-packet CPU cost in the async echo path (~1.4 μs overhead per packet vs sync DPDK), likely from 2× Mutex lock/unlock per packet and async_trait vtable dispatch. Spin-polling doesn't help because the empty-poll idle path is not what's limiting throughput. Reverts the spin-poll change in DpdkUdpSocket::recv_from and the matching benchmark code. Adds Run #18 to docs/perf-test-log.md documenting the negative result and future directions that could actually help (lock-free Arc<UdpSocket>, batch APIs, dedicated poll thread + mpsc). https://claude.ai/code/session_01EpC1AocdV7TkvTiMXmtCkm
Synthetic Performance Results (run)Commit: ✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01) Synthetic UDP Performance ResultsMeasures framework overhead: sync
Avg sync/async ratio: 0.9x, worst: 1.1x
|
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: SummaryAll tests PASSED. ARP seeding: kernel /proc/net/arp (automatic)
|
✅ Integration Tests Passed (Run 24373229952)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log receiver-test-client-iperf.log sender-test-client-iperf.log Full Application Logs (last 200 lines each)receiver-echo-server.logsender-echo-server.logsender-test-client.logreceiver-test-client-iperf.logsender-test-client-iperf.log
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
apps/synthetic-bench/) that measures framework overhead of the async Tokio wrapper vs syncdpdk_udp::UdpSocket. Runs in CI on every push with no AWS credentials needed (~20s).try_recv_from()todpdk_udp::UdpSocket: non-blocking single-poll receive that returnsOk(None)immediately instead of sleeping in a loop. Includes both pipeline and inline paths.DpdkUdpSocketandcompat::tokio::UdpSocket) to eliminate three sources of overhead that capped throughput at ~50K ops/sec:spawn_blockingdispatch (~10-20μs per call)tokio::sync::Mutexasync locking overhead → replaced withstd::sync::Mutexbuf.to_vec()heap allocation on every send/recv → eliminatedOnly
connect(ARP resolution) anddisable_perf_reporting(thread join) retainspawn_blocking.Synthetic benchmark results (sync vs async, same backend)
Average ratio: 1.0x — framework overhead eliminated. Previously ~50K pps async, now millions.
Test plan
cargo buildpassescargo test— all 365 tests pass, 0 failureshttps://claude.ai/code/session_01EpC1AocdV7TkvTiMXmtCkm