Repository navigation
tokio UDP perf (0.3.1): tokio-echo fair/concurrent modes - #138
Merged
Merged
Conversation
Implement tasks 1.1-1.7 of the tokio-udp-perf (0.3.1) spec, scoped to
apps/tokio-echo/src/main.rs only (no library or harness change).
- Add a --mode {fair|concurrent} clap ValueEnum arg, defaulting to fair;
unknown values are rejected with a non-zero exit and a clear message
during Args::parse() (Req 1.3).
- fair (default): receive then send the same bytes back in the SAME task
via echo_fair() - no to_vec, no Semaphore, no tokio::spawn, no
tokio::time::timeout (Req 1.1).
- concurrent: the E2 'spawn' pattern - buf[..len].to_vec(), acquire an
owned Semaphore permit, tokio::spawn, and send under tokio::time::timeout,
echoing the EXACT received bytes (no more format!("echo: {}", ..)) (Req 1.2).
--workers/--timeout apply only in this mode (Req 1.8).
- Gate all from_utf8_lossy/format! payload work behind --verbose in both
modes; echo exact bytes when not verbose (Req 1.4, 1.5).
- Print mode=<fair|concurrent> at startup; keep the SIGTERM/SIGINT shutdown
select! and disable_perf_reporting() so [NIC-FINAL] is still emitted
(Req 1.6, 1.7).
- Add a parse-only --idle-backoff-ms <N> flag (default absent = OFF),
printed at startup; PR 1 does NOT reference set_idle_backoff (lands in
PR 5) (Req 1.4a / 1.8).
Tests (Req 1.5): factor the receive-loop body into echo_fair and
echo_concurrent_handler and exercise them over a loopback TokioUdpSocket
(the --force-tokio backend) - fair echoes exact (incl. non-UTF-8) bytes,
concurrent echoes exact bytes with no 'echo: ' prefix - plus clap arg tests
asserting fair default, fair/concurrent accepted, --mode bogus rejected, and
--idle-backoff-ms optional. Avoids a new assert_cmd dependency per task 1.5's
allowance to factor the loop into a directly-callable function.
TCP Synthetic Performance Results (run)Commit: TCP Synthetic Performance ResultsMeasures TCP engine framework overhead using a mock PacketBackend.
conn_est=1881ns, throughput=4532.2Mbps, tick=704ns |
Synthetic Performance Results (run)Commit: ✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01) Synthetic UDP Performance ResultsMeasures framework overhead: sync IPv4 Baseline
IPv6
IPv6 vs IPv4 Comparison (sync path)
IPv4 avg sync/async ratio: 1.0x, worst: 1.1x | IPv6 vs IPv4 worst ratio: 1.36x (OK)
|
Synthetic Performance Results — Graviton (run)Commit: ✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01) Synthetic UDP Performance ResultsMeasures framework overhead: sync IPv4 Baseline
IPv6
IPv6 vs IPv4 Comparison (sync path)
IPv4 avg sync/async ratio: 0.9x, worst: 1.0x | IPv6 vs IPv4 worst ratio: 1.33x (OK)
|
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: DeployInfrastructure ready.
|
[CI] Stage: DeployInfrastructure ready.
|
Integration Tests — ✅ all passed
✅ All 2 tests passed |
Integration Tests — ✅ all passed
✅ All 17 tests passed |
Integration Tests — ✅ all passed
✅ All 17 tests passed |
✅ QUIC Integration Tests Passed (run)Branch: Test Results
Summary: 2 tests, 0 failures, 0 skipped Application Logs (last 20 lines each)Network Statereceiver-network-interfaces.logsender-network-interfaces.log |
[Perf] Stage: DeployDeploying |
✅ Integration Tests Passed (Run 40256846781)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log receiver-test-client-iperf.log sender-test-client-iperf.log Full Application Logs (last 200 lines each)receiver-echo-server.logsender-echo-server.logsender-test-client.logreceiver-test-client-iperf.logsender-test-client-iperf.log
|
✅ Integration Tests Passed — Graviton (run)Branch: Test Results
Application Logs (last 20 lines)receiver-echo-server.log sender-echo-server.log sender-test-client.log |
[Perf] Stage: Deploy FAILEDCDK deploy failed. CloudFormation events: |
Performance Test Failure (Run 40259496360)Branch: failure-summary.json{
"failed_step": "perf-test",
"error": "Script exited with code 2",
"exit_code": 2,
"timestamp": "2026-10-26T02:38:32.626606Z",
"trex_instance_id": "",
"dut_instance_id": "",
"commit": "49b7b410576ee297c82c0b0becbd32a7d97ba5ed",
"run_url": "https://github.com/gspivey/dpdk-stdlib-rust/actions/runs/40259496360"
}```
### Application Logs (last 20 lines)
<details><summary>Full Application Logs (last 200 lines each)</summary>
</details>
<details><summary>Network & PCI State</summary>
</details>
<details><summary>Kernel Console (dmesg)</summary>
</details> |
[Perf] Stage: DeployDeploying |
[CI] Stage: DeployInfrastructure ready.
|
[Perf] Stage: Deploy FAILEDCDK deploy failed. CloudFormation events: |
Performance Test Failure (Run 40260119861)Branch: failure-summary.json{
"failed_step": "perf-test",
"error": "Script exited with code 2",
"exit_code": 2,
"timestamp": "2026-10-26T02:45:58.918573Z",
"trex_instance_id": "",
"dut_instance_id": "",
"commit": "49b7b410576ee297c82c0b0becbd32a7d97ba5ed",
"run_url": "https://github.com/gspivey/dpdk-stdlib-rust/actions/runs/40260119861"
}```
### Application Logs (last 20 lines)
<details><summary>Full Application Logs (last 200 lines each)</summary>
</details>
<details><summary>Network & PCI State</summary>
</details>
<details><summary>Kernel Console (dmesg)</summary>
</details> |
Owner
Author
|
| Perf run | Result | Cause |
|---|---|---|
| 40259496360 | ❌ Deploy failed | TrexInstance CREATE_FAILED → stack ROLLBACK_COMPLETE |
| 40260119861 (retry) | ❌ Deploy failed | same |
TREX_AMI_ID: ami-0e4c5917a5313cbfd
PerfTestStack | CREATE_FAILED | AWS::EC2::Instance | TrexInstance
The image id '[ami-0e4c5917a5313cbfd]' does not exist
(Status Code: 400, HandlerErrorCode: InvalidRequest)
The DPDK AMI (ami-0809db4f14f2addd4) resolved fine; only the TRex AMI ami-0e4c5917a5313cbfd no longer exists in the account, so TrexInstance cannot be created and the stack rolls back before any traffic runs. No tokio-dpdk pps numbers were produced.
Why I'm stopping here
- This is a perf-infrastructure problem (a stale/deregistered TRex AMI, refreshed by the AMI-build pipeline — ROADMAP item 43), out of scope for PR 1, whose boundary is
apps/tokio-echo/src/main.rsonly. Fixing the AMI reference / rebuilding the TRex AMI is not a change this PR should carry. - The SOP forbids substituting local/synthetic numbers into
docs/perf-test-log.md— only GitHub Actions perf data may be recorded — so the task 1.7 baseline entry, the ROADMAP tick, and the squash-merge cannot proceed until the perf stack can deploy. - I retried once (the cheap, non-destructive mitigation); it reproduced identically, so this is a confirmed blocker rather than a transient flake.
What unblocks it
Rebuild/refresh the TRex AMI (or update the TREX_AMI_ID the perf workflow resolves) so PerfTestStack can create TrexInstance, then re-run perf-tests.yml --ref agent/tokio-echo-fair-concurrent-modes. Once a perf run succeeds, the remaining steps are mechanical: append the GitHub-Actions tokio-dpdk fair numbers to docs/perf-test-log.md, flip ROADMAP item 45 to - [x] Complete · PR: #138, commit both to this branch, and squash-merge to development.
The code for tasks 1.1–1.7 is complete and green; only the infra-dependent perf baseline recording remains.
Integration Tests — ✅ all passed
✅ All 21 tests passed |
✅ TCP Integration Tests Passed (run)Branch: Test Results
Summary: 21 tests, 0 failures, 0 skipped Application Logs (last 20 lines each)Network Statereceiver-network-interfaces.logsender-network-interfaces.log |
Owner
Author
Re: "CI completed with failures" — the only red check is
|
| Check | Conclusion |
|---|---|
build (cargo build + cargo test) |
✅ success |
| integration-tests | ✅ success |
| tcp-integration-tests | ✅ success |
| quic-integration-tests | ✅ success |
| synthetic-perf | ✅ success |
| tcp-synthetic-perf | ✅ success |
| validate-cdk | ✅ success |
| nightly-gate | ✅ success |
| perf-tests | ❌ failure |
perf-tests is the only failing check, and its failure is the stale TRex AMI infrastructure issue already detailed above (exit_code: 2, empty trex_instance_id/dut_instance_id, ami-0e4c5917a5313cbfd does not exist → PerfTestStack ROLLBACK). It reproduced identically on both runs 40259496360 and 40260119861.
There is no code fix to make. This PR changes only apps/tokio-echo/src/main.rs; it does not touch the perf stack, the AMI lookup, or the workflow. The red is environmental and out of scope for PR 1. Re-triggering again would fail identically on the same missing AMI, and the SOP forbids substituting local/synthetic perf numbers into docs/perf-test-log.md.
Unblock: rebuild/refresh the TRex AMI (or correct the resolved TREX_AMI_ID) so PerfTestStack can create TrexInstance, then re-run perf-tests.yml --ref agent/tokio-echo-fair-concurrent-modes. On a successful run I will append the GitHub-Actions tokio-dpdk fair numbers to docs/perf-test-log.md, flip ROADMAP item 45 to - [x] Complete · PR: #138, commit both to this branch, and squash-merge to development.
[Perf] Stage: DeployDeploying |
[Perf] Stage: Instances Ready
|
[Perf] Stage: TRex ConfigStarting TRex configuration (MAC discovery + NIC binding)... |
[Perf] Stage: TRex Config OK
|
[Perf] Stage: TRex StartedTRex server running. Beginning benchmarks... |
[Perf] DUT ReadyDUT instance |
[Perf] Stage: Benchmark (1/4)Running |
[Perf] Benchmark Diag:
|
[Perf] Benchmark Diag:
|
[Perf] Stage: Benchmark (2/4)Running |
[Perf] Benchmark Diag:
|
[Perf] Benchmark Diag:
|
[Perf] Stage: Benchmark (3/4)Running |
[Perf] Benchmark Diag:
|
[Perf] Benchmark Diag:
|
[Perf] Stage: Benchmark (4/4)Running |
[Perf] Benchmark Diag:
|
[Perf] Benchmark Diag:
|
[Perf] Diag: testpmd logtestpmd output (last 30 lines) |
[Perf] Stage: Results[05:39:48] INFO Generating markdown summary... Performance Test Results — c6in.xlargeCommit: 64B packets
512B packets
1400B packets
8500B packets
NIC Drops Instrumentation Self-CheckCompares
plain-rust Kernel NIC Drops (ethtool -S delta)ethtool snapshots not available — baseline or final file missing in |
gspivey
added a commit
that referenced
this pull request
Oct 26, 2026
Finalize commits missed when #138 merged (one-off pre-step). - Adds the `docs/perf-test-log.md` Run #36 entry using tokio-dpdk fair numbers from GH Actions run 40270657131 (perf-results artifact; no local/synthetic data), with regression check vs Run #35. tokio-dpdk 700K/64B: 320,776 -> 681,301 RX. One flagged variance: rust-dpdk 700K/64B 7.5% drop vs 0.8% (code untouched). - Ticks item 45 as `- [x] Complete · PR: #138` and moves it to Completed. - Marks tasks 1.1-1.7 done in `.kiro/specs/tokio-udp-perf/tasks.md`. Docs only; no code changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Kiro <kiro@gspivey.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Roadmap item
ROADMAP item 45 — tokio UDP perf (0.3.1): tokio-echo fair/concurrent modes (PR 1 of the
tokio-udp-perfspec).Spec:
.kiro/specs/tokio-udp-perf/· tasks1.1,1.2,1.3,1.4,1.4a,1.5,1.6,1.7.Boundary:
apps/tokio-echo/src/main.rsonly (plus its tests). No library or harness change; no published-crate API change.What this does
--mode {fair|concurrent}clapValueEnumarg, defaultfair; an unknown value is rejected with a non-zero exit and a clear message duringArgs::parse()(1.3).fair(default): receive then send the same bytes back in the same task viaecho_fair()— noto_vec, noSemaphore, notokio::spawn, notokio::time::timeout(1.1).concurrent: the E2 "spawn" pattern —buf[..len].to_vec(), acquire an ownedSemaphorepermit,tokio::spawn, send undertokio::time::timeout, echoing the exact received bytes (no moreformat!("echo: {}", ..)) (1.2).--workers/--timeoutapply only here (1.8).from_utf8_lossy/format!payload work is gated behind--verbosein both modes; non-verbose echoes exact bytes (1.4, 1.5).mode=<fair|concurrent>at startup; keeps the SIGTERM/SIGINT shutdownselect!anddisable_perf_reporting()so[NIC-FINAL]is still emitted (1.6, 1.7).--idle-backoff-ms <N>flag (default absent = OFF), printed at startup. PR 1 deliberately does not referenceset_idle_backoff(that method lands in PR 5) (1.4a / 1.8).Tests added (task 1.5)
The receive-loop body is factored into
echo_fair/echo_concurrent_handlerso the test can call it directly over a loopbackTokioUdpSocket(the--force-tokiobackend):fair_echoes_exact_bytes_over_loopback— fair echoes exact bytes, including non-UTF-8 bytes (proves noformat!/from_utf8_lossytransform on the hot path).concurrent_echoes_exact_bytes_no_prefix— concurrent echoes exact bytes with noecho:prefix (regression guard for the old behaviour).--mode bogusrejected (InvalidValue),--idle-backoff-msoptional, existing flags still parse,Args::command().debug_assert().cargo build && cargo testfrom the workspace root is green in stub mode (tokio-echo: 8/8; full suite: all passing, no regressions) (1.6).Tradeoffs
assert_cmdor factoring the loop into a directly-callable function. I chose the latter to avoid adding a new dev-dependency (perAGENTS.md: keep the dependency surface small), which also keeps the test fast and deterministic (no subprocess/signal orchestration).tokio-dpdkfair baseline indocs/perf-test-log.mdis run after CI is green (the benchmarked app's fair default changes its numbers ~2.3×). PR 1 adds no perf gates (the gate harness lands in PR 2), so that run records a new baseline, not a gate.