Skip to content

feat: add quic-tcp-ec2-integration spec and ROADMAP items 37-38 - #122

Merged
gspivey merged 1 commit into
developmentfrom
feat/quic-tcp-ec2-integration-spec
Oct 12, 2026
Merged

gspivey merged 1 commit into
developmentfrom
feat/quic-tcp-ec2-integration-spec

Conversation

@gspivey

@gspivey gspivey commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Summary

Adds spec for QUIC + TCP EC2 integration and performance CI gaps.

What's Missing Today (and What This Spec Fixes)

Gap This spec
QUIC orchestrator: no run_quic_tier1 function Item 37: adds function + --tier quic filter + cert transfer via SSM
QUIC CI: no quic-integration-tests.yml workflow Item 38: new workflow, continue-on-error until 10 clean runs
QUIC perf CI: no quic-perf-tests.yml workflow Item 38: new workflow dispatching run-perf-tests.sh --configs quic-native-dpdk-nic
TCP continue-on-error gate: no defined removal procedure Item 38: 10 consecutive Clean_Run counting runbook

Spec Contents

  • requirements.md: 10 requirements, all acceptance criteria measurable (EARS clean)
  • design.md: full orchestrator pseudocode for run_quic_tier1, quic_transfer_cert, quic_verify_build; CI workflow YAML structure; gate counting runbook
  • tasks.md: 19 tasks in 2 ROADMAP items

Review History

  • EARS validator: ✅ clean (fixed 4 vague qualifiers)
  • Claude opus-5 review round 1: 5 CRITICALs, 7 HIGHs
  • Claude opus-5 review round 2: 2 CRITICALs, 4 HIGHs (new from fixes)
  • Claude opus-5 review round 3: 1 HIGH
  • Claude opus-5 review round 4: SHIPPABLE — 0 CRITICAL, 0 HIGH

ROADMAP Items Added

  • Item 37: Wire QUIC Tier 1 into orchestrator (tasks 1–6)
  • Item 38: QUIC integration + perf CI + TCP/QUIC gate evaluation (tasks 7–19)

Key Design Decisions

  • QUIC is quic-only: not part of run-all, so primary integration-tests job is unchanged
  • Cross-workflow serialization via shared concurrency.group: integration-tests (can't use needs: across workflow files)
  • Cert transfer via base64: improves on the existing run-quic-integration-tests.sh sed-escaping approach (which is vulnerable to % in cert content)
  • ssm_run_get_output arg order: (instance_id, timeout, cmd) — deliberately reordered from run-quic-perf.sh:175's (instance_id, cmd, timeout) to match the orchestrator convention

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

Synthetic Performance Results — Graviton (run)

Commit: 1d308388

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

IPv4 Baseline

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to 64B 11.0M 10.0M 1.1x 100
RX recv_from 64B 3.3M 3.7M 0.9x 271
TX send_to 1400B 1.7M 1.7M 1.0x 603
RX recv_from 1400B 1.1M 1.1M 1.0x 913

IPv6

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to (IPv6) 64B 9.0M 8.5M 1.1x 117
RX recv_from (IPv6) 64B 3.7M 4.2M 0.9x 237
TX send_to (IPv6) 1400B 2.7M 2.7M 1.0x 372
RX recv_from (IPv6) 1400B 1.0M 1.1M 1.0x 931

IPv6 vs IPv4 Comparison (sync path)

Test Payload IPv4 PPS IPv6 PPS IPv4/IPv6 Ratio
TX send_to (sync) 64B 11.0M 9.0M 1.23x
RX recv_from (sync) 64B 3.3M 3.7M 0.90x
TX send_to (sync) 1400B 1.7M 2.7M 0.62x
RX recv_from (sync) 1400B 1.1M 1.0M 1.01x

IPv4 avg sync/async ratio: 1.0x, worst: 1.1x | IPv6 vs IPv4 worst ratio: 1.23x (OK)

OK: IPv6 is 22.9% slower than IPv4 — within acceptable threshold (<50%). Expected due to larger headers (40B vs 20B) and mandatory UDP checksum.

Good: Async wrapper is within 1.1x of sync — minimal framework overhead.

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

Integration Test Failure — Graviton (Run 33505065517)

Branch: 122/merge | Commit: 1d308388331a74d510f3271cc58febf6c492337a

failure-summary.json

{
  "failed_step": "deploy_infrastructure",
  "error": "Infrastructure deployment failed",
  "exit_code": 2,
  "timestamp": "2026-09-01T12:09:19.887053Z",
  "sender_instance_id": "i-0b8664dad5014cf4d",
  "receiver_instance_id": "i-09d42a02c9cf7bd55",
  "commit": "1d308388331a74d510f3271cc58febf6c492337a",
  "run_url": "https://github.com/gspivey/dpdk-stdlib-rust/actions/runs/33505065517"
}```

### receiver-console-output.log (65117 bytes, last 80 lines)

[ 2332.997695] __mutex_lock.constprop.0+0x2cc/0x91c
[ 2332.998344] __mutex_lock_slowpath+0x1c/0x28
[ 2332.998914] mutex_lock+0x60/0x78
[ 2332.999369] driver_set_override+0x88/0x150
[ 2332.999925] driver_override_store+0x24/0x40
[ 2333.000510] dev_attr_store+0x20/0x40
[ 2333.001008] sysfs_kf_write+0x4c/0x60
[ 2333.001500] kernfs_fop_write_iter+0x140/0x1d8
[ 2333.002099] vfs_write+0x2a4/0x3a0
[ 2333.002556] ksys_write+0x74/0x104
[ 2333.003018] __arm64_sys_write+0x24/0x3c
[ 2333.003542] invoke_syscall+0x4c/0x100
[ 2333.004050] el0_svc_common.constprop.0+0x54/0x188
[ 2333.004698] do_el0_svc+0x3c/0x98
[ 2333.005152] el0_svc+0x34/0x108
[ 2333.005569] el0t_64_sync_handler+0xbc/0x140
[ 2333.006148] INFO: task cat:46309 blocked for more than 122 seconds.
[ 2333.006999] Tainted: G U 6.1.177-224.371.amzn2023.aarch64 #1
[ 2333.008024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2333.009086] task:cat state:D stack:0 pid:46309 ppid:46308 flags:0x00000004
[ 2333.010220] Call trace:
[ 2333.010544] __switch_to+0xe0/0x140
[ 2333.011019] __schedule+0x268/0x700
[ 2333.011488] schedule+0x58/0xf4
[ 2333.011909] schedule_preempt_disabled+0x18/0x24
[ 2333.012526] __mutex_lock.constprop.0+0x2cc/0x91c
[ 2333.013165] __mutex_lock_slowpath+0x1c/0x28
[ 2333.013747] mutex_lock+0x60/0x78
[ 2333.014199] driver_override_show+0x30/0x74
[ 2333.014754] dev_attr_show+0x28/0x70
[ 2333.015236] sysfs_kf_seq_show+0x90/0x140
[ 2333.015784] kernfs_seq_show+0x34/0x40
[ 2333.016283] seq_read_iter+0x1dc/0x4ec
[ 2333.016795] kernfs_fop_read_iter+0x40/0x4c
[ 2333.017361] vfs_read+0x210/0x2c0
[ 2333.017813] ksys_read+0x74/0x104
[ 2333.018263] __arm64_sys_read+0x24/0x3c
[ 2333.018783] invoke_syscall+0x4c/0x100
[ 2333.019280] el0_svc_common.constprop.0+0x54/0x188
[ 2333.019934] do_el0_svc+0x3c/0x98
[ 2333.020386] el0_svc+0x34/0x108
[ 2333.020797] el0t_64_sync_handler+0xbc/0x140
[ 2348.330489] vfio-pci 0000:00:06.0: Relaying device request to user (#60)
[ 2450.717749] vfio-pci 0000:00:06.0: Relaying device request to user (#70)
[ 2553.115204] vfio-pci 0000:00:06.0: Relaying device request to user (#80)
[ 2655.512224] vfio-pci 0000:00:06.0: Relaying device request to user (#90)
[ 2757.909470] vfio-pci 0000:00:06.0: Relaying device request to user (#100)
[ 2860.306709] vfio-pci 0000:00:06.0: Relaying device request to user (#110)
[ 2962.703946] vfio-pci 0000:00:06.0: Relaying device request to user (#120)
[ 3065.101187] vfio-pci 0000:00:06.0: Relaying device request to user (#130)
[ 3167.498434] vfio-pci 0000:00:06.0: Relaying device request to user (#140)
[ 3269.895690] vfio-pci 0000:00:06.0: Relaying device request to user (#150)
[ 3372.292934] vfio-pci 0000:00:06.0: Relaying device request to user (#160)
[ 3474.690174] vfio-pci 0000:00:06.0: Relaying device request to user (#170)
[ 3577.087484] vfio-pci 0000:00:06.0: Relaying device request to user (#180)
[ 3679.484659] vfio-pci 0000:00:06.0: Relaying device request to user (#190)
[ 3781.881903] vfio-pci 0000:00:06.0: Relaying device request to user (#200)
2026/08/18 06:38:45Z: SSM Agent unable to acquire credentials: unexpected error getting instance profile role credentials or calling UpdateInstanceInformation. Skipping default host management fallback: retrieved credentials failed to report to ssm. Error: RequestError: send request failed
caused by: Post "https://ssm.us-east-1.amazonaws.com/": dial tcp 44.216.200.77:443: i/o timeout
[ 3884.279144] vfio-pci 0000:00:06.0: Relaying device request to user (#210)
[ 3986.676371] vfio-pci 0000:00:06.0: Relaying device request to user (#220)
[ 4089.073608] vfio-pci 0000:00:06.0: Relaying device request to user (#230)
[ 4191.470850] vfio-pci 0000:00:06.0: Relaying device request to user (#240)
[ 4293.868104] vfio-pci 0000:00:06.0: Relaying device request to user (#250)
[ 4396.265349] vfio-pci 0000:00:06.0: Relaying device request to user (#260)
[ 4498.662592] vfio-pci 0000:00:06.0: Relaying device request to user (#270)
[ 4601.059830] vfio-pci 0000:00:06.0: Relaying device request to user (#280)
[ 4703.457075] vfio-pci 0000:00:06.0: Relaying device request to user (#290)
[ 4805.854333] vfio-pci 0000:00:06.0: Relaying device request to user (#300)
[ 4908.251629] vfio-pci 0000:00:06.0: Relaying device request to user (#310)
[ 5010.648849] vfio-pci 0000:00:06.0: Relaying device request to user (#320)
[ 5113.046102] vfio-pci 0000:00:06.0: Relaying device request to user (#330)
[ 5215.443341] vfio-pci 0000:00:06.0: Relaying device request to user (#340)
[ 5263.101135] vfio-pci 0000:00:06.0: Removing from iommu group 0
[330664.887347] systemd-journald[841]: Data hash table of /var/log/journal/ec2ca47fa0117cf11393321e1c9d9603/system.journal has a fill level at 75.0 (174766 of 233016 items, 58720256 file size, 335 bytes per hash table item), suggesting rotation.
[330664.890261] systemd-journald[841]: /var/log/journal/ec2ca47fa0117cf11393321e1c9d9603/system.journal: Journal header limits reached or header out-of-date, rotating.
[701216.601955] systemd-journald[841]: Data hash table of /var/log/journal/ec2ca47fa0117cf11393321e1c9d9603/system.journal has a fill level at 75.0 (174764 of 233016 items, 58720256 file size, 335 bytes per hash table item), suggesting rotation.
[701216.604805] systemd-journald[841]: /var/log/journal/ec2ca47fa0117cf11393321e1c9d9603/system.journal: Journal header limits reached or header out-of-date, rotating.
[1071617.858684] systemd-journald[841]: Data hash table of /var/log/journal/ec2ca47fa0117cf11393321e1c9d9603/system.journal has a fill level at 75.0 (174766 of 233016 items, 58720256 file size, 335 bytes per hash table item), suggesting rotation.
[1071617.861535] systemd-journald[841]: /var/log/journal/ec2ca47fa0117cf11393321e1c9d9603/system.journal: Journal header limits reached or header out-of-date, rotating.


### sender-console-output.log (29 bytes, last 80 lines)

(console output unavailable)


@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

❌ Integration Tests Failed — Graviton (run)

Branch: 122/merge | Commit: 1d308388

Test Results

Application Logs (last 20 lines)

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

TCP Synthetic Performance Results (run)

Commit: 1d308388

TCP Synthetic Performance Results

Measures TCP engine framework overhead using a mock PacketBackend.

Test Metric Value Unit
connection_establishment latency_ns 1881 ns
single_stream_throughput throughput_mbps 4412.1 Mbps
engine_tick latency_ns 709 ns

conn_est=1881ns, throughput=4412.1Mbps, tick=709ns

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

Synthetic Performance Results (run)

Commit: 1d308388

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

IPv4 Baseline

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to 64B 11.8M 11.3M 1.0x 88
RX recv_from 64B 3.6M 4.7M 0.8x 213
TX send_to 1400B 1.8M 1.8M 1.0x 551
RX recv_from 1400B 1.1M 1.1M 1.0x 869

IPv6

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to (IPv6) 64B 9.0M 8.7M 1.0x 114
RX recv_from (IPv6) 64B 4.0M 5.5M 0.7x 181
TX send_to (IPv6) 1400B 3.1M 3.0M 1.0x 330
RX recv_from (IPv6) 1400B 1.2M 1.2M 1.0x 834

IPv6 vs IPv4 Comparison (sync path)

Test Payload IPv4 PPS IPv6 PPS IPv4/IPv6 Ratio
TX send_to (sync) 64B 11.8M 9.0M 1.32x
RX recv_from (sync) 64B 3.6M 4.0M 0.89x
TX send_to (sync) 1400B 1.8M 3.1M 0.60x
RX recv_from (sync) 1400B 1.1M 1.2M 0.96x

IPv4 avg sync/async ratio: 0.9x, worst: 1.0x | IPv6 vs IPv4 worst ratio: 1.32x (OK)

OK: IPv6 is 32.1% slower than IPv4 — within acceptable threshold (<50%). Expected due to larger headers (40B vs 20B) and mandatory UDP checksum.

Good: Async wrapper is within 1.0x of sync — minimal framework overhead.

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

[CI] Stage: Deploy

Infrastructure ready.

  • Sender: i-060745aaa10c66387 (DPDK ENI: 10.0.1.21)
  • Receiver: i-0726ce28db7472ef5 (DPDK ENI: 10.0.1.29)
  • Both instances SSM-ready.

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

Integration Tests — ⚠️ 2 infra flakes · 0 real failures

1d30838 · run · Integration Tests · 15 tests: 13 ✅ · 0 ❌ real · 2 ⚠️ infra

Tier Test Result Time Reason
tier1-dpdk-echo arp_resolution ✅ pass 2.412s
tier1-dpdk-echo udp_send_receive ✅ pass 1.752s
tier1-dpdk-echo echo_roundtrip ✅ pass 1.551s
tier1-dpdk-echo payload_integrity ✅ pass 0.744s
tier1-dpdk-echo jumbo_diagnostics ✅ pass 0.025s
tier1-dpdk-echo jumbo_echo_8000 ✅ pass 1.746s
tier1-tcp-echo-async execution ⚠️ infra 0.000s ENI bind failed on receiver instance
tier1-tcp-echo bidir_echo ✅ pass 0.736s
tier1-tcp-echo bidir_multi ✅ pass 0.771s
tier2-kernel-interop arp_resolution ✅ pass 0.064s
tier2-kernel-interop udp_send_receive ✅ pass 1.079s
tier2-kernel-interop echo_roundtrip ✅ pass 0.871s
tier2-kernel-interop payload_integrity ✅ pass 0.068s
tier2-tcp-echo execution ⚠️ infra 0.000s ENI bind failed on receiver instance
tier3-iperf-interop our_app_sends ✅ pass 2.568s

⚠️ Infra flakes (2) — SSM/ENI setup, not code (task #10)

  • tier1-tcp-echo-async / execution — ENI bind failed on receiver instance
  • tier2-tcp-echo / execution — ENI bind failed on receiver instance

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

❌ Integration Tests Failed (Run 33505065263)

Branch: 122/merge | Commit: 1d308388

Test Results

  • tier1-dpdk-echo.xml: 6 tests, 0 failures, skipped
  • tier1-tcp-echo-async.xml: 1 tests, 1 failures, skipped
    FAIL: ENI bind failed on receiver instance
    
  • tier1-tcp-echo.xml: 2 tests, 0 failures, skipped
  • tier2-kernel-interop.xml: 4 tests, 0 failures, skipped
  • tier2-tcp-echo.xml: 1 tests, 1 failures, skipped
    FAIL: ENI bind failed on receiver instance
    
  • tier3-our-app-sends.xml: 1 tests, 0 failures, skipped

Application Logs (last 20 lines)

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.21:9000 (MAC: 02:e9:c0:d9:bc:49)
echo listening on 10.0.1.21:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.29:9000: 'arp-probe #1'
Test complete
[2026-09-01T12:44:59Z] INFO: ARP resolution succeeded (got response from peer)
[2026-09-01T12:44:59Z] INFO: Test: udp_send_receive
[2026-09-01T12:45:00Z] INFO: UDP send/receive succeeded
[2026-09-01T12:45:00Z] INFO: Test: echo_roundtrip
[2026-09-01T12:45:01Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-09-01T12:45:01Z] INFO: Test: payload_integrity
[2026-09-01T12:45:01Z] INFO: Response received, checking payload match...
[2026-09-01T12:45:01Z] INFO: Payload integrity verified (found in response)
[2026-09-01T12:45:01Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-09-01T12:45:01Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

sender-test-client-iperf.log

Received 30 bytes from 10.0.1.29:9000: 'dpdk-to-kernel-test-payload #3'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #4'
Received 30 bytes from 10.0.1.29:9000: 'dpdk-to-kernel-test-payload #4'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #5'
Received 30 bytes from 10.0.1.29:9000: 'dpdk-to-kernel-test-payload #5'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #6'
Received 30 bytes from 10.0.1.29:9000: 'dpdk-to-kernel-test-payload #6'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #7'
Received 30 bytes from 10.0.1.29:9000: 'dpdk-to-kernel-test-payload #7'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #8'
Received 30 bytes from 10.0.1.29:9000: 'dpdk-to-kernel-test-payload #8'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #9'
Received 30 bytes from 10.0.1.29:9000: 'dpdk-to-kernel-test-payload #9'
Sent 31 bytes: 'dpdk-to-kernel-test-payload #10'
Received 31 bytes from 10.0.1.29:9000: 'dpdk-to-kernel-test-payload #10'
Test complete
[2026-09-01T12:47:08Z] INFO: our-app-sends: sent 10 packets, received 10 responses
[2026-09-01T12:47:08Z] INFO: our-app-sends: PASS (sent >= 5 packets)
[2026-09-01T12:47:08Z] INFO: JUnit XML written to: /tmp/test-results/tier3-our-app-sends.xml
[2026-09-01T12:47:08Z] INFO: our-app-sends test complete
Full Application Logs (last 200 lines each)

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.21:9000 (MAC: 02:e9:c0:d9:bc:49)
echo listening on 10.0.1.21:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

[2026-09-01T12:42:47Z] INFO: Test: arp_resolution
UDP Test Client
Target: 10.0.1.29:9000
Bind address: 10.0.1.21:0
Message: 'arp-probe'
Count: 1
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.21:32768 (MAC: 02:e9:c0:d9:bc:49)
Backend: dpdk
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.29:9000: 'arp-probe #1'
Test complete
[2026-09-01T12:42:50Z] INFO: ARP resolution succeeded (got response from peer)
[2026-09-01T12:42:50Z] INFO: Test: udp_send_receive
[2026-09-01T12:42:51Z] INFO: UDP send/receive succeeded
[2026-09-01T12:42:52Z] INFO: Test: echo_roundtrip
[2026-09-01T12:42:53Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-09-01T12:42:53Z] INFO: Test: payload_integrity
[2026-09-01T12:42:54Z] INFO: Response received, checking payload match...
[2026-09-01T12:42:54Z] INFO: Payload integrity verified (found in response)
[2026-09-01T12:42:54Z] INFO: Test: jumbo_diagnostics
[2026-09-01T12:42:54Z] INFO: === JUMBO FRAME DIAGNOSTICS ===
[2026-09-01T12:42:54Z] INFO: Interface MTU:
  1500
  9001
  65536
[2026-09-01T12:42:54Z] INFO:   docker0: MTU=1500
[2026-09-01T12:42:54Z] INFO:   ens5: MTU=9001
[2026-09-01T12:42:54Z] INFO:   lo: MTU=65536
[2026-09-01T12:42:54Z] INFO: Routing table MTU column:
Iface	Destination	Gateway 	Flags	RefCnt	Use	Metric	Mask		MTU	Window	IRTT                                                       
ens5	00000000	0101000A	0003	0	0	50	00000000	0	0	0                                                                              
ens5	00000000	0101000A	0003	0	0	512	00000000	0	0	0                                                                             
ens5	0200000A	0101000A	0007	0	0	512	FFFFFFFF	0	0	0                                                                             
ens5	0001000A	00000000	0001	0	0	512	00FFFFFF	0	0	0                                                                             
[2026-09-01T12:42:54Z] INFO: DPDK port config (from echo server log):
[2026-09-01T12:42:54Z] INFO:   (no MTU info in echo log)
[2026-09-01T12:42:54Z] INFO: === END JUMBO DIAGNOSTICS ===
[2026-09-01T12:42:54Z] INFO: Test: jumbo_echo_8000
[2026-09-01T12:42:56Z] INFO: Jumbo output: UDP Test Client
Target: 10.0.1.29:9000
Bind address: 10.0.1.21:0
Payload size: 8000 bytes
Count: 3
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.21:32768 (MAC: 02:e9:c0:d9:bc:49)
Backend: dpdk
Sending packets...
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.29:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.29:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.29:9000 (expected 8000, OK)
Test complete
[2026-09-01T12:42:56Z] INFO: Jumbo frame echo succeeded: 3/3 responses with correct size
[2026-09-01T12:42:56Z] INFO: JUnit XML written to: /tmp/test-results/tier1-dpdk-echo.xml
[2026-09-01T12:42:56Z] INFO: Tier 1 sender tests complete. Results: /tmp/test-results/tier1-dpdk-echo.xml
[2026-09-01T12:44:59Z] INFO: Test: arp_resolution
UDP Test Client
Target: 10.0.1.29:9000
Bind address: 0.0.0.0:0
Message: 'arp-probe'
Count: 1
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.29:9000: 'arp-probe #1'
Test complete
[2026-09-01T12:44:59Z] INFO: ARP resolution succeeded (got response from peer)
[2026-09-01T12:44:59Z] INFO: Test: udp_send_receive
[2026-09-01T12:45:00Z] INFO: UDP send/receive succeeded
[2026-09-01T12:45:00Z] INFO: Test: echo_roundtrip
[2026-09-01T12:45:01Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-09-01T12:45:01Z] INFO: Test: payload_integrity
[2026-09-01T12:45:01Z] INFO: Response received, checking payload match...
[2026-09-01T12:45:01Z] INFO: Payload integrity verified (found in response)
[2026-09-01T12:45:01Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-09-01T12:45:01Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

sender-test-client-iperf.log

[2026-09-01T12:47:05Z] INFO: Pre-flight: checking DPDK state and ARP cache...
[2026-09-01T12:47:05Z] INFO: Local IP: 10.0.1.21, Peer IP: 10.0.1.29, Port: 9000
[2026-09-01T12:47:05Z] INFO: /proc/net/arp contents:
IP address       HW type     Flags       HW address            Mask     Device
10.0.1.29        0x1         0x2         02:07:69:40:bc:47     *        ens5
10.0.1.1         0x1         0x2         02:18:8e:6b:ae:91     *        ens5
10.0.1.21        0x1         0x2         02:e9:c0:d9:bc:49     *        ens5
[2026-09-01T12:47:05Z] INFO: DPDK runtime state:
No /var/run/dpdk/ directory
[2026-09-01T12:47:05Z] INFO: vfio-pci bindings:
0000:00:06.0
bind
module
new_id
remove_id
uevent
unbind
[2026-09-01T12:47:05Z] INFO: Test binary: /opt/dpdk-stdlib/target/release/test-client
-rwxr-xr-x. 2 root root 2041184 Sep  1 12:39 /opt/dpdk-stdlib/target/release/test-client
[2026-09-01T12:47:05Z] INFO: Launching test-client: /opt/dpdk-stdlib/target/release/test-client --target 10.0.1.29 --port 9000 --bind-ip 10.0.1.21 --count 10 --delay 200
[2026-09-01T12:47:08Z] INFO: Test client output: UDP Test Client
Target: 10.0.1.29:9000
Bind address: 10.0.1.21:0
Message: 'dpdk-to-kernel-test-payload'
Count: 10
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.21:32768 (MAC: 02:e9:c0:d9:bc:49)
Backend: dpdk
Sending packets...
Sent 30 bytes: 'dpdk-to-kernel-test-payload #1'
Received 30 bytes from 10.0.1.29:9000: 'dpdk-to-kernel-test-payload #1'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #2'
Received 30 bytes from 10.0.1.29:9000: 'dpdk-to-kernel-test-payload #2'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #3'
Received 30 bytes from 10.0.1.29:9000: 'dpdk-to-kernel-test-payload #3'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #4'
Received 30 bytes from 10.0.1.29:9000: 'dpdk-to-kernel-test-payload #4'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #5'
Received 30 bytes from 10.0.1.29:9000: 'dpdk-to-kernel-test-payload #5'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #6'
Received 30 bytes from 10.0.1.29:9000: 'dpdk-to-kernel-test-payload #6'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #7'
Received 30 bytes from 10.0.1.29:9000: 'dpdk-to-kernel-test-payload #7'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #8'
Received 30 bytes from 10.0.1.29:9000: 'dpdk-to-kernel-test-payload #8'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #9'
Received 30 bytes from 10.0.1.29:9000: 'dpdk-to-kernel-test-payload #9'
Sent 31 bytes: 'dpdk-to-kernel-test-payload #10'
Received 31 bytes from 10.0.1.29:9000: 'dpdk-to-kernel-test-payload #10'
Test complete
[2026-09-01T12:47:08Z] INFO: our-app-sends: sent 10 packets, received 10 responses
[2026-09-01T12:47:08Z] INFO: our-app-sends: PASS (sent >= 5 packets)
[2026-09-01T12:47:08Z] INFO: JUnit XML written to: /tmp/test-results/tier3-our-app-sends.xml
[2026-09-01T12:47:08Z] INFO: our-app-sends test complete
⚠️ SSM Command Failures (receiver-ssm-failure.log)
=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0726ce28db7472ef5 (receiver)
Command ID: 16ac2bdd-447a-41d4-b301-fbddf3ce608b

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0726ce28db7472ef5 (receiver)
Command ID: 65b39d01-64d9-48ff-9326-95af2df6ab39

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0726ce28db7472ef5 (receiver)
Command ID: 728b43f9-b148-40e6-a8e4-18f61c3bb338

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0726ce28db7472ef5 (receiver)
Command ID: 0be7599b-54d1-432c-963d-293ebf7e1cd4

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0726ce28db7472ef5 (receiver)
Command ID: f746e2a1-3d3a-4177-8469-234770fb7c4f

=== STDOUT ===


=== STDERR ===


=== Polling timeout 120s (SSM timeout: 90s, command: pkill -f 'target/release/echo' 2>/dev/null || true; pkill -f 'target/release/test-client' 2>/dev/null || true; pkill -f 'target/release/tokio-echo' 2>/dev/null || true; pkill -f 'target/release/tcp-echo' 2>/dev/null || true; pkill -f 'target/release/tokio-tcp-echo' 2>/dev/null || true; pkill -f 'target/release/tcp-test-client' 2>/dev/null || true; pkill -f 'target/release/tcp-kernel-client' 2>/dev/null || true; pkill -f 'target/release/plain-tcp-echo' 2>/dev/null || true; pkill -f 'python3.*udp_echo' 2>/dev/null || true; sleep 2; rm -rf /var/run/dpdk/ 2>/dev/null || true; cd /opt/dpdk-stdlib && bash scripts/integration-tests/configure-eni.sh --action unbind && bash scripts/integration-tests/configure-eni.sh --action assign-ip --ip 10.0.1.29) ===
Status: InProgress
Instance: i-0726ce28db7472ef5 (receiver)
Command ID: c319391e-d50f-4c6f-92e7-796d057bea2c

=== STDOUT ===


=== STDERR ===


=== Polling timeout 120s (SSM timeout: 90s, command: cd /opt/dpdk-stdlib && bash scripts/integration-tests/configure-eni.sh --action bind) ===
Status: InProgress
Instance: i-0726ce28db7472ef5 (receiver)
Command ID: 6329f769-ff28-4bbb-8ad1-96096fa8e7ae

=== STDOUT ===


=== STDERR ===


=== Polling timeout 120s (SSM timeout: 90s, command: pkill -f 'target/release/echo' 2>/dev/null || true; pkill -f 'target/release/test-client' 2>/dev/null || true; pkill -f 'target/release/tokio-echo' 2>/dev/null || true; pkill -f 'target/release/tcp-echo' 2>/dev/null || true; pkill -f 'target/release/tokio-tcp-echo' 2>/dev/null || true; pkill -f 'target/release/tcp-test-client' 2>/dev/null || true; pkill -f 'target/release/tcp-kernel-client' 2>/dev/null || true; pkill -f 'target/release/plain-tcp-echo' 2>/dev/null || true; pkill -f 'python3.*udp_echo' 2>/dev/null || true; sleep 2; rm -rf /var/run/dpdk/ 2>/dev/null || true; cd /opt/dpdk-stdlib && bash scripts/integration-tests/configure-eni.sh --action unbind && bash scripts/integration-tests/configure-eni.sh --action assign-ip --ip 10.0.1.29) ===
Status: InProgress
Instance: i-0726ce28db7472ef5 (receiver)
Command ID: ad76abec-eb45-4841-82a9-e4327e0cadc8

=== STDOUT ===


=== STDERR ===


=== Polling timeout 120s (SSM timeout: 90s, command: cd /opt/dpdk-stdlib && bash scripts/integration-tests/configure-eni.sh --action bind) ===
Status: InProgress
Instance: i-0726ce28db7472ef5 (receiver)
Command ID: 57443b34-8c4a-4bfe-9315-063a8286c1de

=== STDOUT ===


=== STDERR ===


=== Polling timeout 60s (SSM timeout: 30s, command: cd /opt/dpdk-stdlib && bash scripts/integration-tests/configure-eni.sh --action status) ===
Status: InProgress
Instance: i-0726ce28db7472ef5 (receiver)
Command ID: be017b46-6790-45bb-9ef9-cf1abb6cf660

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 60s ===
Status: InProgress
Instance: i-0726ce28db7472ef5 (receiver)
Command ID: ecde9438-9a02-407a-bb35-9dbaa3b90fcd

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 60s ===
Status: InProgress
Instance: i-0726ce28db7472ef5 (receiver)
Command ID: 4dfa9180-e5ce-43c8-bb37-288fa05ce958

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 60s ===
Status: InProgress
Instance: i-0726ce28db7472ef5 (receiver)
Command ID: 4c78c0ac-9613-4718-b662-a3d40dc11c5b

=== STDOUT ===


=== STDERR ===


⚠️ SSM Command Failures (sender-ssm-failure.log)
=== Polling timeout after 30s ===
Status: InProgress
Instance: i-060745aaa10c66387 (sender)
Command ID: f9686a65-5f7d-475f-acfc-662142e15e18

=== STDOUT ===


=== STDERR ===


Network & PCI State

sender-network-interfaces.log

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:c8:41:8e:6a:f5 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-07bdaa0d4225e8cd5
    altname device-number-0.0
    inet 10.0.1.169/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3488sec preferred_lft 3488sec
    inet6 fe80::c8:41ff:fe8e:6af5/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:71:b1:7d:d0 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
9: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:e9:c0:d9:bc:49 brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    altname eni-015c281bbbe82f2ce
    altname device-number-1.0
    inet 10.0.1.21/24 metric 513 brd 10.0.1.255 scope global dynamic ens6
       valid_lft 3118sec preferred_lft 3118sec
    inet6 fe80::e9:c0ff:fed9:bc49/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

receiver-networking-diag-baseline.txt

=== NETWORKING DIAGNOSTICS ===
timestamp: 2026-09-01T12:42:08Z
hostname: ip-10-0-1-226.ec2.internal
kernel: 6.1.180-225.360.amzn2023.x86_64

=== DPDK PORT STATUS ===

Network devices using DPDK-compatible driver
============================================
0000:00:06.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:05.0 'Elastic Network Adapter (ENA) ec20' if=ens5 drv=ena unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

=== IP ADDRESSES ===
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:c4:37:97:07:d3 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0176903a00974ca9f
    altname device-number-0.0
    inet 10.0.1.226/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3146sec preferred_lft 3146sec
    inet6 fe80::c4:37ff:fe97:7d3/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:28:f3:f4:c5 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

=== ARP TABLE ===
10.0.1.1 dev ens5 lladdr 02:18:8e:6b:ae:91 REACHABLE 

=== ROUTE TABLE ===
default via 10.0.1.1 dev ens5 metric 50 
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.226 metric 512 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.226 metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.226 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.226 metric 512 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 

=== IMDS: ENI INFORMATION ===
ENI MACs found: 02:07:69:40:bc:47/ 02:c4:37:97:07:d3/ 

--- ENI: 02:07:69:40:bc:47/ ---
  device-number: 1
  local-ipv4s: 10.0.1.29
  subnet-id: subnet-020578816b369060b
  subnet-cidr: 10.0.1.0/24

--- ENI: 02:c4:37:97:07:d3/ ---
  device-number: 0
  local-ipv4s: 10.0.1.226
  subnet-id: subnet-020578816b369060b
  subnet-cidr: 10.0.1.0/24


=== GATEWAY ARP TEST ===
Gateway IP: 10.0.1.1
Gateway ARP entry:
10.0.1.1 dev ens5 lladdr 02:18:8e:6b:ae:91 REACHABLE 

arping result:
ARPING 10.0.1.1 from 10.0.1.226 ens5
Unicast reply from 10.0.1.1 [02:18:8E:6B:AE:91]  0.536ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

=== HUGEPAGE STATUS ===
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

=== VFIO STATUS ===
total 0
drwxr-xr-x.  2 root root       80 Sep  1 12:41 .
drwxr-xr-x. 14 root root     3100 Sep  1 12:41 ..
crw-------.  1 root root 243,   0 Sep  1 12:41 noiommu-0
crw-rw-rw-.  1 root root  10, 196 Sep  1 12:34 vfio

noiommu mode:
Y

=== DPDK SHARED MEMORY ===
no /var/run/dpdk/ directory (clean state)

=== DPDK-RELATED DMESG (last 30 lines) ===
[    0.066497] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.073154] audit: type=2000 audit(1788266065.109:1): state=initialized audit_enabled=0 res=1
[    0.075308] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.099726] ACPI: Interpreter enabled
[    0.099726] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.123214] pci 0000:00:05.0: enabling Extended Tags
[    0.147859] SGI XFS with ACLs, security attributes, quota, no debug enabled
[    0.178076] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.258295] IPI shorthand broadcast: enabled
[    0.258357] AES CTR mode by8 optimization enabled
[    5.684771] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    5.728229] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    5.794797] VFIO - User Level meta-driver version: 0.3
[    7.239212] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.17.2g
[    7.256614] ena 0000:00:05.0: ENA device version: 0.10
[    7.257298] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    7.354617] ena 0000:00:05.0: ENA Large LLQ is disabled
[    7.366849] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0500000, mac addr 02:c4:37:97:07:d3
[    7.604749] ena 0000:00:05.0 ens5: renamed from eth0
[    8.869451] ena 0000:00:05.0 ens5: Local page cache is disabled for less than 16 channels
[  411.081825] pci 0000:00:06.0: enabling Extended Tags
[  411.085363] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  411.093338] ena 0000:00:06.0: ENA device version: 0.10
[  411.094073] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  411.193323] ena 0000:00:06.0: ENA Large LLQ is disabled
[  411.205390] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:07:69:40:bc:47
[  411.263358] ena 0000:00:06.0 ens6: renamed from eth0
[  411.551938] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[  450.808815] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  450.810083] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device

=== DPDK-RELATED PROCESSES ===
no DPDK processes running

=== END DIAGNOSTICS ===

sender-networking-diag-baseline.txt

=== NETWORKING DIAGNOSTICS ===
timestamp: 2026-09-01T12:42:01Z
hostname: ip-10-0-1-169.ec2.internal
kernel: 6.1.180-225.360.amzn2023.x86_64

=== DPDK PORT STATUS ===

Network devices using DPDK-compatible driver
============================================
0000:00:06.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:05.0 'Elastic Network Adapter (ENA) ec20' if=ens5 drv=ena unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

=== IP ADDRESSES ===
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:c8:41:8e:6a:f5 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-07bdaa0d4225e8cd5
    altname device-number-0.0
    inet 10.0.1.169/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3153sec preferred_lft 3153sec
    inet6 fe80::c8:41ff:fe8e:6af5/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:71:b1:7d:d0 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

=== ARP TABLE ===
10.0.1.1 dev ens5 lladdr 02:18:8e:6b:ae:91 REACHABLE 

=== ROUTE TABLE ===
default via 10.0.1.1 dev ens5 metric 50 
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.169 metric 512 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.169 metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.169 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.169 metric 512 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 

=== IMDS: ENI INFORMATION ===
ENI MACs found: 02:c8:41:8e:6a:f5/ 02:e9:c0:d9:bc:49/ 

--- ENI: 02:c8:41:8e:6a:f5/ ---
  device-number: 0
  local-ipv4s: 10.0.1.169
  subnet-id: subnet-020578816b369060b
  subnet-cidr: 10.0.1.0/24

--- ENI: 02:e9:c0:d9:bc:49/ ---
  device-number: 1
  local-ipv4s: 10.0.1.21
  subnet-id: subnet-020578816b369060b
  subnet-cidr: 10.0.1.0/24


=== GATEWAY ARP TEST ===
Gateway IP: 10.0.1.1
Gateway ARP entry:
10.0.1.1 dev ens5 lladdr 02:18:8e:6b:ae:91 REACHABLE 

arping result:
ARPING 10.0.1.1 from 10.0.1.169 ens5
Unicast reply from 10.0.1.1 [02:18:8E:6B:AE:91]  0.537ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

=== HUGEPAGE STATUS ===
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

=== VFIO STATUS ===
total 0
drwxr-xr-x.  2 root root       80 Sep  1 12:41 .
drwxr-xr-x. 14 root root     3100 Sep  1 12:41 ..
crw-------.  1 root root 243,   0 Sep  1 12:41 noiommu-0
crw-rw-rw-.  1 root root  10, 196 Sep  1 12:34 vfio

noiommu mode:
Y

=== DPDK SHARED MEMORY ===
no /var/run/dpdk/ directory (clean state)

=== DPDK-RELATED DMESG (last 30 lines) ===
[    0.066598] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.073173] audit: type=2000 audit(1788266064.131:1): state=initialized audit_enabled=0 res=1
[    0.075455] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.099874] ACPI: Interpreter enabled
[    0.099874] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.124370] pci 0000:00:05.0: enabling Extended Tags
[    0.159489] SGI XFS with ACLs, security attributes, quota, no debug enabled
[    0.189567] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.267342] IPI shorthand broadcast: enabled
[    0.267403] AES CTR mode by8 optimization enabled
[    6.674725] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    6.711881] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    6.768952] VFIO - User Level meta-driver version: 0.3
[    8.230416] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.17.2g
[    8.241577] ena 0000:00:05.0: ENA device version: 0.10
[    8.242271] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    8.344569] ena 0000:00:05.0: ENA Large LLQ is disabled
[    8.359305] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0500000, mac addr 02:c8:41:8e:6a:f5
[    8.594576] ena 0000:00:05.0 ens5: renamed from eth0
[    9.899613] ena 0000:00:05.0 ens5: Local page cache is disabled for less than 16 channels
[  411.143375] pci 0000:00:06.0: enabling Extended Tags
[  411.147022] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  411.156860] ena 0000:00:06.0: ENA device version: 0.10
[  411.157569] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  411.256845] ena 0000:00:06.0: ENA Large LLQ is disabled
[  411.268807] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:e9:c0:d9:bc:49
[  411.326879] ena 0000:00:06.0 ens6: renamed from eth0
[  411.615469] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[  445.005962] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  445.007231] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device

=== DPDK-RELATED PROCESSES ===
no DPDK processes running

=== END DIAGNOSTICS ===
⚠️ Crash Diagnostics

sender-dmesg-crashes.log

[    0.073173] pid_max: default: 32768 minimum: 301
[    0.126414] iommu: Default domain type: Translated 
[    0.127111] NetLabel:  unlabeled traffic allowed by default
[    0.155434] PCI: CLS 0 bytes, default 64
[    0.210797] nvme nvme0: 2/0/0 default/read/poll queues
[    0.431497] systemd[1]: systemd 252.23-12.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    0.431589] systemd[1]: No hostname configured, using default hostname.
[    0.516834] systemd[1]: Queued start job for default target initrd.target.
[    4.402702] systemd[1]: systemd 252.23-12.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[   15.343141] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[  806.222930] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:15650)
[ 1067.268622] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:20523)
[ 1068.018520] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:20543)
Kernel Console (dmesg)

receiver-console-output.log (PCI/driver events only)

[  587.984696] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  587.986126] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[  618.952872] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:12935)
[  693.454341] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  694.466099] ena 0000:00:06.0: ENA device version: 0.10
[  694.466769] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  694.567847] ena 0000:00:06.0: ENA Large LLQ is disabled
[  694.579200] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:07:69:40:bc:47
[  694.588749] ena 0000:00:06.0 eth0: Local page cache is disabled for less than 16 channels
[  904.900498] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  904.901948] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[  926.188739] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:26592)
[ 1000.722697] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1001.735282] ena 0000:00:06.0: ENA device version: 0.10
[ 1001.736046] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1001.835314] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1001.848253] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:07:69:40:bc:47
[ 1001.915425] ena 0000:00:06.0 ens6: renamed from eth0
[ 1001.956972] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 1028.239041] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1028.240501] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1050.487043] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-echo:31317)
[ 1124.104490] vfio-pci 0000:00:06.0: Relaying device request to user (#0)
[ 1226.652334] vfio-pci 0000:00:06.0: Relaying device request to user (#10)
[ 1329.050962] vfio-pci 0000:00:06.0: Relaying device request to user (#20)
[ 1431.449589] vfio-pci 0000:00:06.0: Relaying device request to user (#30)
[ 1533.848230] vfio-pci 0000:00:06.0: Relaying device request to user (#40)
[ 1636.246864] vfio-pci 0000:00:06.0: Relaying device request to user (#50)
[ 1738.645477] vfio-pci 0000:00:06.0: Relaying device request to user (#60)
[ 1841.044092] vfio-pci 0000:00:06.0: Relaying device request to user (#70)

sender-console-output.log (PCI/driver events only)

[  943.770563] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:18036)
[  945.520556] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:18059)
[  947.280534] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:18082)
[  991.822266] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  992.835067] ena 0000:00:06.0: ENA device version: 0.10
[  992.835838] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  992.937021] ena 0000:00:06.0: ENA Large LLQ is disabled
[  992.950155] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:e9:c0:d9:bc:49
[  993.007456] ena 0000:00:06.0 ens6: renamed from eth0
[  993.059242] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 1022.371152] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1022.372437] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1067.268622] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:20523)
[ 1068.018520] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:20543)
[ 1115.323746] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1116.341479] ena 0000:00:06.0: ENA device version: 0.10
[ 1116.342223] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1116.443375] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1116.457736] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:e9:c0:d9:bc:49
[ 1116.505453] ena 0000:00:06.0 ens6: renamed from eth0
[ 1116.557315] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 1281.097680] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1281.098956] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1437.344062] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1438.357102] ena 0000:00:06.0: ENA device version: 0.10
[ 1438.357998] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1438.459164] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1438.472420] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:e9:c0:d9:bc:49
[ 1438.520233] ena 0000:00:06.0 ens6: renamed from eth0
[ 1438.601920] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

⚠️ TCP Integration Tests (non-blocking) (run)

Branch: 122/merge | Commit: 1d308388

TCP tests run with continue-on-error: true — failures do not block CI.

Test Results

Summary: 0 tests, 0 failures, 0 skipped

Application Logs (last 20 lines each)
Network State

Wire the existing QUIC Tier 1 harness into the shared integration orchestrator,
add QUIC integration and performance CI, and define evidence-based gates for
graduating TCP and QUIC from non-blocking to blocking CI.

Spec includes:
- requirements.md: 10 functional requirements, 4 review rounds
- design.md: orchestrator pseudocode, CI workflow structure, gate runbook
- tasks.md: 19 implementation tasks split into 2 ROADMAP items

Review: 4 Claude opus-5 rounds (SHIPPABLE, 0 CRITICAL / 0 HIGH remaining)
EARS: clean
@gspivey
gspivey force-pushed the feat/quic-tcp-ec2-integration-spec branch from 276be5a to 8394723 Compare October 11, 2026 02:29
@github-actions

Copy link
Copy Markdown

Synthetic Performance Results — Graviton (run)

Commit: a1ebc5d6

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

IPv4 Baseline

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to 64B 11.8M 11.1M 1.1x 90
RX recv_from 64B 3.6M 4.7M 0.8x 212
TX send_to 1400B 1.8M 1.8M 1.0x 551
RX recv_from 1400B 1.1M 1.2M 0.9x 818

IPv6

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to (IPv6) 64B 8.9M 8.3M 1.1x 120
RX recv_from (IPv6) 64B 4.0M 5.4M 0.7x 185
TX send_to (IPv6) 1400B 2.9M 2.9M 1.0x 345
RX recv_from (IPv6) 1400B 1.1M 1.2M 0.9x 824

IPv6 vs IPv4 Comparison (sync path)

Test Payload IPv4 PPS IPv6 PPS IPv4/IPv6 Ratio
TX send_to (sync) 64B 11.8M 8.9M 1.32x
RX recv_from (sync) 64B 3.6M 4.0M 0.90x
TX send_to (sync) 1400B 1.8M 2.9M 0.63x
RX recv_from (sync) 1400B 1.1M 1.1M 0.99x

IPv4 avg sync/async ratio: 0.9x, worst: 1.1x | IPv6 vs IPv4 worst ratio: 1.32x (OK)

OK: IPv6 is 32.5% slower than IPv4 — within acceptable threshold (<50%). Expected due to larger headers (40B vs 20B) and mandatory UDP checksum.

Good: Async wrapper is within 1.1x of sync — minimal framework overhead.

@github-actions

Copy link
Copy Markdown

TCP Synthetic Performance Results (run)

Commit: a1ebc5d6

TCP Synthetic Performance Results

Measures TCP engine framework overhead using a mock PacketBackend.

Test Metric Value Unit
connection_establishment latency_ns 1223 ns
single_stream_throughput throughput_mbps 6181.2 Mbps
engine_tick latency_ns 481 ns

conn_est=1223ns, throughput=6181.2Mbps, tick=481ns

@github-actions

Copy link
Copy Markdown

Synthetic Performance Results (run)

Commit: a1ebc5d6

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to [2001:db8::1]:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

IPv4 Baseline

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to 64B 11.8M 11.2M 1.1x 89
RX recv_from 64B 3.6M 4.7M 0.8x 212
TX send_to 1400B 1.8M 1.8M 1.0x 547
RX recv_from 1400B 1.1M 1.2M 0.9x 829

IPv6

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to (IPv6) 64B 8.9M 8.7M 1.0x 114
RX recv_from (IPv6) 64B 4.0M 5.5M 0.7x 180
TX send_to (IPv6) 1400B 2.9M 2.9M 1.0x 342
RX recv_from (IPv6) 1400B 1.1M 1.2M 0.9x 833

IPv6 vs IPv4 Comparison (sync path)

Test Payload IPv4 PPS IPv6 PPS IPv4/IPv6 Ratio
TX send_to (sync) 64B 11.8M 8.9M 1.32x
RX recv_from (sync) 64B 3.6M 4.0M 0.89x
TX send_to (sync) 1400B 1.8M 2.9M 0.63x
RX recv_from (sync) 1400B 1.1M 1.1M 0.99x

IPv4 avg sync/async ratio: 0.9x, worst: 1.1x | IPv6 vs IPv4 worst ratio: 1.32x (OK)

OK: IPv6 is 32.3% slower than IPv4 — within acceptable threshold (<50%). Expected due to larger headers (40B vs 20B) and mandatory UDP checksum.

Good: Async wrapper is within 1.1x of sync — minimal framework overhead.

@github-actions

Copy link
Copy Markdown

[CI] Stage: Deploy

Infrastructure ready.

  • Sender: i-0e4aa62c3f1b3e085 (DPDK ENI: 10.0.1.26)
  • Receiver: i-0df28cd10244d19d3 (DPDK ENI: 10.0.1.75)
  • Both instances SSM-ready.

@github-actions

Copy link
Copy Markdown

Integration Tests — ❌ 3 real failures

a1ebc5d · run · Integration Tests (Graviton) · 17 tests: 14 ✅ · 3 ❌ real · 0 ⚠️ infra

Tier Test Result Time Reason
tier1-dpdk-echo arp_resolution ✅ pass 0.726s
tier1-dpdk-echo udp_send_receive ✅ pass 1.724s
tier1-dpdk-echo echo_roundtrip ✅ pass 1.521s
tier1-dpdk-echo payload_integrity ✅ pass 0.724s
tier1-dpdk-echo jumbo_diagnostics ✅ pass 0.022s
tier1-dpdk-echo jumbo_echo_8000 ✅ pass 1.731s
tier1-tcp-echo bidir_echo ✅ pass 0.711s
tier1-tcp-echo bidir_multi ✅ pass 0.740s
tier1-tcp-echo bidir_echo ❌ real 60.228s client exited non-zero
tier1-tcp-echo bidir_multi ❌ real 60.228s client exited non-zero
tier2-kernel-interop arp_resolution ✅ pass 0.606s
tier2-kernel-interop udp_send_receive ✅ pass 1.039s
tier2-kernel-interop echo_roundtrip ✅ pass 0.841s
tier2-kernel-interop payload_integrity ✅ pass 0.039s
tier2-tcp-echo kernel_to_dpdk_echo ❌ real 10.013s kernel client exited non-zero
tier3-iperf-interop iperf_sends ✅ pass 1.854s
tier3-iperf-interop our_app_sends ✅ pass 2.532s

❌ Real failures (3) — code/test issues

  • tier1-tcp-echo / bidir_echo — client exited non-zero
    details
TCP Test Client
Target: 10.0.1.75:9000
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'apos;PA'apos;
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Mode: bidir (echo verification, 64B payload)
- **tier1-tcp-echo / bidir_multi** — client exited non-zero
details
TCP Test Client
Target: 10.0.1.75:9000
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'apos;PA'apos;
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Mode: bidir (echo verification, 64B payload)
- **tier2-tcp-echo / kernel_to_dpdk_echo** — kernel client exited non-zero
details
tcp-kernel-client: 20 round-trips of 64B to 10.0.1.75:9000
TCP_KERNEL_FAIL iteration 1/20: connect failed: connection timed out

@github-actions

Copy link
Copy Markdown

Integration Test Failure — Graviton (Run 38105318120)

Branch: 122/merge | Commit: a1ebc5d6cd30850be1fb0ed211973ab1ee84fb71

receiver-user-data.log (7305 bytes, last 80 lines)

   Compiling lazy_static v1.5.1
   Compiling clap_lex v1.1.1
   Compiling clap_builder v4.6.7
   Compiling futures-util v0.3.34
   Compiling clap_derive v4.6.7
   Compiling rand_core v0.5.1
   Compiling ppv-lite86 v0.2.21
   Compiling getrandom v0.4.3
   Compiling hex v0.4.3
   Compiling rand_core v0.10.1
   Compiling rand_chacha v0.2.2
   Compiling clap v4.6.7
   Compiling futures-executor v0.3.34
   Compiling s2n-quic-crypto v0.81.0
   Compiling ring v0.17.14
   Compiling equivalent v1.0.2
   Compiling foldhash v0.2.0
   Compiling deranged v0.5.8
   Compiling time-core v0.1.9
   Compiling allocator-api2 v0.2.21
   Compiling powerfmt v0.2.1
   Compiling num-conv v0.2.2
   Compiling s2n-quic-platform v0.81.0
   Compiling time v0.3.55
   Compiling hashbrown v0.17.1
   Compiling s2n-quic-tls v0.81.0
   Compiling futures v0.3.34
   Compiling rand v0.7.3
   Compiling chacha20 v0.10.2
   Compiling dpdk-stdlib-tcp v0.2.0 (/opt/dpdk-stdlib/dpdk-stdlib-tcp)
   Compiling base64 v0.22.1
   Compiling intrusive-collections v0.10.3
   Compiling untrusted v0.9.0
   Compiling fnv v1.0.7
   Compiling cuckoofilter v0.5.0
   Compiling s2n-quic-transport v0.81.0
   Compiling pem v3.0.6
   Compiling rand v0.10.3
   Compiling s2n-quic-tls-default v0.81.0
   Compiling yasna v0.5.2
   Compiling rustls-pki-types v1.15.1
   Compiling hash_hasher v2.0.4
   Compiling s2n-quic v1.81.0
   Compiling rcgen v0.13.2
   Compiling serde_core v1.0.229
   Compiling dpdk-stdlib-quic v0.2.0 (/opt/dpdk-stdlib/dpdk-stdlib-quic)
   Compiling async-trait v0.1.92
   Compiling zmij v1.0.23
   Compiling dpdk-stdlib-tokio v0.2.0 (/opt/dpdk-stdlib/dpdk-stdlib-tokio)
   Compiling serde_json v1.0.152
   Compiling serde v1.0.229
   Compiling serde_derive v1.0.229
   Compiling itoa v1.0.18
   Compiling tcp-synthetic-bench v0.2.0 (/opt/dpdk-stdlib/apps/tcp-synthetic-bench)
   Compiling synthetic-bench v0.2.0 (/opt/dpdk-stdlib/apps/synthetic-bench)
warning: linker stderr: lto-wrapper: using serial compilation of 10 LTRANS jobs
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `dpdk-stdlib-quic` (bin "quic-echo-server") generated 1 warning
warning: `dpdk-stdlib-quic` (bin "quic-bench") generated 1 warning (1 duplicate)
warning: `dpdk-stdlib-quic` (bin "quic-perf-client") generated 1 warning (1 duplicate)
   Compiling tokio-echo v0.1.0 (/opt/dpdk-stdlib/apps/tokio-echo)
   Compiling test-client v0.1.0 (/opt/dpdk-stdlib/apps/test-client)
   Compiling tokio-tcp-echo v0.1.0 (/opt/dpdk-stdlib/apps/tokio-tcp-echo)
   Compiling tcp-echo v0.1.0 (/opt/dpdk-stdlib/apps/tcp-echo)
   Compiling tcp-test-client v0.1.0 (/opt/dpdk-stdlib/apps/tcp-test-client)
   Compiling plain-tcp-echo v0.2.0 (/opt/dpdk-stdlib/apps/plain-tcp-echo)
   Compiling tcp-kernel-client v0.2.0 (/opt/dpdk-stdlib/apps/tcp-kernel-client)
   Compiling plain-echo v0.1.0 (/opt/dpdk-stdlib/apps/plain-echo)
   Compiling echo v0.1.0 (/opt/dpdk-stdlib/apps/echo)
warning: `dpdk-stdlib-quic` (bin "quic-test-client") generated 1 warning (1 duplicate)
    Finished `release` profile [optimized] target(s) in 2m 59s
=== Build complete ===
-rwxr-xr-x. 2 root root 1470472 Oct 11 02:40 target/release/echo
-rwxr-xr-x. 2 root root 2008448 Oct 11 02:40 target/release/test-client
=== Setup complete! ===
Rust project built successfully
Instance ready for testing
Project location: /opt/dpdk-stdlib

sender-user-data.log (7305 bytes, last 80 lines)

   Compiling lazy_static v1.5.1
   Compiling slab v0.4.12
   Compiling futures-util v0.3.34
   Compiling clap_derive v4.6.7
   Compiling clap_builder v4.6.7
   Compiling rand_core v0.5.1
   Compiling ppv-lite86 v0.2.21
   Compiling rand_core v0.10.1
   Compiling hex v0.4.3
   Compiling getrandom v0.4.3
   Compiling rand_chacha v0.2.2
   Compiling s2n-quic-crypto v0.81.0
   Compiling clap v4.6.7
   Compiling futures-executor v0.3.34
   Compiling ring v0.17.14
   Compiling allocator-api2 v0.2.21
   Compiling deranged v0.5.8
   Compiling foldhash v0.2.0
   Compiling s2n-quic-platform v0.81.0
   Compiling time-core v0.1.9
   Compiling equivalent v1.0.2
   Compiling powerfmt v0.2.1
   Compiling num-conv v0.2.2
   Compiling time v0.3.55
   Compiling hashbrown v0.17.1
   Compiling futures v0.3.34
   Compiling rand v0.7.3
   Compiling chacha20 v0.10.2
   Compiling dpdk-stdlib-tcp v0.2.0 (/opt/dpdk-stdlib/dpdk-stdlib-tcp)
   Compiling untrusted v0.9.0
   Compiling base64 v0.22.1
   Compiling fnv v1.0.7
   Compiling intrusive-collections v0.10.3
   Compiling s2n-quic-transport v0.81.0
   Compiling cuckoofilter v0.5.0
   Compiling pem v3.0.6
   Compiling rand v0.10.3
   Compiling yasna v0.5.2
   Compiling rustls-pki-types v1.15.1
   Compiling hash_hasher v2.0.4
   Compiling rcgen v0.13.2
   Compiling s2n-quic-tls v0.81.0
   Compiling s2n-quic-tls-default v0.81.0
   Compiling s2n-quic v1.81.0
   Compiling serde_core v1.0.229
   Compiling dpdk-stdlib-quic v0.2.0 (/opt/dpdk-stdlib/dpdk-stdlib-quic)
   Compiling async-trait v0.1.92
   Compiling zmij v1.0.23
   Compiling dpdk-stdlib-tokio v0.2.0 (/opt/dpdk-stdlib/dpdk-stdlib-tokio)
   Compiling serde v1.0.229
   Compiling serde_json v1.0.152
   Compiling serde_derive v1.0.229
   Compiling itoa v1.0.18
   Compiling tokio-tcp-echo v0.1.0 (/opt/dpdk-stdlib/apps/tokio-tcp-echo)
   Compiling tcp-synthetic-bench v0.2.0 (/opt/dpdk-stdlib/apps/tcp-synthetic-bench)
   Compiling synthetic-bench v0.2.0 (/opt/dpdk-stdlib/apps/synthetic-bench)
   Compiling tokio-echo v0.1.0 (/opt/dpdk-stdlib/apps/tokio-echo)
   Compiling test-client v0.1.0 (/opt/dpdk-stdlib/apps/test-client)
warning: linker stderr: lto-wrapper: using serial compilation of 10 LTRANS jobs
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `dpdk-stdlib-quic` (bin "quic-test-client") generated 1 warning
warning: `dpdk-stdlib-quic` (bin "quic-bench") generated 1 warning (1 duplicate)
warning: `dpdk-stdlib-quic` (bin "quic-echo-server") generated 1 warning (1 duplicate)
   Compiling tcp-test-client v0.1.0 (/opt/dpdk-stdlib/apps/tcp-test-client)
   Compiling tcp-echo v0.1.0 (/opt/dpdk-stdlib/apps/tcp-echo)
   Compiling plain-tcp-echo v0.2.0 (/opt/dpdk-stdlib/apps/plain-tcp-echo)
   Compiling tcp-kernel-client v0.2.0 (/opt/dpdk-stdlib/apps/tcp-kernel-client)
   Compiling plain-echo v0.1.0 (/opt/dpdk-stdlib/apps/plain-echo)
   Compiling echo v0.1.0 (/opt/dpdk-stdlib/apps/echo)
warning: `dpdk-stdlib-quic` (bin "quic-perf-client") generated 1 warning (1 duplicate)
    Finished `release` profile [optimized] target(s) in 2m 58s
=== Build complete ===
-rwxr-xr-x. 2 root root 1470472 Oct 11 02:40 target/release/echo
-rwxr-xr-x. 2 root root 2008448 Oct 11 02:39 target/release/test-client
=== Setup complete! ===
Rust project built successfully
Instance ready for testing
Project location: /opt/dpdk-stdlib

receiver-console-output.log (45235 bytes, last 80 lines)

[  332.125829] cloud-init[1742]:    Compiling serde v1.0.229
[  332.517843] cloud-init[1742]:    Compiling serde_derive v1.0.229
[  333.567095] cloud-init[1742]:    Compiling itoa v1.0.18
[  335.785465] cloud-init[1742]:    Compiling tcp-synthetic-bench v0.2.0 (/opt/dpdk-stdlib/apps/tcp-synthetic-bench)
[  339.406480] cloud-init[1742]:    Compiling synthetic-bench v0.2.0 (/opt/dpdk-stdlib/apps/synthetic-bench)
[  356.000471] cloud-init[1742]: warning: linker stderr: lto-wrapper: using serial compilation of 10 LTRANS jobs
[  356.002185] cloud-init[1742]:   |
[  356.002302] cloud-init[1742]:   = note: `#[warn(linker_messages)]` on by default
[  356.007347] cloud-init[1742]: warning: `dpdk-stdlib-quic` (bin "quic-echo-server") generated 1 warning
[  374.746519] cloud-init[1742]: warning: `dpdk-stdlib-quic` (bin "quic-bench") generated 1 warning (1 duplicate)
[  376.157837] cloud-init[1742]: warning: `dpdk-stdlib-quic` (bin "quic-perf-client") generated 1 warning (1 duplicate)
[  376.160801] cloud-init[1742]:    Compiling tokio-echo v0.1.0 (/opt/dpdk-stdlib/apps/tokio-echo)
[  380.173789] cloud-init[1742]:    Compiling test-client v0.1.0 (/opt/dpdk-stdlib/apps/test-client)
[  383.808719] cloud-init[1742]:    Compiling tokio-tcp-echo v0.1.0 (/opt/dpdk-stdlib/apps/tokio-tcp-echo)
[  387.308465] cloud-init[1742]:    Compiling tcp-echo v0.1.0 (/opt/dpdk-stdlib/apps/tcp-echo)
[  389.027205] cloud-init[1742]:    Compiling tcp-test-client v0.1.0 (/opt/dpdk-stdlib/apps/tcp-test-client)
[  390.744101] cloud-init[1742]:    Compiling plain-tcp-echo v0.2.0 (/opt/dpdk-stdlib/apps/plain-tcp-echo)
[  391.975682] cloud-init[1742]:    Compiling tcp-kernel-client v0.2.0 (/opt/dpdk-stdlib/apps/tcp-kernel-client)
[  393.376745] cloud-init[1742]:    Compiling plain-echo v0.1.0 (/opt/dpdk-stdlib/apps/plain-echo)
[  394.528303] cloud-init[1742]:    Compiling echo v0.1.0 (/opt/dpdk-stdlib/apps/echo)
[  394.881194] cloud-init[1742]: warning: `dpdk-stdlib-quic` (bin "quic-test-client") generated 1 warning (1 duplicate)
[  395.577995] cloud-init[1742]:     Finished `release` profile [optimized] target(s) in 2m 59s
[  395.664485] cloud-init[1742]: === Build complete ===
[  395.668688] cloud-init[1742]: -rwxr-xr-x. 2 root root 1470472 Oct 11 02:40 target/release/echo
[  395.668786] cloud-init[1742]: -rwxr-xr-x. 2 root root 2008448 Oct 11 02:40 target/release/test-client
[  395.673898] cloud-init[1742]: === Setup complete! ===
[  395.674403] cloud-init[1742]: Rust project built successfully
[  395.674653] cloud-init[1742]: Instance ready for testing
[  395.675407] cloud-init[1742]: Project location: /opt/dpdk-stdlib
ci-info: +++++++++++++++++++++++++++++++++++++++++++++Authorized keys from /home/ec2-user/.ssh/authorized_keys for user ec2-user++++++++++++++++++++++++++++++++++++++++++++++
ci-info: +---------+-------------------------------------------------------------------------------------------------+---------+---------------------------------------------+
ci-info: | Keytype |                                       Fingerprint (sha256)                                      | Options |                   Comment                   |
ci-info: +---------+-------------------------------------------------------------------------------------------------+---------+---------------------------------------------+
ci-info: | ssh-rsa | 4b:c1:40:2b:25:d3:a1:fc:f0:05:5b:66:8d:f3:5b:46:6f:88:ec:6f:f9:da:0c:05:4d:35:64:66:8e:3e:a7:31 |    -    | packer_6ac6255e-58e4-294b-ac49-bb72971a684d |
ci-info: +---------+-------------------------------------------------------------------------------------------------+---------+---------------------------------------------+
<14>Oct 11 02:40:14 cloud-init: #############################################################
<14>Oct 11 02:40:14 cloud-init: -----BEGIN SSH HOST KEY FINGERPRINTS-----
<14>Oct 11 02:40:14 cloud-init: 256 SHA256:m6rDhdOa3Ku5DfhTIxku2+HkOHLfivm8jblFhDsN2N8 root@ip-10-0-1-67.ec2.internal (ECDSA)
<14>Oct 11 02:40:14 cloud-init: 256 SHA256:weKr/OJIkfgSQNlhxbxED6Rv02R2Od5DIxdhLxJ6FvI root@ip-10-0-1-67.ec2.internal (ED25519)
<14>Oct 11 02:40:14 cloud-init: -----END SSH HOST KEY FINGERPRINTS-----
<14>Oct 11 02:40:14 cloud-init: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE4AXj5RU+2rAVV+2kzXNuK9ZuntSqJFj9OYbrOJfO/aVHWMs4wgUTsN2EfIJt17FCfiVv+rVq4OShvKUl7hzBw= root@ip-10-0-1-67.ec2.internal
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILeV8QdJANT3xCp4CoSw5rWzRcw4T/XG2NlHQxPr6psY root@ip-10-0-1-67.ec2.internal
-----END SSH HOST KEY KEYS-----
[  396.030793] cloud-init[1742]: Cloud-init v. 22.2.2 finished at Sun, 11 Oct 2026 02:40:14 +0000. Datasource DataSourceEc2.  Up 396.01 seconds
[  412.056684] pci 0000:00:06.0: [1d0f:ec20] type 00 class 0x020000
[  412.057496] pci 0000:00:06.0: BAR 0 [mem 0x00000000-0x00001fff]
[  412.058191] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00001fff]
[  412.058989] pci 0000:00:06.0: BAR 2 [mem 0x00000000-0x0003ffff pref]
[  412.059957] pci 0000:00:06.0: enabling Extended Tags
[  412.060720] pci 0000:00:06.0: PME# supported from D0 D1 D2 D3hot D3cold
[  412.061676] pci 0000:00:06.0: BAR 2 [mem 0x80080000-0x800bffff pref]: assigned
[  412.062538] pci 0000:00:06.0: BAR 0 [mem 0x8004a000-0x8004bfff]: assigned
[  412.063457] pci 0000:00:06.0: BAR 1 [mem 0x8004c000-0x8004dfff]: assigned
[  412.064516] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  412.076478] ena 0000:00:06.0: ENA device version: 0.10
[  412.077048] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  412.176279] ena 0000:00:06.0: Forcing large headers and decreasing maximum TX queue size to 512
[  412.177361] ena 0000:00:06.0: ENA Large LLQ is enabled
[  412.188088] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem 8004a000, mac addr 02:ff:c2:f8:b8:a3
[  412.246353] ena 0000:00:06.0 ens6: renamed from eth0
[  412.423844] zram_generator::config[8537]: zram0: system has too much memory (3811MB), limit is 800MB, ignoring.
[  412.526799] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[  440.951878] zram_generator::config[9560]: zram0: system has too much memory (3811MB), limit is 800MB, ignoring.
[  441.881683] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  441.882586] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[  588.689694] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:11698)
[  838.441613] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:15666)
[ 1095.603660] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-tcp-echo:19585)
[ 1271.733686] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1272.743242] ena 0000:00:06.0: ENA device version: 0.10
[ 1272.743842] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1272.843048] ena 0000:00:06.0: Forcing large headers and decreasing maximum TX queue size to 512
[ 1272.844113] ena 0000:00:06.0: ENA Large LLQ is enabled
[ 1272.854911] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem 8004a000, mac addr 02:ff:c2:f8:b8:a3
[ 1272.893231] ena 0000:00:06.0 ens6: renamed from eth0
[ 1272.935774] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[ 1272.937756] IPv6: ADDRCONF(NETDEV_CHANGE): ens6: link becomes ready
[ 1273.080081] zram_generator::config[22505]: zram0: system has too much memory (3811MB), limit is 800MB, ignoring.

sender-console-output.log (45697 bytes, last 80 lines)

[  336.405895] cloud-init[1723]:    Compiling itoa v1.0.18
[  338.713611] cloud-init[1723]:    Compiling tokio-tcp-echo v0.1.0 (/opt/dpdk-stdlib/apps/tokio-tcp-echo)
[  338.751727] cloud-init[1723]:    Compiling tcp-synthetic-bench v0.2.0 (/opt/dpdk-stdlib/apps/tcp-synthetic-bench)
[  339.920952] cloud-init[1723]:    Compiling synthetic-bench v0.2.0 (/opt/dpdk-stdlib/apps/synthetic-bench)
[  342.306101] cloud-init[1723]:    Compiling tokio-echo v0.1.0 (/opt/dpdk-stdlib/apps/tokio-echo)
[  342.636054] cloud-init[1723]:    Compiling test-client v0.1.0 (/opt/dpdk-stdlib/apps/test-client)
[  367.303865] cloud-init[1723]: warning: linker stderr: lto-wrapper: using serial compilation of 10 LTRANS jobs
[  367.304516] cloud-init[1723]:   |
[  367.304646] cloud-init[1723]:   = note: `#[warn(linker_messages)]` on by default
[  367.321454] cloud-init[1723]: warning: `dpdk-stdlib-quic` (bin "quic-test-client") generated 1 warning
[  379.967873] cloud-init[1723]: warning: `dpdk-stdlib-quic` (bin "quic-bench") generated 1 warning (1 duplicate)
[  388.681634] cloud-init[1723]: warning: `dpdk-stdlib-quic` (bin "quic-echo-server") generated 1 warning (1 duplicate)
[  388.685985] cloud-init[1723]:    Compiling tcp-test-client v0.1.0 (/opt/dpdk-stdlib/apps/tcp-test-client)
[  390.456026] cloud-init[1723]:    Compiling tcp-echo v0.1.0 (/opt/dpdk-stdlib/apps/tcp-echo)
[  392.224048] cloud-init[1723]:    Compiling plain-tcp-echo v0.2.0 (/opt/dpdk-stdlib/apps/plain-tcp-echo)
[  393.500397] cloud-init[1723]:    Compiling tcp-kernel-client v0.2.0 (/opt/dpdk-stdlib/apps/tcp-kernel-client)
[  394.998239] cloud-init[1723]:    Compiling plain-echo v0.1.0 (/opt/dpdk-stdlib/apps/plain-echo)
[  396.176145] cloud-init[1723]:    Compiling echo v0.1.0 (/opt/dpdk-stdlib/apps/echo)
[  400.457021] cloud-init[1723]: warning: `dpdk-stdlib-quic` (bin "quic-perf-client") generated 1 warning (1 duplicate)
[  400.460146] cloud-init[1723]:     Finished `release` profile [optimized] target(s) in 2m 58s
[  400.614804] cloud-init[1723]: === Build complete ===
[  400.621179] cloud-init[1723]: -rwxr-xr-x. 2 root root 1470472 Oct 11 02:40 target/release/echo
[  400.624311] cloud-init[1723]: -rwxr-xr-x. 2 root root 2008448 Oct 11 02:39 target/release/test-client
[  400.627364] cloud-init[1723]: === Setup complete! ===
[  400.628828] cloud-init[1723]: Rust project built successfully
[  400.630163] cloud-init[1723]: Instance ready for testing
[  400.630208] cloud-init[1723]: Project location: /opt/dpdk-stdlib
ci-info: +++++++++++++++++++++++++++++++++++++++++++++Authorized keys from /home/ec2-user/.ssh/authorized_keys for user ec2-user++++++++++++++++++++++++++++++++++++++++++++++
ci-info: +---------+-------------------------------------------------------------------------------------------------+---------+---------------------------------------------+
ci-info: | Keytype |                                       Fingerprint (sha256)                                      | Options |                   Comment                   |
ci-info: +---------+-------------------------------------------------------------------------------------------------+---------+---------------------------------------------+
ci-info: | ssh-rsa | 4b:c1:40:2b:25:d3:a1:fc:f0:05:5b:66:8d:f3:5b:46:6f:88:ec:6f:f9:da:0c:05:4d:35:64:66:8e:3e:a7:31 |    -    | packer_6ac6255e-58e4-294b-ac49-bb72971a684d |
ci-info: +---------+-------------------------------------------------------------------------------------------------+---------+---------------------------------------------+
<14>Oct 11 02:40:20 cloud-init: #############################################################
<14>Oct 11 02:40:20 cloud-init: -----BEGIN SSH HOST KEY FINGERPRINTS-----
<14>Oct 11 02:40:20 cloud-init: 256 SHA256:jtkxhK/3E/aJxb/2lwAieR1ZEWkcKSwcBOn7KZHo4o8 root@ip-10-0-1-150.ec2.internal (ECDSA)
<14>Oct 11 02:40:20 cloud-init: 256 SHA256:0cVtFIjOc/oiHgcC4FuKbWMeeLAysAChf7N1I0pxrXc root@ip-10-0-1-150.ec2.internal (ED25519)
<14>Oct 11 02:40:20 cloud-init: -----END SSH HOST KEY FINGERPRINTS-----
<14>Oct 11 02:40:20 cloud-init: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAcGUentbgO/CsKG107i+VyG70Z3MFimPIYWFAcbC95kIvcIrrkG9rp8ao7lt8CknOsc/MIdomAToeK1+s+WisM= root@ip-10-0-1-150.ec2.internal
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGntWZLQaxY/kji6AWlyG/z908fbaRgk4Scml+r2Z6bh root@ip-10-0-1-150.ec2.internal
-----END SSH HOST KEY KEYS-----
[  401.156190] cloud-init[1723]: Cloud-init v. 22.2.2 finished at Sun, 11 Oct 2026 02:40:20 +0000. Datasource DataSourceEc2.  Up 401.15 seconds
[  411.848286] pci 0000:00:06.0: [1d0f:ec20] type 00 class 0x020000
[  411.849088] pci 0000:00:06.0: BAR 0 [mem 0x00000000-0x00001fff]
[  411.849794] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00001fff]
[  411.850593] pci 0000:00:06.0: BAR 2 [mem 0x00000000-0x0003ffff pref]
[  411.851561] pci 0000:00:06.0: enabling Extended Tags
[  411.852328] pci 0000:00:06.0: PME# supported from D0 D1 D2 D3hot D3cold
[  411.853302] pci 0000:00:06.0: BAR 2 [mem 0x80080000-0x800bffff pref]: assigned
[  411.854160] pci 0000:00:06.0: BAR 0 [mem 0x8004a000-0x8004bfff]: assigned
[  411.855083] pci 0000:00:06.0: BAR 1 [mem 0x8004c000-0x8004dfff]: assigned
[  411.856167] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  411.865914] ena 0000:00:06.0: ENA device version: 0.10
[  411.866505] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  411.965773] ena 0000:00:06.0: Forcing large headers and decreasing maximum TX queue size to 512
[  411.966866] ena 0000:00:06.0: ENA Large LLQ is enabled
[  411.977952] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem 8004a000, mac addr 02:ff:c1:2e:cd:cb
[  412.025942] ena 0000:00:06.0 ens6: renamed from eth0
[  412.212927] zram_generator::config[7796]: zram0: system has too much memory (3811MB), limit is 800MB, ignoring.
[  412.325564] ena 0000:00:06.0 ens6: Local page cache is disabled for less than 16 channels
[  690.263820] zram_generator::config[13727]: zram0: system has too much memory (3811MB), limit is 800MB, ignoring.
[  691.185299] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  691.186218] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[  852.760113] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:16197)
[  853.489992] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:16217)
[  855.220022] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:16476)
[  856.760006] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:16500)
[  857.519949] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:16560)
[  937.458824] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:17660)
[ 1000.697495] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:18710)
[ 1109.875617] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:20617)
[ 1113.595641] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tcp-test-client:20638)
[ 1200.074093] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:21957)
[ 1205.804001] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:22215)
[ 1222.543830] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:22493)
[ 1239.283524] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:22694)
[ 1392.600723] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:25201)
[ 1436.510019] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:25716)

@github-actions

Copy link
Copy Markdown

❌ Integration Tests Failed — Graviton (run)

Branch: 122/merge | Commit: a1ebc5d6

Test Results

  • tier1-dpdk-echo.xml: 6 tests, 0 failures
  • tier1-tcp-echo-async.xml: 2 tests, 0 failures
  • tier1-tcp-echo.xml: 2 tests, 2 failures
  • tier2-kernel-interop.xml: 4 tests, 0 failures
  • tier2-tcp-echo.xml: 1 tests, 1 failures
  • tier3-iperf-sends.xml: 1 tests, 0 failures
  • tier3-our-app-sends.xml: 1 tests, 0 failures

Application Logs (last 20 lines)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.75:9000 (MAC: 02:ff:c2:f8:b8:a3)
echo listening on 10.0.1.75:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.26:9000 (MAC: 02:ff:c1:2e:cd:cb)
echo listening on 10.0.1.26:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.26:32768 (MAC: 02:ff:c1:2e:cd:cb)
Backend: dpdk
Sending packets...
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.75:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.75:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.75:9000 (expected 8000, OK)
Test complete
[2026-10-11T02:47:58Z] INFO: Jumbo frame echo succeeded: 3/3 responses with correct size
[2026-10-11T02:47:58Z] INFO: JUnit XML written to: /tmp/test-results/tier1-dpdk-echo.xml
[2026-10-11T02:47:58Z] INFO: Tier 1 sender tests complete. Results: /tmp/test-results/tier1-dpdk-echo.xml

@github-actions

Copy link
Copy Markdown

[CI] Stage: Deploy

Infrastructure ready.

  • Sender: i-006b129a6e35a204e (DPDK ENI: 10.0.1.215)
  • Receiver: i-0b724f263d1e62367 (DPDK ENI: 10.0.1.226)
  • Both instances SSM-ready.

@github-actions

Copy link
Copy Markdown

⚠️ TCP Integration Tests (non-blocking) (run)

Branch: 122/merge | Commit: a1ebc5d6

TCP tests run with continue-on-error: true — failures do not block CI.

Test Results

  • tier1-tcp-echo-full.xml: 1 tests, 1 failures, skipped
    FAIL: ENI bind failed on receiver instance
    
  • tier1-tcp-shutdown.xml: 1 tests, 1 failures, skipped
    FAIL: ENI bind failed on receiver instance
    
  • tier2-tcp-flow-control.xml: 1 tests, 1 failures, skipped
    FAIL: ENI bind failed on receiver instance
    
  • tier2-tcp-retransmit.xml: 1 tests, 1 failures, skipped
    FAIL: ENI bind failed on receiver instance
    
  • tier3-tcp-kernel-interop.xml: 1 tests, 1 failures, skipped
    FAIL: ENI bind failed on receiver instance
    

Summary: 5 tests, 5 failures, 0 skipped

Application Logs (last 20 lines each)
Network State

@gspivey
gspivey merged commit 3293456 into development Oct 12, 2026
6 of 9 checks passed
@gspivey
gspivey deleted the feat/quic-tcp-ec2-integration-spec branch October 12, 2026 01:25
gspivey added a commit that referenced this pull request Oct 13, 2026
## ROADMAP items 39–42

Spec for the 3 root-caused TCP integration-test failures that were found
after PR #122 merged.

## Root causes (confirmed against actual code)

**Bug 1 — RFC 6191 TIME_WAIT (engine.rs lines ~1557–1578)**
`handle_time_wait` only handles FIN retransmits; any other packet
including SYN is silently dropped.
Fix: add RST handling (remove TCB), SYN > rcv_nxt handling (new
incarnation), SYN ≤ rcv_nxt (re-ACK to trigger RST→SYN-retransmit
cycle).
**Note:** ISN generator uses per-process random secret (confirmed in
isn.rs), so RFC 6191 ISN comparison alone is ~50/50 — full RST recovery
path is required.

**Bug 2 — Asymmetric routing (Phase C kernel_to_dpdk_echo)**
Kernel client sends from eth0 (management IP); DPDK receiver's SYN-ACK
returns to eth1 subnet; rp_filter=1 drops it.
Fix: add `--bind-ip` to tcp-test-client, pass ETH1_IP in Phase C
harness.

**Bug 3 — vfio-pci bind race (Graviton-specific)**
EAL init fires before /dev/vfio/noiommu-* node appears after
dpdk-devbind.py.
Fix: poll driver symlink + device node with 60s timeout before launching
any DPDK binary.

## SOP exception

Item 39 (known-failures gate) is a prerequisite. Once merged, items
40–42 each remove one entry from `ci/known-failures.txt`. CI is green
when the target test passes AND all remaining listed tests fail as
expected.

## Review history
Spec designed with Claude opus-5.5 + actual code inspection (isn.rs,
engine.rs). Key finding: the ISN coin-flip issue changes the engine fix
from a one-liner to the full RFC 1122 §4.2.2.13 recovery sequence.

<!-- no-visual-delta -->
**Why no screenshot:** spec + ROADMAP only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants