Skip to content

fix(sensing-server): handle WebSocket Lagged + add ping keepalive#484

Merged
ruvnet merged 1 commit into
ruvnet:mainfrom
schwarztim:pr/websocket-keepalive-fix
May 17, 2026
Merged

fix(sensing-server): handle WebSocket Lagged + add ping keepalive#484
ruvnet merged 1 commit into
ruvnet:mainfrom
schwarztim:pr/websocket-keepalive-fix

Conversation

@schwarztim

Copy link
Copy Markdown
Contributor

Motivation

Dashboard WebSocket clients were rapid-cycling (connect/disconnect every 2-4s) under load. Root cause: the broadcast channel raises RecvError::Lagged when a slow consumer falls behind the 256-frame buffer (10 Hz ingest × 50-200 KB frames makes this easy to hit), which the handler treated as a fatal error and closed the connection. Long idle periods also let proxies time out the connection.

Changes

  • v2/crates/wifi-densepose-sensing-server/src/main.rs: catch RecvError::Lagged in both handle_ws_client and handle_ws_pose_client and continue (drop missed messages, log debug); add 30s ping keepalive on the sensing handler.

Test results

  • cargo test --workspace --no-default-features: 1621 passed, 0 failed
  • python v1/data/proof/verify.py: pre-existing hash mismatch on this environment (numpy/scipy version); not caused by this change (confirmed identical failure on unmodified origin/main)
  • Manual: dashboard maintains connection through sustained UDP ingest at 30Hz

Notes

  • No new dependencies, no API changes, no ADR needed.
  • Cherry-picked from fork commit 581daf4f with automatic path adjustment (rust-port/v2/).

Root cause: broadcast channel Lagged error caused instant disconnect
when clients fell behind 256 frames (10Hz * 50-200KB = easy to lag).
Client reconnects, immediately lags again, rapid cycling ensues.

Sensing handler: Lagged error now continues (skips missed frames)
instead of breaking. Added 30s ping interval for proxy keepalive.
Pose handler: same Lagged handling + Pong match arm.

CHANGELOG updated under Unreleased/Fixed.

@ruvnet ruvnet left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two correct fixes in one small PR:

  1. RecvError::Lagged continues instead of breakingtokio::sync::broadcast returns Lagged(n) when the receiver fell behind the channel capacity (256 frames). Previously this was treated as fatal so the client disconnected, immediately reconnected, fell behind again at the same 10 Hz ingest rate, and rapid-cycled every 2–4 s. The fix drops the missed frames and logs at debug level, which is the right behaviour for a 'live stream' WebSocket (no point delivering stale data).
  2. 30 s ping keepalive on handle_ws_clientset_missed_tick_behavior(Skip) is the right configuration so a stalled handler doesn't queue up a burst of pings when it resumes.

CHANGELOG entry is well-written. Merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants