Agent transport: Replace polling with WebSocket push for distributed/read

[sharon-fdm]

User story
As an organization running Fleet at scale,
I want the communication overhead between agents and the server to be minimal,
so that infrastructure costs are significantly reduced while maintaining full functionality.

Goal

Eliminate the empty-poll traffic on distributed/read by letting the server tell agents when there's something to read, instead of every agent asking every 10 seconds.

At 50k hosts, distributed/read is 38.2% of daily requests, and 99.7% come back empty.

Design

The full architectural design, security analysis, deployment plan, and sequence diagrams are in the ADR:

ADR-0011: Agent WebSocket Transport

Summary: Replace polling with a persistent WebSocket connection per agent. The server pushes a "check now" nudge only when there is actual work. Orbit acts as osquery's distributed plugin so osquery's 10s poll becomes a localhost call. The WebSocket is a notification channel only; all data still flows over HTTP.

Phase 1 scope (this issue)

• distributed/read only
• Server: feature flag, WebSocket endpoint, Redis pub/sub wake-up
• Agent: orbit as distributed plugin, WebSocket client with jitter
• Polling stays as permanent fallback

Deliverables

1. A working experiment: patched fleetd + server where no distributed/read requests reach the server while idle, and live queries still work
2. The ADR (linked above) proposing the production design
3. Cost validation from load testing

Sub-tasks (in development order)

Each step is independently testable. See linked sub-issues.

What success looks like

• Zero distributed/read requests reach the server while nothing is happening (verify at the load balancer, not agent logs)
• A live query still completes, with results arriving within ~10 seconds of the "check now" message
• The open connection survives the load balancer's idle timeout (keepalive pings every 5 minutes)
• If the WebSocket can't connect, the agent falls back to polling with no one noticing

Test plan (PR #51427)

https://claude.ai/code/artifact/fe548e97-fe92-43f2-b7d5-f3cf891352aa?org=d01beebc-c6b2-4cd3-85d7-80cc031bf982

Sources: PR #51427 diff, ADR-0011, and the implementation-notes doc QA sections. Priorities: P0 = ship blocker, P1 = before GA, P2 = nice to have.

Setup: two Fleet instances + tooling (expand)

Two instances is the baseline for every scenario (production setup for all customers):

./build/fleet serve --server_address=0.0.0.0:8080 --dev --dev_license --websocket_transport_enabled=true --logging_debug
./build/fleet serve --server_address=0.0.0.0:8082 --dev --dev_license --websocket_transport_enabled=true --logging_debug

# Dashboards (one per instance)
go run ./tools/agentws-viz -server https://localhost:8080 -token $TEST_TOKEN -addr 127.0.0.1:3001 -insecure
go run ./tools/agentws-viz -server https://localhost:8082 -token $TEST_TOKEN -addr 127.0.0.1:3002 -insecure

# Build orbit into the local TUF repo for real-device runs
./tools/tuf/test/build_and_push_orbit.sh

# Simulated fleet at scale
go run ./cmd/osquery-perf -host_count 500 -websocket_prob 1.0 ...

Config knobs: websocket.transport_enabled (default false), websocket.check_interval (30s, drives refetch/re-notify latency), websocket.check_batch_size (500), websocket.ping_interval (5m), websocket.pong_timeout (30s).

Feature off (default) — regression

• P0 Full smoke test with flag unset (enroll, live queries, labels, policies, refetch, software, scripts): identical to previous release; osqueryd polls distributed/read every 10s as before.
• P0 Orbit config response has no websocket_transport directive; no GET /api/fleet/orbit/notifications traffic; orbit never opens a WS connection.
• P1 Agent options served unmodified: distributed_plugin: tls is NOT stripped from served config when the flag is off.

Feature on — happy path

• P0 Enable flag with an already-enrolled polling host: orbit gets the directive, persists websocket_transport.json, restarts once, connects (visible on agentws-viz).
• P0 Active distributed plugin is fleet_orbit_distributed (SELECT * FROM osquery_flags WHERE name='distributed_plugin'); osqueryd launched with --extensions_require=com.fleetdm.orbit.osquery_extension.v1 --extensions_timeout=60; no /api/v1/osquery/distributed/read polling in server logs (that path = built-in tls plugin; orbit's client uses /api/osquery/).
• P0 Config-refresh flip-back guard: wait several config_tls_refresh cycles connected — host never reverts to tls polling (distributed_plugin stripped from served options). This regressed twice during development.
• P0 Refetch a connected host: delivered within one check_interval (≤30s; slower than old ≤10s — known/accepted).
• P0 Interval work on schedule: labels, policies, and host vitals refresh every ~1h on all connected hosts, none starve.
• P1 Idle connection longevity: hosts connected+idle for hours (many 5m ping cycles), incl. laptop sleep/wake — connection survives or cleanly reconnects; no server goroutine/memory growth.
• P1 Brand-new fleetd install with flag already on: enroll → directive → single toggle-restart → connected; no restart loop.

Live queries (highest-risk area — test all APIs, not just UI)

• P0 UI live query "All hosts": results within ~1–2s. (Caveat: "All hosts" resolves via label membership — a fresh host missing the builtin label is silently untargeted; use direct host-ID targeting to distinguish that from a transport bug.)
• P0 Direct host targeting: POST /api/latest/fleet/hosts/{id}/query.
• P0 Remaining APIs: fleetctl query, run saved report (/api/latest/fleet/queries/{id}/run), UI websocket campaign API — against both instances.
• P0 Cache-race soak: ~20 consecutive campaigns back-to-back — every one returns results from every targeted host (delayed notifier must beat the 1s liveQueryMemCacheDuration race). Any silent empty campaign is a failure.
• P0 Sweep backstop: cut host network → start campaign → restore network while campaign open — host answers within ~30s of reconnect. Also: restart Redis between create and answer (lost pub/sub message recovered by sweep).
• P1 Re-notify stops after answer (no repeated nudges once responded); untargeted connected hosts never woken by someone else's campaign.
• P1 Query that crashes osqueryd (e.g. windows_updates in Windows arm64 hosts Querying windows_updates table crashes fleetd on Windows ARM64 (go-ole GetVariantDate ABI bug) #51569): orbit restarts osqueryd; in-flight WS pass closes (no wedged awaiting-write); host answers later campaigns.
• P2 Stop a campaign early: nudging stops; no orphaned livequery:host:{id} Redis targeting.

Failure & recovery

• P0 Server closes connection (restart one instance): orbit resumes distributed polling while disconnected and reconnects with jitter/backoff. Confirm there's no dead window where the host neither polls nor is connected (open question from notes doc).
• P0 Server fully down minutes then back (tunnel up; separately tunnel/ALB down too): backoff, no crash/log-storm; on recovery all hosts reconnect, no poll storm.
• P0 Network allows HTTPS but blocks WS upgrades: orbit permanently falls back to polling, everything works at pre-WS behavior; reconnect attempts don't spam.
• P1 Half-open connection (packet drop without FIN, both directions): server reaps within ping_interval + pong_timeout; orbit read deadline fires; no "connected but deaf" host beyond the keepalive window.
• P1 Random disconnect/packet-loss churn (toxiproxy) 30+ min under trickle of campaigns: no missed interval work; campaigns recovered via sweep; no hub goroutine leak.
• P0 Agent process failures: kill -9 osqueryd; SIGSTOP osqueryd (hang); kill orbit — recovery in all cases, no stuck distributed pass.
• P1 Install-time races: host offline during fleetd install; server down during orbit first start — enrolls/connects when connectivity returns.
• P1 Delete a host in Fleet with an active WS connection: server drops conn; orbit no tight reconnect loop; normal deleted-host re-enroll behavior.

Auth & security

• P0 WS endpoint auth (GET /api/fleet/orbit/notifications): missing/malformed/invalid node keys → 401 before upgrade, no connection held; valid orbit key succeeds.
• P0 Node-key fallback scope: orbit key works on distributed/read+write; osquery keys keep working everywhere; verify orbit key behavior on other osquery endpoints (config, log) matches intent — fallback grants nothing beyond what an enrolled host had. (Candidate for security-auditor pass.)
• P1 /debug/agentws requires the debug auth token.
• P1 TLS variants: via ngrok/ALB, no-TLS direct dev, mTLS setup, hosts with HTTP signature certs (Linux TPM).

ABM and Autopilot enrollment with feature enabled

• Websockets working a-ok.
• Websockets blocked by NATs, or so.

Host online status

• P0 Connected-but-idle host stays Online for hours (record-seen loop bumps seen-time ~every 30s while connected; no offline flapping).
• P0 Disconnect a host: flips Offline after the normal window — not sooner, not never.

Multi-instance & Redis

• P0 Cross-instance notify: host on instance A, live query created via instance B (and vice versa) — near-instant results. redis-cli PUBSUB NUMSUB agent_notifications equals live instance count (watch for zombie fleet serve processes double-subscribing).
• P0 Kill instance A mid-campaign: hosts reconnect to B; unanswered targeted hosts picked up by B's sweep within ~30s.
• P0 Redis restart with instances/hosts up: process-wide subscriber reconnects and resubscribes (NUMSUB recovers); live queries work after, no server restart.
• P1 Redis cluster mode: pub/sub + live query store work; primary failover doesn't permanently kill notifications.
• P2 Both viz dashboards: disjoint host sets summing to all connected.

Toggle & rollback (the escape hatch — must be bulletproof)

• P0 Disable flag on a running deployment: hosts toggle-restart back to polling on next config fetch; distributed_plugin no longer stripped so osqueryd reverts to tls; all hosts healthy with no manual intervention.
• P0 Roll server back to previous release (no websocket config) with hosts WS-connected: directive absent → orbit reverts to polling; full recovery.
• P1 Rapid flag flapping (on→off→on): websocket_transport.json consistent; exactly one restart per change; no restart loop.

Backwards / forwards compatibility

• P0 Plain osquery hosts, flag on and off: unaffected (keep tls plugin from their command line even when served options omit it).
• P0 Latest released fleetd vs new server, flag on and off: old orbit ignores the directive, keeps polling. Mixed fleet at scale via osquery-perf -websocket_prob 0.5.
• P0 New fleetd vs latest released server (Must rule) + auto-update from released orbit via local TUF: runs fine with no directive; updated agent connects once pointed at a flag-on server.

Existing orbit functionality (while WS-connected)

• P0 Software installation and script execution end to end.
• P0 Custom extensions: agent options with customer --extensions_require/--extensions_timeout — orbit merges requires (theirs + orbit's) and takes max timeout; both extensions load.
• P1 Remote flag change via flag_runner: osqueryd restarts with new flags; WS-managed flags survive; connection re-established.
• P2 Fleet Desktop, host vitals, MDM check-ins unchanged.

Platform matrix (real devices; core loop = enroll → connect → live query → refetch → toggle off)

• P0 macOS Apple Silicon (TLS via ngrok)
• P0 macOS Intel
• P0 Linux amd64 (incl. an http-sig/TPM host if available)
• P0 Windows amd64 (orbit toggle-restart under the Windows service manager)
• P1 No-TLS direct-to-Fleet dev setup (ws:// path)

Server config validation

• P1 Zero/negative ping_interval/pong_timeout/check_interval/check_batch_size: server refuses to start with a clear error naming the key.
• P2 Non-default tuning smoke (check_interval=5s, check_batch_size=10, short ping/pong): latencies scale; batching visible with >batch-size hosts.

Scale & load (osquery-perf)

• P0 5k+ simulated hosts, -websocket_prob 1.0, both instances, 1h steady state: memory/CPU/goroutines stable; sim stats show connected ≈ host count, errors ≈ 0.
• P0 Reconnect herd: restart one instance with thousands connected — jitter/backoff spreads reconnects; DB/Redis don't spike to poll-storm levels.
• P1 Campaign against all N hosts, incl. during enroll churn: ~100% answer rate (coalescing fix — no "connected but not responding" hosts).
• P1 Load comparison, same host count, flag off vs on: distributed/read rate drops dramatically; interval-checker cost bounded by batch size; per-host GETBIT probes every 30s while a campaign is active is the known cost (documented for later optimization).

Observability & tooling

• P1 /debug/agentws payload sanity: connected hosts, next_check_in_ms, check_interval_seconds, byte counters match reality.
• P2 agentws-viz against both instances during the failure scenarios: connect/disconnect events render; next-sync countdown is skew-free.
• P2 Log hygiene at info level during connect/disconnect churn: no per-message spam; errors actionable.

Automated suites

• P0 Unit: go test ./server/agentws/... ./server/config/... ./orbit/pkg/wstransport/... ./client/... ./cmd/osquery-perf/...
• P0 Service + pub/sub: MYSQL_TEST=1 REDIS_TEST=1 go test ./server/service/ and REDIS_TEST=1 go test ./server/pubsub/...
• P1 Race detector on new packages: go test -race ./server/agentws/... ./server/pubsub/... ./orbit/pkg/wstransport/... (only known-benign subscriber-shutdown finding allowed).

[sharon-fdm]

@noahtalerman, @rachaelshaw, this feature is designed to be off by default and have full backward compatibility even when turned on.
Compatibility will be both ways: Agent <--> Server.
However, it represents significant changes to the communication when turned on.

Please go over the ADR for details.
I can do a guided tour in one of our standups if that is required.

TMWYT