Skip to content

feat(events): events.stream push transport and a main-actor-free even… - #975

Open
Rulexec wants to merge 1 commit into
umputun:masterfrom
Rulexec:events-stream-push
Open

Rulexec wants to merge 1 commit into
umputun:masterfrom
Rulexec:events-stream-push

Conversation

@Rulexec

@Rulexec Rulexec commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

Not merge-ready. This PR is a demonstration opened for feedback on the direction; the real PR follows once the approach is agreed. The old transport stays fully in place and remains the documented fallback, so comparing the two is one --poll flag away, and the bench script drives both.

the daily-driver instance sat at a constant visible CPU burn while doing nothing. sample pinned it on agtermctl events subscribers: the CLI short-polls events.read at 4 Hz, one connection per request, and every request paid a main-actor dispatch plus a refreshWindowCache() rebuild. Seven subscribers — opencode plugins, an oc-spawn coordinator, a couple of long-lived monitors — meant ~28 connections a second for nothing, and that was the whole idle load. The consumers were fine; the transport was the problem.

Two changes, together:

events.stream — a push transport

  • one connection: the reply is the ordinary events.read anchor batch, then every sequenced event arrives as one NDJSON line. No polling, no reconnects while the app lives.
  • admission reserves a hub slot before the anchor read, so an event landing inside the open window is buffered and flushed past the anchor cursor at commit — never lost to the open sequence. That gap was real: the first implementation read the anchor first and a burst during the handshake silently dropped frames.
  • the fan-out encodes each event once on the main actor and queues the frame on every matching subscriber; a peer that stops reading is dropped, never waited on — the same slow-peer policy as the other adopted streams. Cap 32 concurrent streams; the connection past it gets EventStreamHub.capacityRefusal and the CLI degrades to polling, loudly: one stderr notice, stdout stays event lines only.
  • agtermctl events is stream-first. A server without the command — an old app, agterm-server, a remote window — falls back to events.read polling silently and permanently for the run; an error reply carrying an events payload is a cursor verdict exactly as polling surfaces it. --poll forces the old transport from the start.
  • EOF is the unsubscribe; input after the request means nothing. A dropped stream reconnects through the same refused-connect retry budget polling already had, resuming from the last frame's cursor.

A cheap events.read

  • polling itself also had to stop costing a main-actor hop per request. The ring now exposes a lock-guarded tail snapshot, and the accept thread answers the bootstrap anchor, the at-tail empty page and the three cursor errors with zero main-actor involvement — byte-identical to the dispatched answer, pinned by a hosted parity test. A cursor behind the tail may have matching entries, which only the ring knows, so that one still goes to the main actor.
  • both events commands are exempt from the post-dispatch refreshWindowCache(): nothing they read can change a window node, and they are polled continuously.
  • one parse (ControlEventRequest.parse) feeds the anchor dispatch, the fast path and stream admission, so validation cannot drift between the three.

Old CLIs, remote windows and agterm-server keep polling unchanged; the events.read wire contract did not move. control-api.md, the bundled skill, reference.md and site/commands.html are updated in step.

Why this is faster

  • 50 pollers at 4 Hz is 200 accepts, parses, dispatches and window-cache rebuilds a second. 50 streams is 50 parked writer threads and zero work per event until one arrives.
  • the fast path removes the main-actor serialization from the steady poll loop, so even the fallback transport gets cheaper — and the accept thread stays free for real commands.

Two-minute bench (macOS 26.6.2, Apple silicon, Debug build; full output in /tmp/agterm-events-bench-*.md):

  • setup per phase: a fresh isolated app instance, 10 workspaces × 5 sessions, one agtermctl events --json listener typed into every session — 50 listeners running inside their panes exactly as a user starts them
  • steady = 120 s mean app CPU with all listeners attached; burst = 150 events (a status flip pair and a msg.send per session) fired over the socket
transport idle cpu % steady cpu % (120 s) burst cpu % tree p50/p95 ms burst req p50/p95 ms failed reqs listeners complete / frames
poll (old) 0.5 8.2 5.9 24.4/76.0 5.8/9.9 0 50/50, 7500 frames
stream (new) 0.2 3.7 2.1 12.5/36.4 2.2/5.0 0 50/50, 7500 frames

Read it as: sustained cost halved, and the burst, tree and request latency all follow at 2–3×. Delivery identical. Both phases run the same build — the poll side is already served by the new fast path, so the pre-PR app sits well above that 8.2%. Steady numbers were stable across runs; treat the burst column as relative, not absolute: on a warm machine with two minutes of poll traffic queued behind it, an earlier run measured the poll burst as high as 67.8% against 34.0% — the same ratio, worse absolute numbers.

Run the bench yourself

./scripts/bench-events.sh

It launches its own isolated Debug instances (state under /tmp/agterm-bench-events, stopped by pid), types the listeners into panes via session.type, fires the burst, samples CPU and latency, and writes a markdown report to /tmp/agterm-events-bench-<ts>.md. Takes about five minutes. BENCH_WORKSPACES, BENCH_SESSIONS_PER_WS, BENCH_STEADY_SECONDS and BENCH_APP scale it. It never touches the live ~/Applications app or the default socket.

Gates

  • swift test: 773 + 5013 + 78 + 264 tests, zero failures (one pre-existing known issue)
  • hosted agtermTests/ControlServerTests: 26/26, including fast-path byte parity, stream adoption end-to-end and the cap refusal
  • agtermUITests/ControlEventsStreamUITests: the real app plus the bundled CLI receiving a pushed frame
  • swiftlint --strict: zero findings

Open questions for the real PR

  • 32 concurrent streams per app — enough, or should the cap scale?
  • session wait still polls; moving it onto the stream was deliberately left out of this demonstration.

…ts.read fast path

idle cpu burn traced to agtermctl events subscribers: the cli short-polls events.read
at 4 hz, one connection per request, and every request paid a main-actor dispatch plus
a refreshWindowCache rebuild. seven subscribers meant ~28 connections a second for
nothing.

events.stream answers that with one connection: the reply is the ordinary events.read
anchor batch, then one ndjson frame per sequenced event. admission reserves the hub
slot before the anchor read, so an event in the open window is buffered and flushed
past the anchor cursor at commit; fanout encodes once and queues to every matching
subscriber; a stalled peer is dropped, never waited on; eof is the unsubscribe; cap 32
with EventStreamHub.capacityRefusal, which the cli answers by degrading to polling with
one stderr notice. the cli is stream-first, falls back to polling for servers without
the command, and --poll forces the old transport.

events.read itself also stops costing a main-actor hop per poll: a lock-guarded ring
tail snapshot lets the accept thread answer the bootstrap anchor, the at-tail page and
the three cursor errors byte-identically, and both events commands skip the window
cache refresh. one ControlEventRequest.parse feeds the anchor dispatch, the fast path
and stream admission so validation cannot drift.

scripts/bench-events.sh benches both transports on isolated instances with listeners
typed into panes: 50 listeners over 10 workspaces x 5 sessions, 120 s steady window,
poll 7.7% vs stream 3.6% app cpu, 150-event burst delivered to 50/50 listeners on both.
@Rulexec
Rulexec requested a review from umputun as a code owner November 12, 2026 10:56
@umputun

umputun commented Nov 12, 2026

Copy link
Copy Markdown
Owner

the problem is real, but the baseline moved. #955 (in v0.42.0) already made an idle events.read start at the cursor and skip the window-cache rebuild, which covers most of the cost the PR body describes. Pls re-measure against current master first.

the events.read fast path on the accept thread looks worth having on its own. It needs no protocol change, and old CLIs and session wait get it for free. I'd take it as a separate, focused PR with numbers for master vs fast path.

I'm not convinced about events.stream yet. It adds a second transport with its own cap, fallback and reconnect rules, while session wait and msg read --wait would still poll, so we'd have two models instead of one. The bench shows push beating fast-path polling, but with 50 listeners against a cap of 32, so part of the stream row is still polling, and master isn't measured at all. What do your seven subscribers actually need? Is the polling delay of up to 250 ms a visible problem anywhere, or is it CPU only?

for now I prefer to keep the existing command model. If the remaining polling cost still matters after the fast path, there is a backlog item about long-polling events.read (docs/backlog/control-followers-poll-instead-of-long-polling.md), which would keep one command and one cursor model. Push stays open if a master vs fast path vs push comparison at your workload shows it's needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants