Skip to content

Durable-by-design delivery on bad networks - #168

Merged
wingleeio merged 9 commits into
mainfrom
zeron/resilient-delivery-on-bad-networks
Aug 19, 2026
Merged

wingleeio merged 9 commits into
mainfrom
zeron/resilient-delivery-on-bad-networks

Conversation

@wingleeio

@wingleeio wingleeio commented Aug 19, 2026 •

Copy link
Copy Markdown
Contributor

Implements the full durable-by-design plan — a send is a local write that always either delivers exactly once or visibly says why not, and recovery is event-driven, never timer luck — plus the three priority asks from the Aug 19 zombie-peer-link incident.

Phase 2 — event-driven recovery (ed8cd60)

  • Online-bus subscriptions for the previously wake-only supervisors (registry/chat2 initial joins, LinkCache cooldowns, auth refresh retry) — a room stuck in its FIRST join now un-parks on any sibling dial success.
  • macOS NWPathMonitor (crates/sync/src/net_path.rs) → wake::set_path_online: offline→online broadcasts the online event; while offline, backoff waits park on the event buses (30s safety recheck) instead of burning dials.
  • Window-focus fast path: ProbeSync also probes {edge}/health (3s); success fires the online event so a focus lands a redial in ~1 RTT, while a failed probe leaves backoffs parked.
  • Stability-gated backoff reset (a join must survive 30s — reset-on-join let connect-and-die sockets hot-loop at 250ms forever); caps 30s→16s; device-room silence lease 40s→25s with 10s pings.
  • Reconnect posture is observable: retry_at_ms + sticky last_failure + numbered dial attempts in logs.

Incident asks (Aug 19, done first)

Attachments ride the durable queue (8f3bce8) — staging was a blocking pre-step in front of QueueCommand, so a send with an image died with the dead link. Now: bytes commit to the LOCAL uploads dir, the command queues immediately with pending://{uploadId}/{name} refs, and the engine pushes bytes to the host afterwards (retry-forever on the online bus; chunk/commit timeouts invalidate the link). The host drain defers a command whose refs aren't on disk — BEFORE the processed mark — and UploadCommit kicks drains; at dispatch the refs rewrite to absolute paths so the persisted transcript keeps the legacy shape. Version-gated on both the local engine and the host (>= 0.2.12); old peers keep the legacy upload, now bounded by a 300s total budget. Also fixes the incident's silent-wedge mechanics: interrupt/answer get their own task slot (assigning send_task dropped an in-flight send with no banner and a permanently stuck "Sending…"), failure banners are chat-scoped and survive navigation, chunk retries log at warn, MUTATE/QUEUE_COMMAND are deadline-bounded.

Peer-link end-to-end liveness (449a1db) — the transport lease counts the DO's auto-pong, which answers from the EDGE, so a healthy client↔edge leg masked a dead edge↔host leg for 6 minutes. The client now rides an echo frame on every keepalive and the HOST echoes it back; 20s of echo silence = zombie path → drop and redial. Feature-detected (old hosts keep transport-lease behavior; inbound RPC frames count as proof too), no edge changes (the DO relays kinds untouched). DeviceLink::connect bounded at 15s — the one unbounded await under forward().

Registry-dark dial parking (449a1db) — LinkCacheConfig.liveness: a Dark verdict fails peer calls fast with ZERO dials (was: 3-dial bursts every ~60s to a device offline for days). Dark requires positive evidence (registry room joined + past a 60s warm-up, freshest heartbeat older than 5min); registry-down and every ambiguity stay Unknown, so the rows-down/relay-up incident shape can never park the relay. Presence returning un-parks and clears the cooldown in the same beat.

Phase 1 — UI truth (b98d698)

WatchConnectivity stream (engine pushes Disabled/Offline/Reconnecting{retryAtMs,lastFailure}/Connected + per-open-chat room state, emitted only on change) → sidebar connection pill (visible only while degraded; live retry countdown; last failure stays visible through the next attempt), composer "messages will queue and send automatically" notice, amber Queued row pill + trailer instead of a fake Working spinner, and the pending overlay HOLDS while degraded instead of silently expiring at 30s with no trace. Locally-hosted chats never read as degraded — a queued command executes offline.

Phase 3 — second road + chaos suite (9055e01)

  • RelayCommand: when rows can't reach the edge (10s grace) but the peer link is alive, the sender's engine forwards the queued entry itself over the device-room link. The host evaluates it against its own doc, claims the client-minted id in the processed ledger BEFORE executing — so the doc row arriving later over chat2 sync dedupes to a no-op. Exactly-once across both roads by construction, proven in relay_delivery.rs (execute via relay, then simulated row arrival + duplicate relay → still one turn). Version-gated; relay timeouts mark the link suspect.
  • Deterministic flaky-network suite (chat_client/tests.rs, virtual clock): scripted drop/refuse connector asserting the durable contract — a mid-push socket death + refused dials deliver the batch exactly once AND surface Disconnected (silence is a test failure); connect-and-die sessions grow backoff until a stable session resets it; OS-offline parks dials entirely and the online event un-parks within a beat. Plus the zombie-relay echo regression (device_room.rs tests) and the queued-attachment defer/execute cycle (queued_attachments.rs).

Phase 4 — failed state + canonical timestamps (d4d4045)

Two-minute grace on unadopted sends → explicit Failed row pill and a transcript "Not delivered — click to retry" affordance wired to RetryDelivery (fresh chat2 socket, host nudge, drain pass, fresh delivery escorts with transfers re-derived from the pending refs). Timestamp canonicalization: the executor pre-writes the user message at the entry's issued_at (clamped, idempotent by id), so a send queued for minutes lands in history where the user sent it.

Notes for review

  • Version gates assume this releases as 0.2.12 (QUEUED_ATTACHMENTS_MIN, RELAY_MIN_VERSION) — adjust if the release number differs.
  • net_path.rs is cfg(target_os = "macos") FFI written on a Linux box — needs one mac build check.
  • 685 tests green across sync/rpc/engine/ui (--features mock-server for the sync integration tier). The live half of the acceptance matrix (airplane wifi 20% loss, captive portal, flap-every-5s) still wants a real-network pass; the deterministic analogues are in the suite.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

wing-anara and others added 9 commits August 19, 2026 06:08
…tighter caps

- macOS NWPathMonitor (net_path.rs) feeds a new wake::set_path_online:
  offline→online broadcasts the online event so every parked backoff
  redials the instant the path returns; while offline, backoff waits park
  on the event buses (30s safety re-check) instead of burning dials.
- The initial-join supervisors (registry room, chat2 join), the LinkCache
  cooldowns, and the auth refresh retry now subscribe to the online bus —
  previously wake-only, so a room stuck in its FIRST join was never
  un-parked by a sibling dial success.
- Window-focus fast path: ProbeSync now also probes {edge}/health (3s);
  success fires the online event, so a focus lands a redial in ~1 RTT
  while a failed probe leaves the backoffs parked.
- Stability-gated backoff reset: a join must survive 30s before a
  disconnect resets to base — reset-on-join let a connect-and-die socket
  hot-loop at 250ms forever.
- Constants: room BACKOFF_CAP + engine JOIN_RETRY_CAP 30s→16s; device-room
  SILENCE_LEASE 40s→25s with pings at 10s (one lost pong tolerated).
- Reconnect posture is now observable: registry rooms track retry_at_ms +
  sticky last_failure (cleared only on rejoin) + numbered dial attempts in
  logs; chat2 dials are numbered too.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… in drain

2026-08-19 incident: attachment staging was a blocking pre-step in FRONT of
QueueCommand, so a send with an image died with a zombie peer link instead of
queueing (user row never written; recovery had nothing to resume).

- pending://{uploadId}/{fileName} refs: the composer commits bytes to the
  LOCAL engine's uploads dir (fast, offline-safe), queues the command
  immediately with refs, and the ENGINE pushes bytes to a remote host after
  (retry until landed, event-driven backoff on the online bus; chunk/commit
  timeouts invalidate the peer link so retries dial fresh pipes).
- Host drain defers a Run/Steer whose refs aren't on disk yet — BEFORE the
  processed mark, so it stays eligible; UploadCommit kicks all drains
  (event-driven), a 30s recheck loop covers the bounded 15min give-up, after
  which the command resolves Rejected instead of wedging the chat queue.
- At dispatch the host rewrites refs to absolute paths (attachments list +
  prompt), so the harness and the persisted transcript see the legacy shape;
  ReadAttachmentChunk resolves pending refs host-side for cache misses.
- Version-gated on BOTH the local engine and the host device's stamped
  registry version (>= 0.2.12); older peers keep the legacy blocking upload,
  now bounded by a 300s total budget (per-chunk deadlines alone let a
  degraded relay grind silently for the whole ~560-chunk count).

Silent-wedge fixes from the incident review:
- interrupt/answer get their own task slot — assigning send_task DROPPED an
  in-flight send future (no banner, no cleanup, 'Sending…' stuck forever).
- failure banners are chat-scoped and survive navigation (blanket clear on
  chat switch erased the only trace of a failed send).
- upload chunk retries log at warn (were debug — dropped by the info filter,
  hence the empty incident log); MUTATE + QUEUE_COMMAND deadline-bounded.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Aug 19 incident: a ~5s wifi blip left the device-room peer link silently
dead for 6 minutes — the transport lease counts the DO's auto-pong, which
answers from the EDGE, so a healthy client↔edge leg masks a dead edge↔host
leg while the link retries frames into the void.

- App-level echo: the client rides an 'echo' frame on every keepalive; the
  HOST echoes it back (edge DO relays kinds untouched — no edge change). No
  echo for 20s on a link that HAS echoed = zombie path → drop and redial.
  Feature-detected (old hosts never echo → transport-lease-only behavior)
  and inbound RPC frames count as end-to-end proof too. Regression test:
  FakeRelay blackhole keeps the host registered while client→host frames
  vanish; the link must die with 'host echo silent' (compressed test clocks
  via a doc(hidden) seam).
- DeviceLink::connect bounded at 15s — it was the one unbounded await under
  forward(), so a wedged edge socket hung relayed RPCs indefinitely.
- Registry-dark dial gate: LinkCacheConfig.liveness probe; a Dark verdict
  fails the call fast with ZERO dials (was: 3-dial bursts every ~60s to a
  device offline for 6 days, for the life of the app). Dark needs positive
  evidence — registry room joined + past a 60s warm-up, freshest heartbeat
  (live map ∪ session cache ∪ durable row) older than 5min; registry-down
  and every ambiguity stay Unknown so the 03:45 rows-down/relay-up shape
  can never park the relay. Presence returning un-parks and clears the
  cooldown in the same beat (existing peer-alive hook).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A send now always has a visible, honest state on a bad network; nothing
goes silent and nothing fakes progress.

- WatchConnectivity: engine pushes a Connectivity posture (Disabled /
  Offline / Reconnecting{retryAtMs, lastFailure} / Connected + per-open-chat
  room state) — 1s recompute over in-memory stats, emitted only on change.
  Feeds off the registry room's new reconnect_state (sticky last failure,
  next-dial deadline) and the OS path monitor.
- Sidebar connection pill: appears only while degraded — Offline, or
  Reconnecting with a live retry countdown; the last failure stays visible
  through the next attempt (no flicker back to bare 'connecting…').
- Composer honesty: an amber notice up front when the target's delivery
  path is degraded ('messages will queue and send automatically'), for both
  selected chats and the new-chat canvas (remote target judged by presence).
- Queued, not Working: while a pending send's path is degraded the sidebar
  row wears an amber 'Queued' pill (no spinner) and the transcript trailer
  reads 'Queued — will send automatically'; the pending overlay HOLDS while
  degraded instead of silently expiring at 30s back to Idle (the no-trace
  hole), and expires normally once the path heals.
- chat_delivery_degraded: local-hosted chats never degrade (queued commands
  execute locally offline); remote chats degrade on OS-offline, their own
  room being down, or the host going presence-dark.

Transport-noise suppression audited: socket-close/ping-timeout can only
reach logs and connection state, never chat error chips (harness-wire
errors are the sole chip source) — unchanged by construction.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ork suite (Phase 3)

Second road for a send when rows go dark (03:45 incident shape: nudges
flowed, chat rows didn't — there was no fallback transport at all):

- spawn_command_delivery escorts every remote-hosted queued command:
  attachments push first (the relay must never outrun its bytes), the
  normal rows→edge path gets a 10s grace, then the entry itself relay-
  forwards over the device-room link (RelayCommand), retrying on the
  online-bus backoff up to 15min. Stops the moment either road lands.
- Host side: ingest_relayed_command evaluates the forwarded entry against
  its own doc (dedupe/TTL/supersede unchanged), claims the client-minted id
  in the processed ledger BEFORE executing — the doc row arriving later
  over chat2 sync dedupes to a no-op in the drain. Exactly-once across
  both roads by construction; version-gated on the host's stamped registry
  version (>= 0.2.12); relay timeouts mark the link suspect.
- Integration proof (relay_delivery.rs): A queues for B with NO edge
  configured; the command executes on B via the relay; then the simulated
  doc-row arrival + a duplicate relay both no-op — exactly one turn.
- Flaky-network suite (chat_client tests, virtual clock): a scripted
  drop/refuse connector asserting the durable contract — a mid-push socket
  death + refused dials still deliver the batch exactly once AND surface
  Disconnected (silence is a test failure); connect-and-die sessions grow
  their backoff until a >30s-stable session resets it; OS-offline parks
  dials entirely and the online event un-parks within a beat.
- version_triple moved to zeron-proto (shared by the UI gate, the relay
  gate, and future fleet gates).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Two-minute grace on unadopted sends: the overlay reads Working/Sending
  (or Queued while degraded) through the grace window, then flips to an
  EXPLICIT failed state — sidebar row wears a danger 'Failed' pill and the
  transcript trailer becomes the retry affordance ('Not delivered — click
  to retry'). The old 30s TTL silently expired back to Idle leaving no
  visible trace of the send at all.
- RetryDelivery RPC: re-kicks every road at once — fresh chat2 socket
  (zombie rooms are the usual suspect), host nudge, a local drain pass,
  and a new delivery escort per still-pending command with attachment
  transfers re-derived from the entries' pending:// refs (idempotent;
  the processed ledger keeps execution exactly-once). The UI restarts the
  grace clock so the trailer returns to Sending/Queued while it runs.
- Timestamp canonicalization: the executor pre-writes the user message at
  the entry's issued_at (clamped against clock skew) — a send queued for
  minutes lands in history where the user sent it, not where the host got
  around to draining it. Idempotent by client-minted id, so the dispatch
  path's own execution-time write dedupes.
- Reconnect trace ids landed with Phase 2/3: numbered dial attempts on
  registry + chat2 dials, per-attempt peer-dial logs, command ids on every
  escort/relay line — an incident reads as one numbered sequence.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nt trailer, PR badges)

Semantic reconciliation beyond the textual conflicts:
- upload_attachment: main's parallel windowed chunks + whole-attachment
  deadline + progress counter, KEEPING the caller-minted upload_id (the
  pending:// identity) and warn-level chunk retries. My outer 300s legacy
  budget is dropped — main's size-scaled attachment_deadline supersedes it.
- Both composer flows (queued + legacy) publish main's upload progress.
- main's send_queued_unacked / 'Queued — waiting for connection…' static
  line is superseded by this branch's richer model (degraded → Queued from
  second one, grace window → explicit Failed + retry) — removed along with
  its test; main's upload-progress and change-request tests kept.
- Trailer: main's subagent-doc liveness arm merged with this branch's
  failed-send retry affordance and Queued wording.
- doc_host chat2-join online wake: both sides made the same change (main's
  989be0a) — kept main's comments.
- Sidebar row: main's PR badge + invisible spring kept, with this branch's
  no-spinner-when-Queued/Failed gate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@wingleeio
wingleeio merged commit 7616469 into main Aug 19, 2026
1 check passed
wing-anara added a commit that referenced this pull request Aug 20, 2026
ios: port the network-reliability train (PRs #159/#164/#165/#168/#170) to mobile
tobi404 pushed a commit to tobi404/comet that referenced this pull request Aug 20, 2026
…port)

Ports the desktop reliability train's transport layer to the Swift clients:

- OnlineBus (wake.rs): one process-wide online event — every room join
  broadcasts, NWPathMonitor's offline→online broadcasts, and parked
  backoffs select on it instead of sleeping out timer luck. While the OS
  path is offline, waits park at a 30s safety recheck instead of burning
  dials.
- ChatRoomClient/RegistryClient: backoff cap 30s→16s, stability-gated
  reset (a session must survive 30s to earn a fresh 250ms — reset-on-join
  let connect-and-die sockets hot-loop), and backoff waits ride the bus.
- DeviceRelayClient: end-to-end echo liveness (10s keepalive, 25s
  transport lease, 20s host-echo deadline, feature-detected) — the DO's
  auto-pong answers from the EDGE, so a healthy client↔edge leg masked a
  dead edge↔host leg for minutes (Aug 19 zombie-peer-link incident). A
  timed-out call now drops the link as suspect so the next call re-dials
  instead of hanging on the same dead leg, and a registry-dark liveness
  verdict fails peer calls fast with zero dials.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tobi404 pushed a commit to tobi404/comet that referenced this pull request Aug 20, 2026
…sh#159/zeronsh#164/zeronsh#165/zeronsh#168/zeronsh#170 port)

Send truth:
- ConnectivityCenter: graced connectivity (DegradeGrace 4s per source —
  OS path / registry room / each open chat room; recovery reports
  instantly) feeding one stream every consumer inherits calm from.
- Home bar: quiet "Reconnecting…" spinner / amber "Offline — sends are
  saved" dot, only while degraded past the grace.
- Rows: amber "Queued" / red "Failed" corner pills override the status
  (and suppress the Working spinner — no fake progress); session strip
  shows "Sending…" / "Queued — will send automatically" / "Not delivered
  — tap to retry" with a 2-minute undelivered grace; retry restarts the
  clock, kicks the room, nudges the host, and re-derives escorts.
- Composer: one caption line of pre-send honesty on a degraded path.

Attachments:
- PR zeronsh#164 uploader: ~510KB chunks (was ~45KB), 3-wide window, per-chunk
  retries staggered by seq, whole-upload deadline min(120+15/chunk, 900s),
  "Uploading… N%" narration.
- Queued flow (host ≥ 0.2.12, version-gated via the registry device row):
  the command queues immediately with pending://{uploadId}/{name} refs —
  a durable local write — bytes stash to disk and an escort pushes them
  with retry-forever on the online bus (2s→30s backoff, 15min cap),
  surviving relaunches via re-derivation from the doc's pending refs.
  An image send no longer dies with a dead link (Aug 19 incident class).

New chats:
- New-worktree sends attach RunRequest.worktree {repoPath, base} for the
  HOST to materialize at drain time — the blocking CreateWorktree relay
  RPC before send (the one path that could hang forever on a zombie
  link) is gone; base defaults to HEAD so a branch list that never
  loaded can't silently drop isolation.
- Legacy hosts stage attachments BEFORE the chat row exists — an upload
  failure aborts with nothing created, no stranded empty chat.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ogdakke added a commit to ogdakke/comet that referenced this pull request Aug 23, 2026
* Fix transcript viewport restoration

* fix(ui): refine panel resizing and alignment

* fix(ui): gate Git add menu by workspace

* fix(ui): unify new-session action in titlebar

* fix(ui): size project menu with sidebar

* fix(ui): reverse right-pane takeover icon

* Anchor live streams at the transcript end

* Add entry-level message copying and spacing refinements

* Fix transcript copy fidelity and spacing parity

* Fix viewport test row after upstream merge

* v0.2.20

* Reasoning becomes a first-class transcript part

AgentEvent::ReasoningDelta was dropped at the doc fold for every harness —
no client ever rendered thinking. It now folds into a new 'reasoning' doc
part and renders as a collapsible Thinking block on desktop:

- doc: MessagePart::Reasoning rides a dedicated 'reasoning' LoroText field
  (never 'text' — old desktop readers' unknown-kind fallback renders 'text'
  as prose, so thinking on this field degrades to an invisible empty part;
  old iOS drops unknown kinds). Streaming appends RLE-merge like text, the
  WatchDocMessages hot path carries reasoning growth as TextAppend deltas,
  oversized thinking splits at the continuation cap, and salvage recovers
  the field.
- ui: a Thinking row (tool-group header recipe: chevron tile + quiet label)
  auto-opens while it is the live tail of a streaming reply and collapses
  once settled; a user toggle overrides either way. Body renders dimmed
  along the guide rail.
- edge TS mirror: 'reasoning' kind in the doc part codec + split gate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* opencode: native HTTP/SSE driver replaces the ACP path

opencode's own ACP layer settles a prompt on the FIRST session idle it
observes — uncorrelated with the submitted turn — and ignores
session.error and session.idle entirely (packages/opencode/src/acp/event.ts),
which is exactly the 'says it's done when it's not done' bug; it also
drops all subagent traffic. The desktop app doesn't use ACP, and now
neither do we.

OpencodeHarness speaks the same v1 wire the opencode desktop app does:
spawn `opencode serve --port <free>` with a per-run Basic password, drive
POST /session/{id}/prompt_async, and read the /global/event SSE bus.

- Done is truthful by construction: only session.status{idle} for OUR
  session settles the turn (deterministic_turn_end retires the engine's
  quiesce watchdog). No quiet-settle heuristics.
- Thinking streams: reasoning parts/deltas map to ReasoningDelta.
- Provider failures are visible (the zeronsh#169 class): retry status surfaces an
  error chip at attempt 3 and aborts at 8 instead of retrying forever; a
  turn with no sign of life within ZERON_OPENCODE_STALL_MS (60s) errors
  out with the provider hint.
- Subagents ride the same bus natively: task parts register chips, child
  sessions bind by spawn metadata (title-match fallback), traffic streams
  tagged, the completing task part settles the chip. Child permission asks
  auto-approve too (the ACP layer silently dropped them and subagents hung).
- Live model discovery off GET /provider ({providerID}/{modelID} ids;
  effort ladder maps to the model's advertised reasoning variants and rides
  prompts as `variant`). Slash commands from GET /command route through
  the command endpoint. Resume reuses the durable server-side session.

The ACP opencode spec, the /event sidecar tracker, and the per-run --port
plumbing are gone; grok keeps its own tracker. Covered by a scripted
in-test HTTP/SSE fake (tests/opencode.rs) where the fixtures own the bus
clock — including the foreign-session-idle regression.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* opencode: bound every HTTP call; a boot-window request parks forever

Live finding (1.18.18): during opencode serve's boot window the socket
ACCEPTS but the request is parked — and one parked early enough is never
answered at all, so an unbounded send() wedged startup past the whole 300s
budget. Health probes now time out at 2s and re-poll; ordinary calls get a
60s bound. The synchronous per-turn command endpoint deliberately stays
unbounded (its response can take the whole turn and is ignored — the bus
carries the real events), and the interrupt-path abort gets its own 5s
bound so a wedged server can't block the event loop.

Live-validated end to end against real opencode 1.18.18 + a scripted
OpenAI-compatible provider streaming reasoning_content: parent and
subagent thinking stream as ReasoningDelta, the task chip binds/settles,
and the turn completes only on its own session's idle.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* opencode: rustfmt the new driver module

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* opencode: gate the first prompt on a live event subscription

Live finding on fresh 1.18.21: the v1 bus has NO replay, and a
fast-failing turn (a bad model id errors in ~200ms) can emit its whole
busy → session.error → idle lifecycle before a late /global/event
subscription exists — the driver then saw nothing and fell back to the
60s stall watchdog instead of surfacing the real error. The bus reader
now reports each stream (re)connect after its first frame; the first
prompt waits (bounded 15s) for the subscription to be live, and a
RECONNECT during an active turn re-syncs from GET /session/status so a
gap can no longer swallow the idle that ends the turn.

Re-validated against opencode 1.18.21 installed via the official
installer: the mock subagent+thinking scenario, the bad-model fast
failure (error chip + Done{Errored} in 2s, previously 60s), and a REAL
anonymous OpenCode Zen model (opencode/mimo-v2.5-free) end to end —
reasoning streamed, a real bash tool call executed, usage reported,
truthful Completed settle.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* v0.2.21

* opencode: the picker offers only connected providers' models

Field report (v0.2.21): the model picker was slow to open, slow to
scroll, and full of models the user can't run. GET /provider's `all` is
the entire models.dev catalog — measured live at 194 providers / 7,203
models, nearly all needing an API key the user hasn't set; picking one
just fails the run with "Model not found". The response's `connected`
list names exactly the usable set (credentialed + config-declared + the
anonymous Zen tier), so discovery now filters to it (falling back to the
full catalog only when the field is absent/empty — older servers). On
this rig that's 7 models instead of 7,203; the lag was the list.

Also from the same report: opencode emits each provider failure TWICE on
the bus (once bare, once wrapped in an exception-name/stack prefix), so
every failed turn rendered two identical error chips. Consecutive
session.error events whose first lines contain each other now dedupe to
one chip.

Plus a live models probe example (cursor precedent):
    cargo run -p zeron-harness --example opencode_models_probe

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* v0.2.22

* Model picker scales to thousand-model catalogs; rows attribute their provider

Field follow-up on the opencode picker work: the picker must handle
LARGE lists (not just rely on the connected filter keeping them small),
and identically-named models need visible provenance — 'GLM-5.2' exists
under 64 providers in the catalog, so a user with several connected
providers saw indistinguishable duplicate rows.

- The model list is now a virtualized uniform_list: only the visible
  slice renders, so a 7,203-row catalog opens and scrolls as smoothly as
  seven (validated live against the full unfiltered models.dev catalog
  through the real pipeline). The floating scrollbar reads the
  UniformListScrollHandle's base handle; keyboard nav scrolls via
  scroll_to_item(Nearest).
- The flattened row list is cached per (query, rail, harness, lock,
  catalog revision): it was rebuilt — with per-row Model clones, ranked
  and sorted — 2-3x per keystroke and again per render. Favorite lookups
  use a set instead of a per-row Vec scan.
- Rows keep the harness subline and append the model's description when
  it adds information: opencode ships the provider display name there,
  so duplicate names read 'OpenCode · HPC-AI' vs 'OpenCode · Nvidia'.
  The search haystack includes the description, so typing a provider
  name filters to it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* v0.2.23

* Thinking rides the tool-group accordion

Field follow-up on the thinking feature: the standalone 'Thought process'
row didn't animate like the other transcript accordions (its body
mounted at intrinsic height — no analytic height, no tween), and wing
wants thinking INSIDE the combined 'Ran N commands' fold rather than as
its own row between groups.

Reasoning parts now become thought chips in the tool group itself:

- rows_for_entry routes MessagePart::Reasoning into the pending group
  (agent/spawn groups stay pure, same as the tool genus rule), so
  thought → tool → thought → tool folds into ONE accordion row.
- A thought chip is a synthesized ToolItem ('Thought process' header,
  chat bubble icon) whose text is an output-style detail: soft-wrapped
  at word boundaries into lines, capped like tool outputs (a STREAMING
  thought keeps the tail — the fresh thinking is the signal). Line-based
  details have ANALYTIC heights, so the group's existing 200ms fold
  tween covers thoughts with zero new animation code — the smoothness
  ask is satisfied by construction.
- A live thought chip defaults open (the group auto-opens as the live
  tail already); it settles closed once text or a tool follows. User
  toggles override either way via the existing per-chip fold state.
- The collapsed summary names the thinking: 'Thought process' for a
  thought-only group, 'Thought 3 times · Ran 2 commands' when mixed
  (thought chips are UI-synthesized, so the shared view summary never
  sees them and the terminal viewport is untouched).
- RowKind::Thinking, its renderer, and the special top-gap case are
  deleted — one fewer bespoke row kind.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* v0.2.24

* Chat rows survive configs from newer peers; iOS learns opencode

Field report (multi-device): new sessions never appeared in a work
laptop's sidebar while old sessions kept updating, and on iOS opencode
sessions wore the claude mark and opened showing claude/Fable.

Two independent gaps, one cause — the fleet learned a harness id
('opencode', v0.2.10) that older surfaces don't know:

- Desktop: the typed row decode (read_kind → RawChat) SKIPPED any row it
  couldn't parse, and ChatConfig.harness is a closed enum — so a client
  older than the newest harness id drops every such chat row wholesale.
  Exactly 'old rows update, new rows invisible'. The config field now
  decodes leniently: an unknown harness/reasoning/sandbox value costs
  the CONFIG (row renders with generic defaults), never the row. Same
  rule as transcript salvage: a missing field costs at most what it
  carried. Regression-tested with a future-harness row.
- iOS: HarnessCatalog and BrandMark had no 'opencode' entry, so the
  default arms answered with the claude mark and the claude model list.
  Added the label, the mark (opencode-mark.svg's frame path), and a
  static model fallback (the anonymous Zen tier; a reachable host still
  answers listModels with live discovery).

The laptop itself needs an app update to pick this up — the tolerance
protects every FUTURE skew, it cannot reach builds that predate it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* v0.2.25

* Registry cursor can no longer jump over unapplied rows

Field incident, root-caused for real this time (the earlier stale-version
theory was wrong — the affected laptop ran v0.2.24): new sessions created
on one device never appeared in another device's sidebar, while updates
to already-known rows kept flowing.

The hole is deterministic on the HTTPS transport (used exactly when a
network strips WebSocket upgrades — zeronsh#168's dual transport, v0.2.12): one
sync cycle pushes before it pulls, the push ack set the cursor to OUR
batch's seq, and the pull then fetched 'since' that jumped cursor —
permanently skipping every batch other devices wrote since the last
sync. A single local write (a seen-marker sufficed) while behind made
peers' new chats invisible forever; rows touched later self-healed,
which is why statuses kept updating.

- ack_batch no longer advances the cursor (the ack's seq is our own
  batch; the cursor only moves with actual row payloads — re-pulling our
  own batch is an idempotent LWW no-op).
- apply_rows guards contiguity: a broadcast past a seq gap (hibernating
  DO, half-dead socket) applies its rows but HOLDS the cursor and the
  client reconnects — the fresh hello backfills the window.
- Persisted snapshots gain a resync epoch: cursors written before this
  fix may already have jumped past invisible rows, so the first load
  after upgrading zeroes the cursor once and pulls full state (rows are
  kept; local-only rows re-seed through the existing full-state path).
- iOS mirrors all three (its WS ordering has masked the ack bug so far;
  gap-hold converges on the next natural reconnect).

Verified against production data: the 'missing' chats and their space
rows are complete and correct in the registry — the failure was purely
the reader's cursor.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* v0.2.26

* Orphan sweep waits for server truth; unreadable HTTP ack retries

Two residuals from the missing-sessions audit (v0.2.26 follow-up):

- sweep_orphans ran on whatever the local replica held, including a
  replica that had not heard from the server that boot (gapped cursor,
  pre-heal, offline at boot). On such a view "space row absent" means
  "not received yet", not "deleted" — and the sweep's deletes are
  global and permanent. The sweep now gates on a `synced` latch that
  flips once a server state applies this process (WS hello answer or
  HTTPS pull); local-only hosts (no edge configured) are exempt since
  their table is the whole truth.

- spawn_offline_sync silently ignored a push ack it could not parse
  (captive portal / proxy bodies), leaving the batch marked in-flight.
  With no socket session there is no disconnect to un-mark it, so the
  write was stranded until process restart. An unreadable ack now
  counts as a failed push: the batch becomes pushable again and the
  next cycle retries it.

`synced` also rides RoomStatsSnapshot / SyncStatus for observability.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* v0.2.27

* ci: add permanent internal TestFlight workflow (zeronsh#213)

* Promote queued prompts on delivery and keep hang-turn tests live

write_user_message treated an existing queued id as a no-op, so drain
never flipped the row to Complete. WorkOS-unset boot without a bearer
also failed to enter development, which broke the offline-dev test.

Hang-turn e2e scripts no longer emit Done before interrupt, so a
steerable harness cannot park and empty the queue early.

* Silence merge leftover compile warnings

Wire the model-picker scrollbar drag-move listener so grab_offset is
read. Drop unused RunRequest bindings in test harnesses, stop importing
CHAT_PANEL_MIN into the shell lib, and allow TILE_CORNER_RADIUS on
non-macOS where hover surfaces are not painted.

* Apply the remaining upstream merge leftovers

Right-pane drag and target width now keep a 300px chat floor and use
takeover width instead of the old 52% cap. Opening the model picker
revalidates catalogs and retries a cold OpenCode probe. Studio's
titlebar uses the same right inset as chat. PARITY.md lists Cursor,
Pi, and OpenCode as native drivers.

* Keep queued prompts in the tray until the live turn ends

Queued user entries were painted as transcript bubbles the moment they were
parked. Skip them in rows_for_entry, and add an e2e that the follow-up stays
Queued and unsent until the live turn commits it.

---------

Co-authored-by: José Gurruchaga <jsgrrchg@gmail.com>
Co-authored-by: gaelcado <cado.gael@gmail.com>
Co-authored-by: Wing <44917547+wingleeio@users.noreply.github.com>
Co-authored-by: wing-anara <wing@anara.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
guilhermexp pushed a commit to guilhermexp/comet that referenced this pull request Aug 30, 2026
…on-bad-networks

Durable-by-design delivery on bad networks
guilhermexp pushed a commit to guilhermexp/comet that referenced this pull request Aug 30, 2026
…port)

Ports the desktop reliability train's transport layer to the Swift clients:

- OnlineBus (wake.rs): one process-wide online event — every room join
  broadcasts, NWPathMonitor's offline→online broadcasts, and parked
  backoffs select on it instead of sleeping out timer luck. While the OS
  path is offline, waits park at a 30s safety recheck instead of burning
  dials.
- ChatRoomClient/RegistryClient: backoff cap 30s→16s, stability-gated
  reset (a session must survive 30s to earn a fresh 250ms — reset-on-join
  let connect-and-die sockets hot-loop), and backoff waits ride the bus.
- DeviceRelayClient: end-to-end echo liveness (10s keepalive, 25s
  transport lease, 20s host-echo deadline, feature-detected) — the DO's
  auto-pong answers from the EDGE, so a healthy client↔edge leg masked a
  dead edge↔host leg for minutes (Aug 19 zombie-peer-link incident). A
  timed-out call now drops the link as suspect so the next call re-dials
  instead of hanging on the same dead leg, and a registry-dark liveness
  verdict fails peer calls fast with zero dials.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
guilhermexp pushed a commit to guilhermexp/comet that referenced this pull request Aug 30, 2026
…sh#159/zeronsh#164/zeronsh#165/zeronsh#168/zeronsh#170 port)

Send truth:
- ConnectivityCenter: graced connectivity (DegradeGrace 4s per source —
  OS path / registry room / each open chat room; recovery reports
  instantly) feeding one stream every consumer inherits calm from.
- Home bar: quiet "Reconnecting…" spinner / amber "Offline — sends are
  saved" dot, only while degraded past the grace.
- Rows: amber "Queued" / red "Failed" corner pills override the status
  (and suppress the Working spinner — no fake progress); session strip
  shows "Sending…" / "Queued — will send automatically" / "Not delivered
  — tap to retry" with a 2-minute undelivered grace; retry restarts the
  clock, kicks the room, nudges the host, and re-derives escorts.
- Composer: one caption line of pre-send honesty on a degraded path.

Attachments:
- PR zeronsh#164 uploader: ~510KB chunks (was ~45KB), 3-wide window, per-chunk
  retries staggered by seq, whole-upload deadline min(120+15/chunk, 900s),
  "Uploading… N%" narration.
- Queued flow (host ≥ 0.2.12, version-gated via the registry device row):
  the command queues immediately with pending://{uploadId}/{name} refs —
  a durable local write — bytes stash to disk and an escort pushes them
  with retry-forever on the online bus (2s→30s backoff, 15min cap),
  surviving relaunches via re-derivation from the doc's pending refs.
  An image send no longer dies with a dead link (Aug 19 incident class).

New chats:
- New-worktree sends attach RunRequest.worktree {repoPath, base} for the
  HOST to materialize at drain time — the blocking CreateWorktree relay
  RPC before send (the one path that could hang forever on a zombie
  link) is gone; base defaults to HEAD so a branch list that never
  loaded can't silently drop isolation.
- Legacy hosts stage attachments BEFORE the chat row exists — an upload
  failure aborts with nothing created, no stranded empty chat.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
guilhermexp pushed a commit to guilhermexp/comet that referenced this pull request Aug 30, 2026
guilhermexp pushed a commit to guilhermexp/comet that referenced this pull request Aug 30, 2026
Field incident, root-caused for real this time (the earlier stale-version
theory was wrong — the affected laptop ran v0.2.24): new sessions created
on one device never appeared in another device's sidebar, while updates
to already-known rows kept flowing.

The hole is deterministic on the HTTPS transport (used exactly when a
network strips WebSocket upgrades — zeronsh#168's dual transport, v0.2.12): one
sync cycle pushes before it pulls, the push ack set the cursor to OUR
batch's seq, and the pull then fetched 'since' that jumped cursor —
permanently skipping every batch other devices wrote since the last
sync. A single local write (a seen-marker sufficed) while behind made
peers' new chats invisible forever; rows touched later self-healed,
which is why statuses kept updating.

- ack_batch no longer advances the cursor (the ack's seq is our own
  batch; the cursor only moves with actual row payloads — re-pulling our
  own batch is an idempotent LWW no-op).
- apply_rows guards contiguity: a broadcast past a seq gap (hibernating
  DO, half-dead socket) applies its rows but HOLDS the cursor and the
  client reconnects — the fresh hello backfills the window.
- Persisted snapshots gain a resync epoch: cursors written before this
  fix may already have jumped past invisible rows, so the first load
  after upgrading zeroes the cursor once and pulls full state (rows are
  kept; local-only rows re-seed through the existing full-state path).
- iOS mirrors all three (its WS ordering has masked the ack bug so far;
  gap-hold converges on the next natural reconnect).

Verified against production data: the 'missing' chats and their space
rows are complete and correct in the registry — the failure was purely
the reader's cursor.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
guilhermexp pushed a commit to guilhermexp/comet that referenced this pull request Aug 30, 2026
Field incident, root-caused for real this time (the earlier stale-version
theory was wrong — the affected laptop ran v0.2.24): new sessions created
on one device never appeared in another device's sidebar, while updates
to already-known rows kept flowing.

The hole is deterministic on the HTTPS transport (used exactly when a
network strips WebSocket upgrades — zeronsh#168's dual transport, v0.2.12): one
sync cycle pushes before it pulls, the push ack set the cursor to OUR
batch's seq, and the pull then fetched 'since' that jumped cursor —
permanently skipping every batch other devices wrote since the last
sync. A single local write (a seen-marker sufficed) while behind made
peers' new chats invisible forever; rows touched later self-healed,
which is why statuses kept updating.

- ack_batch no longer advances the cursor (the ack's seq is our own
  batch; the cursor only moves with actual row payloads — re-pulling our
  own batch is an idempotent LWW no-op).
- apply_rows guards contiguity: a broadcast past a seq gap (hibernating
  DO, half-dead socket) applies its rows but HOLDS the cursor and the
  client reconnects — the fresh hello backfills the window.
- Persisted snapshots gain a resync epoch: cursors written before this
  fix may already have jumped past invisible rows, so the first load
  after upgrading zeroes the cursor once and pulls full state (rows are
  kept; local-only rows re-seed through the existing full-state path).
- iOS mirrors all three (its WS ordering has masked the ack bug so far;
  gap-hold converges on the next natural reconnect).

Verified against production data: the 'missing' chats and their space
rows are complete and correct in the registry — the failure was purely
the reader's cursor.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants