Skip to content

Keep undelivered notifications and stop push floods - #172

Merged
danielmaier42 merged 3 commits into
mainfrom
feature/durable-notifications
Nov 11, 2026
Merged

danielmaier42 merged 3 commits into
mainfrom
feature/durable-notifications

Conversation

@danielmaier42

Copy link
Copy Markdown
Owner

What changes for you

A push raised without network is no longer lost. A MacBook on battery with its lid closed sleeps despite caffeinate -i and wakes for about a minute every quarter of an hour, usually without network. Failure pushes raised in those wakes were tried once and dropped (the live log had 97 such losses, 89 of them DNS failures). Now every notification is written per channel to an outbox on disk (deliveries.json) before it goes out. A channel that does not take it is tried again after 30 seconds, then with doubling pauses up to 5 minutes, at once after the Mac wakes or the network is back, and across a daemon restart. Pushover, ntfy and the webhook are tried for 24 hours, the macOS banner for 1 hour. After that the daemon writes NOTIFICATION LOST to dispatchqd.err.log. A refusal that would only repeat, such as a revoked token, is logged and not retried.

Every accepted send is logged in dispatchqd.out.log, with Pushover's request id or ntfy's message id as the receipt. Credentials are never logged.

No runs started into a wake without network. An agent task on Claude Code or Codex waits while its API host does not resolve. Its queue reason is "No network connection", and it starts on the first check that has network. Script jobs and opencode are not held back. A base URL, Bedrock, Vertex or a proxy in the provider's environment is respected.

Failures go out with high priority (Pushover priority 1, ntfy "high") and name the run id, which finds the run in the Log.

One push per failing streak. A watcher that fails every few minutes used to push every failure. Now the first failure pushes. Later failures only update that row in the Notifications view ("failing since 08:00 (4 times)"), with a reminder push after 12 hours. Exactly one "works again" push goes out when the task next succeeds. The streak is kept in failing.json, so a restart does not reset it.

Long waits and long pauses are announced once.

  • A due task that has not started after 30 minutes gets one notice per wait (24 hours for backfill tasks).
  • A wait you caused yourself (runs paused) or one already announced elsewhere (out of allowance, provider not ready, no network) gets no notice.
  • A rate-limit pause is announced once when it lasts 2 hours or more. Back-to-back short pauses add up.

Cancellations are not announced. You did them yourself, and the Log records them as canceled.

README: the claim that caffeinate holds the Mac awake is corrected. A new "When DispatchQ notifies" section and the data-on-disk rows describe the above.

Evidence

  • Every behaviour fix has a regression test that fails with the fix removed and passes with it in place (outbox retry and restart, receipt log, high priority, failure streak and reminder row, recovery, pause, waiting notice, network gate, expiry of old entries, API host from the provider's environment).
  • End-to-end run with an isolated daemon, a failing script watcher and a fake ntfy server that refused the first three tries. The failure push was accepted on try 4, 3m32s after it was raised, with high priority. Four failures produced one push and one counting row. The recovery produced one push.
  • go build, gofmt, go vet, golangci-lint run (0 issues) and go test ./... -race all pass. The web change (a "Task waiting" kind in the Notifications view) passes tsc, eslint and its tests.

Not covered

  • Pause and waiting notices are remembered in memory only, so after a restart a long pause or wait can be announced once more.
  • A tap on a phone push cannot open the run, because the dashboard is loopback-only. The run id in the message is the link to it.

🤖 Generated with Claude Code

Daniel Maier and others added 3 commits November 11, 2026 17:47
A push raised in a short wake without network was tried once and lost.
Every notification now waits per channel in a delivery outbox on disk and
is retried with backoff (at once after a wake or when the network is
back) for a day, with a log line per accepted send and its receipt.

Agent tasks on Claude Code and Codex wait while their API host does not
resolve instead of failing in a DarkWake. Failures go out with high
priority and the run id, once per failing streak with one recovery push;
long rate-limit pauses and long waits of due tasks are announced once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claim outbox entries before writing them so a concurrent retry sweep
cannot send them twice, drop entries past their window without a try,
retry channels side by side and stop the tries on shutdown, detect a
wake on the wall clock, check the API host the instance actually uses,
keep a pause chain open across runs that started before it, count only
unexplained waiting time, and keep one inbox row for a failure streak
across reminders.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@danielmaier42
danielmaier42 merged commit e372e4b into main Nov 11, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant