Scheduled automation dispatch silently lost while target host is verifiably online; manual 'automations run' succeeds minutes later #6128

Activity

1. github-actions commented on Aug 2, 2026

   

   github-actionsBot

   on Aug 2, 2026 – with GitHub Actions

   Contributor

   More actions

   Summary

   The always-on-host loss is almost certainly not a host/relay/tunnel problem — it's a race between the two halves of the scheduled path. evaluate enqueues the dispatch job and flips enabled:false for an exhausted recurrence in the same request, and the dispatch handler refuses to run any automation whose enabled is false. For the terminal occurrence of any bounded (COUNT/UNTIL) rule the disable lands before QStash delivers the job, so the dispatch is dropped before dispatchAutomation is ever called: no run row, no workspaces.create, and an automation that now looks like a cleanly completed series. FREQ=DAILY;COUNT=1 is the worst case — its only slot is also its terminal slot, so it can never dispatch. Run-now works because it calls dispatchAutomation directly and never consults enabled.

   Affected code

   • apps/api/src/app/api/automations/evaluate/route.ts:56-85 — enqueues the QStash job (56-70), then advances or disables (72-85). The disable is apps/api/src/app/api/automations/evaluate/route.ts:82: set(next ? { nextRunAt: next } : { enabled: false }), where next is null once the rule is exhausted (packages/shared/src/rrule.ts:359-370).
   • apps/api/src/app/api/automations/dispatch/[id]/route.ts:58-60 — if (!automation.enabled) return Response.json({ ok: true, skipped: "disabled" }).
   • packages/trpc/src/router/automation/dispatch.ts:43-186 — the dispatcher itself.
   • packages/trpc/src/router/automation/automation.ts:551-565 — runNow, which calls dispatchAutomation with no enabled check and no QStash hop.
   • apps/api/src/app/api/automations/run-failed/route.ts:90-105 — the only path that backfills a dispatch_failed row.

   Suspected root cause

   For a COUNT=1 automation whose slot is 18:46:

   1. The 18:46 tick selects it (enabled = true AND next_run_at <= now, line 48) and enqueues dispatch/<id> with scheduledFor=18:46:00 and deduplicationId=<id>_<ms>.
   2. await qstash.batchJSON(...) resolves as soon as QStash accepts the message; the handler immediately issues the UPDATE ... SET enabled = false. A single UPDATE against the DB reliably beats QStash's outbound HTTP delivery.
   3. QStash POSTs to dispatch/<id>; the handler reloads the row, sees enabled === false, and returns 200 {skipped: "disabled"}.

   The 200 is what makes it silent: QStash treats it as success, so failureCallback never fires and no dispatch_failed row is written (run-failed/route.ts only runs on non-2xx after retries). The retries: 2 on line 66 can't help either — every retry hits the same guard. And the deduplicationId means the next tick won't re-enqueue.

   This also explains your "no run row" observation specifically. Every path inside dispatchAutomation persists a row — recordSkipped for both no-host and offline-host (dispatch.ts:51, dispatch.ts:57), the dispatching insert (dispatch.ts:66-79), and the dispatch_failed update (dispatch.ts:174-181). So a slot with zero rows must have been lost upstream of it, which leaves only the !automation (deleted) and !automation.enabled guards. Your automation still exists, so it's the enabled guard.

   What I did not find a common cause for: the earlier Mac-host losses. skipped_offline and dispatch: workspaces.create: Tunnel disconnected both come from inside dispatchAutomation and did produce rows, so those are a genuinely separate (real host-availability) class. That's consistent with them being non-terminal occurrences of your COUNT=3 rules — the first N-1 slots reach the dispatcher and record an outcome, the final one vanishes. It does mean the two failure classes have been getting conflated in the 8/8 tally.

   Reproduction

   Fastest end-to-end check, and a discriminating one — create two automations against a host you know is online:

   • FREQ=DAILY;COUNT=1, slot a few minutes out → expect zero run rows and enabled:false.
   • FREQ=MINUTELY;COUNT=2 → expect the first slot to dispatch normally and the second to vanish with no row. If slot 1 works and slot 2 doesn't, the terminal-slot hypothesis is confirmed and host state is fully excluded.
   • Any unbounded rule (no COUNT/UNTIL) should be unaffected, since next is never null.

   A bun:test reproduction doesn't need QStash: call the exported POST from evaluate/route.ts (or just replicate lines 72-85) against a seeded COUNT=1 automation, then call the POST from dispatch/[id]/route.ts with {automationId, scheduledFor} — with signature verification stubbed — and assert that automationRuns for that automationId is empty and the response is {skipped: "disabled"}. That's the bug, deterministically, without any timing dependence.

   Fix direction

   Note that simply reordering evaluate won't help — the enqueue already precedes the disable; the problem is that the guard can't distinguish "user paused this" from "we just disabled this because of the very slot being dispatched."

   Minimal fix: teach the dispatch handler about the terminal slot. evaluate already leaves next_run_at pointing at the terminal occurrence when it disables (line 82 only sets enabled), so dispatch/[id]/route.ts:58 can become "bail only if disabled and this isn't the terminal slot":

   if (!automation.enabled && !parsed.data.terminal) {
     return Response.json({ ok: true, skipped: "disabled" });
   }

   with terminal: z.boolean().default(false) added to payloadSchema and set by evaluate where it already computes next === null. Cross-checking bucketToMinute(automation.nextRunAt) against parsed.data.scheduledFor closes the remaining window where a user pauses between enqueue and delivery.

   Structurally cleaner, and it addresses the "don't consume COUNT without a dispatch attempt" ask directly: stop overloading enabled for exhaustion. The schema has only the one boolean (packages/db/src/schema/schema.ts:786), which is why a finished series and a silently-lost final slot are indistinguishable in the UI. A separate terminal state (completedAt, or a status enum) would let evaluate mark exhaustion after the dispatch outcome is recorded, and let the UI say "completed" vs "paused".

   Two smaller things worth folding in, both matching your asks:

   • Make the drop non-silent regardless: return non-2xx (or write a dispatch_failed/skipped row) on the disabled branch, so it flows through run-failed and Sentry instead of looking like a success.
   • On retry/backfill generally: evaluate advances next_run_at unconditionally at line 82, and skipped_offline is terminal in dispatch.ts:51-64, so a missed minute is indeed gone permanently. A bounded retry would need either a re-enqueue on skipped_offline or a catch-up sweep that re-selects recent slots lacking a successful run row.
2. Kitenite commented on Aug 26, 2026

   

   Kitenite

   on Aug 26, 2026

   Member

   More actions

   Fixed by #6635 (c08650514) — closing.

   The root cause in the analysis comment was right: evaluate enqueued the dispatch job and flipped the trigger disabled in the same request, and dispatch refuses any automation that is not enabled. For the terminal occurrence of a bounded rule the disable won the race, so the job was dropped before dispatchAutomation ever ran — no run row, no workspaces.create, exactly what you saw.

   #6635 removed that write. The line that did it:

   -				.set(next ? { nextRunAt: next } : { enabled: false })

   evaluate now only ever advances nextRunAt, and never disables anything.

   The other half is that bounded rules no longer reach the scheduler at all — COUNT/UNTIL are refused at save (packages/shared/src/automation-triggers.ts:541) with "Schedules repeat — remove COUNT or UNTIL." Your automation used FREQ=DAILY;COUNT=1, which is precisely the shape that is now rejected up front instead of silently self-destructing on its one and only fire.

   Reopen if you see a scheduled dispatch go missing on a repeating rule — that would be a different bug from this one, and worth its own issue with the slot timestamp and automation id.
3. Kitenite

   closed this as completedon Aug 26, 2026