Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 5 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -776,8 +776,8 @@ first. The window opens on the Dashboard:
- **Notifications** — every notification DispatchQ sent, newest first, with an
unread badge. It is the one view with a read state; the Log and Artifacts
carry no badges, since what matters in them arrives here as well: a run's outcome, a new artifact, a provider that stopped or
resumed running tasks or is paused for hours, a task that has waited long
without starting, and whatever a task said with `dispatchq notify`. A task
resumed running tasks or is paused for hours, and whatever a task said with
`dispatchq notify`. A task
that keeps failing has one row that counts its failures ("failing since
08:00 (5 times)") and moves to the top with each. Each
row shows the notification as it was shown on screen — a glyph for its kind,
Expand Down Expand Up @@ -2342,14 +2342,13 @@ webhook) and to the **Notifications** view. What is announced, and how often:
| The task works again | Once, with the run that ended the streak ("Failed 5 times since 08:00; this run succeeded."). Replaces the task's own success notification for that run. | Normal |
| A run succeeds | Only when the task opts in (*Notify on result*). | Normal |
| A provider is paused by its rate limit | Once per pause, when the pause lasts 2 hours or more, counted from the first limit (back-to-back short pauses add up). Shorter pauses resume by themselves and are not announced. | Normal |
| A due task has not started | Once per wait, after 30 minutes; a backfill task, whose job is to wait for spare allowance, after 24 hours. Not for waits announced otherwise or that are your own doing: runs paused, a provider out of allowance or not ready, no network. | Normal |
| A due task has not started | Not announced. The Queue shows the wait duration and current reason; long waits are recorded in the daemon log. | — |
| A provider stops or resumes running tasks | Once per change (unchanged). | Normal |
| A run is cancelled, a resume dropped, a run taken over | Not announced: you did it yourself, and the Log records it as `canceled`. | — |

The failure streaks are kept in `failing.json`, so a restart neither announces
a streak again nor forgets to say it ended. Pause and waiting announcements
are kept in memory: after a restart a long pause or wait can be announced once
more.
a streak again nor forgets to say it ended. Pause announcements are kept in
memory: after a restart a long pause can be announced once more.

**Delivery.** Each channel is delivered to on its own. Before a notification
goes out, a delivery per channel is written to `deliveries.json`; a channel
Expand Down
64 changes: 1 addition & 63 deletions internal/engine/alerts.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import (

// When the daemon speaks up about something that is not a single run's
// outcome. Each is announced once per episode: a task that keeps failing,
// a provider that stays paused, a task that keeps waiting.
// a provider that stays paused.
const (
// FailureReminderAfter is how long a task may keep failing before the
// streak is pushed again. In between, every further failure only updates
Expand All @@ -27,13 +27,6 @@ const (
// pauseChainGap is how soon after a pause ended a new one still counts as
// the same: a resumed session that hits the limit again at once.
pauseChainGap = 30 * time.Minute
// HoldNotifyAfter is how long a due task may wait without starting before
// the operator is told, for a reason nothing else announces (it runs alone
// and the parallel work keeps going, its pool has no member, ...).
HoldNotifyAfter = 30 * time.Minute
// BackfillHoldNotifyAfter is the same for a backfill task, whose job is
// to wait for spare allowance: only a wait of a whole day is unusual.
BackfillHoldNotifyAfter = 24 * time.Hour
)

// alerts is the memory of the episodes that are not kept on disk.
Expand Down Expand Up @@ -283,58 +276,3 @@ func (e *Engine) endPause(providerID string, started time.Time) {
delete(e.alerts.pauses, providerID)
}
}

// holdLimits says how long each task may wait before its wait is announced:
// never while held for a reason that is announced elsewhere or is the
// operator's own doing (explained), a day for backfill work, half an hour
// for anything else.
func holdLimits(cfg store.Config, explained map[string]bool) func(id string) time.Duration {
backfill := map[string]bool{}
for _, t := range cfg.Tasks {
backfill[t.ID] = t.Backfill
}
return func(id string) time.Duration {
switch {
case explained[id]:
return 0
case backfill[id]:
return BackfillHoldNotifyAfter
default:
return HoldNotifyAfter
}
}
}

// taskNames maps task ids to their names, for the hold announcements.
func taskNames(cfg store.Config) map[string]string {
names := make(map[string]string, len(cfg.Tasks))
for _, t := range cfg.Tasks {
names[t.ID] = t.Name
}
return names
}

// announceHolds tells the operator about each task that waited too long, off
// the scheduler's lock.
func (e *Engine) announceHolds(long []longHold, names map[string]string) {
if e.notifier == nil || len(long) == 0 {
return
}
e.wg.Add(1)
go func() {
defer e.wg.Done()
for _, h := range long {
name := names[h.taskID]
if name == "" {
name = h.taskID
}
e.send(notify.Notification{
Kind: store.InboxKindWaiting,
Title: "DispatchQ: " + name + " is waiting",
Message: fmt.Sprintf("Waiting for %s: %s", h.waited.Round(time.Minute), h.reason),
TaskID: h.taskID,
TaskName: name,
})
}
}()
}
109 changes: 51 additions & 58 deletions internal/engine/alerts_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -222,32 +222,57 @@ func TestCancellationIsNotAnnounced(t *testing.T) {
}
}

// A due task that sits in the queue for long used to show up only in the log.
func TestLongWaitIsAnnouncedOncePerWait(t *testing.T) {
fc := clock.NewFake(time.Date(2026, 10, 5, 15, 40, 0, 0, time.UTC))
r := &stub{block: make(chan struct{})}
e, st := newTestEngine(t, r, fc)
e.holdLog = &bytes.Buffer{}
n := &capturingNotifier{}
e.SetNotifier(n)
saveTasks(t, st, asapTask("first", false), asapTask("second", true))

tick(t, e)
waitFor(t, func() bool { r.mu.Lock(); defer r.mu.Unlock(); return r.active == 1 })
fc.Advance(HoldNotifyAfter - time.Minute)
tick(t, e)
fc.Advance(time.Minute)
tick(t, e)
fc.Advance(time.Hour)
tick(t, e)
waitFor(t, func() bool { return len(n.all()) > 0 })
close(r.block)
e.WaitIdle()

msgs := n.all()
if len(msgs) != 1 || msgs[0].Kind != store.InboxKindWaiting || msgs[0].Title != "DispatchQ: second is waiting" ||
!strings.Contains(msgs[0].Message, "Waiting for 30m0s") {
t.Fatalf("pushes = %+v, want one waiting notice", msgs)
// Waiting remains visible in the queue and daemon log without interrupting the operator.
func TestLongWaitRemainsVisibleWithoutNotifications(t *testing.T) {
for _, backfill := range []bool{false, true} {
name := "regular"
if backfill {
name = "backfill"
}
t.Run(name, func(t *testing.T) {
fc := clock.NewFake(time.Date(2026, 10, 5, 15, 40, 0, 0, time.UTC))
r := &stub{block: make(chan struct{})}
e, st := newTestEngine(t, r, fc)
var log bytes.Buffer
e.holdLog = &log
n := &capturingNotifier{}
e.SetNotifier(n)
second := asapTask("second", true)
second.Backfill = backfill
saveTasks(t, st, asapTask("first", false), second)
tick(t, e)
waitFor(t, func() bool { r.mu.Lock(); defer r.mu.Unlock(); return r.active == 1 })
// Release the blocking fixture even if an assertion fails.
released := false
defer func() {
if !released {
close(r.block)
}
e.WaitIdle()
}()
since := fc.Now()
for _, elapsed := range []time.Duration{30 * time.Minute, 24 * time.Hour, 48 * time.Hour} {
fc.Advance(elapsed)
tick(t, e)
}
h, ok := e.Holds()["second"]
if !ok || h.Reason == "" || !h.Since.Equal(since) {
t.Fatalf("queue lost the waiting task: %+v, present %v", h, ok)
}
if !strings.Contains(log.String(), `task "second" has been due for`) {
t.Fatalf("missing diagnostic wait log: %q", log.String())
}
// Wait for any asynchronous notification goroutines before asserting.
close(r.block)
e.WaitIdle()
released = true
if msgs := n.all(); len(msgs) != 0 {
t.Fatalf("waiting pushed notifications: %+v", msgs)
}
if entries := inbox(t, st); len(entries) != 0 {
t.Fatalf("waiting added inbox entries: %+v", entries)
}
})
}
}

Expand Down Expand Up @@ -276,19 +301,6 @@ func TestExpectedWaitsAreNotAnnounced(t *testing.T) {
}
}

func TestBackfillWaitIsAnnouncedOnlyAfterADay(t *testing.T) {
limits := holdLimits(store.Config{Tasks: []task.Task{{ID: "b", Backfill: true}, {ID: "n"}}}, map[string]bool{"x": true})
if got := limits("b"); got != BackfillHoldNotifyAfter {
t.Errorf("backfill limit = %s", got)
}
if got := limits("n"); got != HoldNotifyAfter {
t.Errorf("normal limit = %s", got)
}
if got := limits("x"); got != 0 {
t.Errorf("explained limit = %s, want never", got)
}
}

// A run started in a wake without network failed at once, and so did its
// push. It now waits for the network instead.
func TestAgentTaskWaitsForTheNetwork(t *testing.T) {
Expand Down Expand Up @@ -391,25 +403,6 @@ func TestRunStartedBeforeThePauseDoesNotEndIt(t *testing.T) {
}
}

// Time held for a reason that is not announced does not count towards the
// waiting notice once the reason changes.
func TestExplainedWaitDoesNotCountTowardsTheNotice(t *testing.T) {
now := time.Date(2026, 11, 10, 9, 0, 0, 0, time.UTC)
var h holds
log := &bytes.Buffer{}
explained := func(string) time.Duration { return 0 }
unexplained := func(string) time.Duration { return HoldNotifyAfter }
h.update(map[string]string{"a": "No network connection"}, now, log, explained)
now = now.Add(3 * time.Hour)
if long := h.update(map[string]string{"a": "runs alone"}, now, log, unexplained); len(long) != 0 {
t.Fatalf("announced at once after an explained wait: %+v", long)
}
now = now.Add(HoldNotifyAfter)
if long := h.update(map[string]string{"a": "runs alone"}, now, log, unexplained); len(long) != 1 || long[0].waited != HoldNotifyAfter {
t.Fatalf("long = %+v, want one notice counting 30 minutes", long)
}
}

func TestAPIHostFollowsTheInstanceEnvironment(t *testing.T) {
cases := []struct {
kind provider.Kind
Expand Down
16 changes: 4 additions & 12 deletions internal/engine/engine.go
Original file line number Diff line number Diff line change
Expand Up @@ -341,11 +341,7 @@ func (e *Engine) Tick(ctx context.Context) error {
// held collects why each due task does not start this tick; whatever is
// left in it at the end is what the queue shows.
held := map[string]string{}
// explained collects the tasks held for a reason that is announced on its
// own (a provider pause, a provider that cannot run) or needs no telling
// (no network): their wait is not notified as well.
explained := map[string]bool{}
assigned := e.assign(ctx, providers, due, cfg.Settings.BackfillUrgencyOrDefault(), held, explained)
assigned := e.assign(ctx, providers, due, cfg.Settings.BackfillUrgencyOrDefault(), held)

// Their harnesses are then probed *outside* the lock. A readiness check
// spawns a CLI and a CLI can hang; holding e.mu across that would stall the
Expand Down Expand Up @@ -396,11 +392,9 @@ func (e *Engine) Tick(ctx context.Context) error {
continue
case !a.task.IsScript() && !health.ready(a.resolved):
held[a.task.ID] = health.reason(a.resolved)
explained[a.task.ID] = true
continue
case !a.task.IsScript() && !online.reachable(a.resolved):
held[a.task.ID] = "No network connection; it starts once " + a.resolved.Instance.Label() + " can be reached again."
explained[a.task.ID] = true
continue
}
runnable = append(runnable, a.task)
Expand All @@ -417,8 +411,7 @@ func (e *Engine) Tick(ctx context.Context) error {
overdue := schedule.Overdue(runnable, since, now)
selected := schedule.Select(runnable, e.runningState(), overdue)
maps.Copy(held, schedule.Held(runnable, e.runningState(), overdue))
long := e.holds.update(held, now, e.holdLog, holdLimits(cfg, explained))
e.announceHolds(long, taskNames(cfg))
e.holds.update(held, now, e.holdLog)
starts := make([]start, 0, len(selected))
for _, t := range selected {
starts = append(starts, start{task: t, resolved: ready[t.ID], deps: deps[t.ID]})
Expand Down Expand Up @@ -463,7 +456,7 @@ func (e *Engine) holdAll(cfg store.Config, reason string) {
for _, t := range due {
held[t.ID] = reason
}
e.holds.update(held, e.clock.Now(), e.holdLog, nil)
e.holds.update(held, e.clock.Now(), e.holdLog)
}

// allDue is dueTasks with the dependencies worked out first.
Expand Down Expand Up @@ -528,7 +521,7 @@ type assignment struct {
// (see provider.Selection.Backfill); while there is none it is dropped too.
//
// Why a dropped task does not start is filed in held, by task id.
func (e *Engine) assign(ctx context.Context, set provider.Set, due []task.Task, backfill float64, held map[string]string, explained map[string]bool) []assignment {
func (e *Engine) assign(ctx context.Context, set provider.Set, due []task.Task, backfill float64, held map[string]string) []assignment {
out := make([]assignment, 0, len(due))
// Which pool member holds a task's interrupted session is read once per
// pass. Without it a pool would pick by rank and drop the session.
Expand Down Expand Up @@ -571,7 +564,6 @@ func (e *Engine) assign(ctx context.Context, set provider.Set, due []task.Task,
if err == nil && !e.gateOpen(res.Instance.ID) {
held[t.ID] = fmt.Sprintf("%s is out of allowance until %s.",
res.Instance.Label(), reopenText(e.gates.For(res.Instance.ID).BlockedUntil(), e.clock.Now()))
explained[t.ID] = true
continue
}
if err == nil {
Expand Down
46 changes: 2 additions & 44 deletions internal/engine/holds.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,37 +24,17 @@ type holds struct {
// logged is the reason last written to the log per task, so a long wait is
// logged once, and again only when its reason changes.
logged map[string]string
// announced holds the tasks whose current wait was notified, so a wait is
// announced once however long it goes on. counting is since when each
// task has waited for a reason worth announcing: time held for one that is
// not (no network, out of allowance) does not count towards the limit.
announced map[string]bool
counting map[string]time.Time
}

// longHold is a wait that has just gone on long enough to be announced.
type longHold struct {
taskID string
reason string
waited time.Duration
}

// update replaces the holds with one tick's verdicts. A task that keeps
// waiting keeps its start time even when the reason changes, because to the
// person looking at the queue it is still the same wait. Tasks missing from
// reasons started, or are not due any more, and are forgotten.
//
// limit says how long each task may wait before its wait is announced, zero
// for never (nil: nothing is announced); the waits that just crossed it are
// returned, each once per wait.
func (h *holds) update(reasons map[string]string, now time.Time, log io.Writer, limit func(id string) time.Duration) []longHold {
func (h *holds) update(reasons map[string]string, now time.Time, log io.Writer) {
h.mu.Lock()
defer h.mu.Unlock()
next := make(map[string]schedule.Hold, len(reasons))
logged := make(map[string]string, len(reasons))
announced := make(map[string]bool, len(h.announced))
counting := make(map[string]time.Time, len(h.counting))
var long []longHold
for id, reason := range reasons {
since := now
if prev, ok := h.byTask[id]; ok {
Expand All @@ -70,30 +50,8 @@ func (h *holds) update(reasons map[string]string, now time.Time, log io.Writer,
id, waited.Round(time.Minute), reason)
logged[id] = reason
}
if h.announced[id] {
announced[id] = true
continue
}
after := time.Duration(0)
if limit != nil {
after = limit(id)
}
if after <= 0 {
continue
}
from, ok := h.counting[id]
if !ok {
from = now
}
counting[id] = from
if counted := now.Sub(from); counted >= after {
fmt.Fprintf(log, "dispatchqd: task %q has waited %s; notifying\n", id, counted.Round(time.Minute))
long = append(long, longHold{taskID: id, reason: reason, waited: counted})
announced[id] = true
}
}
h.byTask, h.logged, h.announced, h.counting = next, logged, announced, counting
return long
h.byTask, h.logged = next, logged
}

// snapshot returns a copy of the current holds.
Expand Down
Loading