Notify before patching: end-user notifications platform (delivery, device API, outcomes)

[cdcme]

Related user story

#39178

Task

Build the delivery half of end-user notifications as a reusable platform, with no patch-specific logic. Core enqueues a message for a host, dispatches it to Fleet Desktop, confirms it was displayed, maps failures to causes, retries, and serves the page the toast renders. What the message says, when it is sent, and what happens next belong to a kind, added in the next sub-issue.

The next consumer is already filed: #44672 asks for the same delivery path for any failing policy.

Naming

notifications is taken twice, both flag-style: fleet.OrbitConfigNotifications (server/fleet/orbit.go) and fleet.DesktopNotifications (server/fleet/device.go:18). Use end_user_notifications in the database and Go. Use /device/{token}/notifications/{uuid} in the device API, where the audience is implicit.

Migration

CREATE TABLE end_user_notifications (
  id                INT UNSIGNED NOT NULL AUTO_INCREMENT,
  uuid              VARCHAR(36) NOT NULL,
  host_id           INT UNSIGNED NOT NULL,
  kind              VARCHAR(63) NOT NULL,
  payload           JSON NOT NULL,
  status            VARCHAR(31) NOT NULL,
  attempt_count     INT UNSIGNED NOT NULL DEFAULT 0,
  next_attempt_at   TIMESTAMP NULL,
  execution_id      VARCHAR(255) NULL,
  last_exit_code    INT NULL,
  last_reason       VARCHAR(63) NULL,
  displayed_at      TIMESTAMP NULL,
  expires_at        TIMESTAMP NULL,
  created_at        TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
  updated_at        TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (id),
  UNIQUE KEY idx_eun_uuid (uuid),
  KEY idx_eun_dispatch (status, next_attempt_at),
  KEY idx_eun_host (host_id, status),
  CONSTRAINT fk_eun_host_id FOREIGN KEY (host_id) REFERENCES hosts (id) ON DELETE CASCADE
);

Create it with make migration name=CreateEndUserNotifications. The Down function is a no-op, per convention.

One row per notification, not per attempt. A reminder re-dispatches the same row with an updated payload. This is what makes a reminder reuse the notification's ID, which the audit log documents.

expires_at is not a deadline. It means "stop trying to deliver." A kind's deadline means "act." Keeping them separate stops a delivery failure from silently cancelling whatever the kind was going to do.

Types

New file server/fleet/end_user_notifications.go:

type EndUserNotification struct {
	ID           uint       `db:"id"`
	UUID         string     `db:"uuid"`
	HostID       uint       `db:"host_id"`
	Kind         string     `db:"kind"`
	Payload      json.RawMessage `db:"payload"`
	Status       NotificationStatus `db:"status"`
	AttemptCount uint       `db:"attempt_count"`
	NextAttemptAt *time.Time `db:"next_attempt_at"`
	ExecutionID  *string    `db:"execution_id"`
	LastExitCode *int64     `db:"last_exit_code"`
	LastReason   *NotificationReason `db:"last_reason"`
	DisplayedAt  *time.Time `db:"displayed_at"`
	ExpiresAt    *time.Time `db:"expires_at"`
}

type NotificationStatus string

const (
	NotificationPending    NotificationStatus = "pending"
	NotificationDispatched NotificationStatus = "dispatched"
	NotificationDisplayed  NotificationStatus = "displayed"
	NotificationFailed     NotificationStatus = "failed"
	NotificationExpired    NotificationStatus = "expired"
	NotificationActed      NotificationStatus = "acted"
)

type NotificationReason string

const (
	ReasonDeferred            NotificationReason = "deferred"
	ReasonNoGUIUser           NotificationReason = "no_gui_user"
	ReasonScreenLocked        NotificationReason = "screen_locked"
	ReasonNoDisplay           NotificationReason = "no_display"
	ReasonPageLoadFailed      NotificationReason = "page_load_failed"
	ReasonHTTPError           NotificationReason = "http_error"
	ReasonInternalError       NotificationReason = "internal_error"
	ReasonFleetDesktopMissing NotificationReason = "fleet_desktop_missing"
	ReasonFleetDesktopTooOld  NotificationReason = "fleet_desktop_too_old"
	ReasonBadInvocation       NotificationReason = "bad_invocation"
	ReasonBadConfiguration    NotificationReason = "bad_configuration"
)

// NotificationView is what the device page draws. It is data-driven, so a new
// kind ships without new React.
type NotificationView struct {
	Title       string               `json:"title"`
	Description string               `json:"description"`
	LogoURL     string               `json:"logo_url,omitempty"`
	Items       []NotificationItem   `json:"items"`
	Actions     []NotificationAction `json:"actions"`
}

type NotificationItem struct {
	Name    string `json:"name"`
	IconURL string `json:"icon_url,omitempty"`
	Status  string `json:"status,omitempty"` // e.g. "Installing..."
}

type NotificationAction struct {
	ID    string `json:"id"`    // "update_now", "remind", "dismiss"
	Label string `json:"label"`
	Style string `json:"style"` // "primary" | "secondary"
}

type NotificationOutcome struct {
	Displayed bool
	ExitCode  int64
	Reason    NotificationReason
	Output    string
	ExecutionID string
}

type NotificationKind interface {
	Name() string
	Render(ctx context.Context, n *EndUserNotification) (*NotificationView, error)
	OnOutcome(ctx context.Context, n *EndUserNotification, out NotificationOutcome) error
	OnAction(ctx context.Context, n *EndUserNotification, action string) error
}

Why a NotificationView rather than per-kind React. The patch toast is a title, a description, a list of apps, and two buttons. The policy-failure toast in #44672 is a title, a description, a list of policies, and two buttons. Rendering is data-driven; only outcomes and actions branch by kind.

Why activities stay with the kind. Fleet's activity copy is specific ("Fleet notified end user 1 hour before patching 1Password on Anna's MacBook Pro"), and the audit log lists one entry per type. A generic end_user_notification_displayed would read as noise. Core hands OnOutcome a filled NotificationOutcome so the kind's activity is a struct literal.

NotificationReason is internal. It drives the retry policy below: a missing Fleet Desktop needs admin action and gets a long backoff, a locked screen gets a short one. It is deliberately not surfaced in any activity payload. The frontend derives its copy from the script's exit code instead, fetched with GET /api/v1/fleet/scripts/results/{execution_id}. Keep ExecutionID on NotificationOutcome so the kind can put it in its activity.

Datastore

Add to server/fleet/datastore.go, then run make generate-mock:

NewEndUserNotification(ctx context.Context, n *EndUserNotification) (*EndUserNotification, error)
GetEndUserNotificationByUUID(ctx context.Context, uuid string) (*EndUserNotification, error)
ListEndUserNotificationsToDispatch(ctx context.Context, limit int) ([]*EndUserNotification, error)
SetEndUserNotificationDispatched(ctx context.Context, uuid, executionID string) error
SetEndUserNotificationOutcome(ctx context.Context, uuid string, out NotificationOutcome, nextAttemptAt *time.Time) error
RedispatchEndUserNotification(ctx context.Context, uuid string, payload json.RawMessage) error
GetEndUserNotificationByExecutionID(ctx context.Context, executionID string) (*EndUserNotification, error)
ExpireEndUserNotifications(ctx context.Context) (int64, error)

ListEndUserNotificationsToDispatch returns status = 'pending' AND next_attempt_at <= NOW() for hosts with no unfinished notify script. Serialize on dispatch, not on lifetime: several notifications can be alive on one host with independent deadlines, but only one toast can be on screen.

This is one of two places "another notification was displayed" is handled. A host that already has an unfinished notify execution is skipped, and the deferred notification records last_reason = 'deferred' with no script run.

Marko also added a notify exit code for the same condition on 2026-08-11, so the binary reports it when it beats the dispatcher to the check. Map that code to ReasonDeferred as well. Keep both: the server-side check avoids a pointless round-trip, and the binary is authoritative about what is on screen. Get the exit code number from Marko, and note it needs a Fleet Desktop release beyond 1.5.0.

Dispatcher cron

Add CronEndUserNotifications CronScheduleName = "end_user_notifications" to server/fleet/cron_schedules.go, and register a schedule in cmd/fleet/cron.go following newRecoveryLockPasswordSchedule at cmd/fleet/cron.go:2652. Interval 1 minute, so a reminder fires within a minute of its target.

Each pass:

1. Expire notifications past expires_at.
2. For each dispatchable notification, render the wrapper script and enqueue it with ds.NewInternalHostScriptExecutionRequest (server/datastore/mysql/scripts.go:47). Internal scripts run even when scripts are disabled for the fleet, and stay out of the host activity feed.
3. Record status = 'dispatched' and the execution ID.

The wrapper script is the one in this comment, with the URL substituted. It checks that Fleet Desktop is installed (exit 100), that FleetDesktopCapabilities.notify is present in Info.plist (exit 101), and that someone is logged in at the GUI (exit 40), then runs the binary as the console user.

Resolve open question 5 before writing the URL. Interpolating the host's device auth token into script_contents stores a live credential and breaks if the token rotates mid-countdown.

Outcome handling

Hook into SaveHostScriptResult at server/service/orbit.go:1220. When the execution ID belongs to a notification, map the exit code and call the kind's OnOutcome:

Exit code	Status	Reason	Retry
0	displayed		no
40, 41, 42	failed	no_gui_user, screen_locked, no_display	yes, short backoff
30, 31, 70	failed	page_load_failed, http_error, internal_error	yes, short backoff
100, 101	failed	fleet_desktop_missing, fleet_desktop_too_old	yes, long backoff
2, 20	failed	bad_invocation, bad_configuration	no, log at error
deferred code, number TBC	failed	deferred	yes, short backoff

Codes come from apps/fleet-desktop-macos/FleetDesktop/cli.swift:8-17. Every code above except the last ships in Fleet Desktop 1.5.0. The deferred code is new and needs a Fleet Desktop release; treat that as a dependency, not as work in this story.

Why 100 and 101 retry on a long backoff. Both need an admin to deploy or upgrade Fleet Desktop. Retrying every minute would fill the activity feed with the same failure.

Device API

Register in server/service/handler.go beside the existing device routes at lines 940 to 961, using errorLimiter and the same device-auth middleware:

de.WithCustomMiddleware(errorLimiter).GET("/api/_version_/fleet/device/{token}/notifications/{uuid}",
    getDeviceNotificationEndpoint, getDeviceNotificationRequest{})
de.WithCustomMiddleware(errorLimiter).POST("/api/_version_/fleet/device/{token}/notifications/{uuid}/actions",
    deviceNotificationActionEndpoint, deviceNotificationActionRequest{})

GET returns the kind's NotificationView. POST takes {"action": "update_now"} and calls the kind's OnAction. Both verify the notification belongs to the authenticating host. Follow the self-service install endpoints at server/service/handler.go:951-956 for shape.

Why no list endpoint. The toast opens one notification by UUID. A "show everything failing on this host" view was cut from this story and moved to a follow-up (see #50916).

expires_at is optional, and the patch kind leaves it null. A patch policy keeps failing until the app is updated, so Fleet keeps retrying and there is no natural expiry. This also keeps the deferred case unambiguous downstream: a deferral is then the only outcome that emits an activity without a script run.

Kind registry

A map[string]NotificationKind built at service construction. Three methods, no dynamic registration. A switch on kind inside core would put patch knowledge back in core, which is what this split exists to prevent.

Condition of satisfaction

Migration

• MYSQL_TEST=1 go test ./server/datastore/mysql/migrations/... passes, and schema.sql is regenerated.
• Deleting a host cascades its notifications.

Dispatch and serialization

• A pending notification is dispatched, and status becomes dispatched with a non-null execution_id.
• A second pending notification on the same host is not dispatched while the first execution is unfinished. It records last_reason = 'deferred' and stays pending.
• Two notifications on different hosts dispatch in the same pass.
• A notification past expires_at becomes expired and is never dispatched.
• The enqueued script is internal: it does not appear in the host activity feed, and it runs on a host with scripts disabled.

Outcome mapping

• Each of exit codes 0, 2, 20, 30, 31, 40, 41, 42, 70, 100, and 101 maps to the status and reason in the table above. Cover all eleven.
• Exit 0 sets displayed_at and calls OnOutcome with Displayed: true.
• Exit 2 and 20 do not schedule a retry.
• Exit 100 and 101 schedule a longer next_attempt_at than exit 41.
• A script result whose execution ID is not a notification is unaffected.

Re-dispatch

• RedispatchEndUserNotification keeps the same uuid, replaces the payload, and returns the row to pending.

Device API

• GET /device/{token}/notifications/{uuid} returns the kind's view.
• A UUID belonging to another host returns 404, not the other host's content.
• An unknown UUID returns 404.
• POST .../actions with an unknown action returns 422.
• POST .../actions calls the kind's OnAction.

Tests

• MYSQL_TEST=1 go test ./server/datastore/mysql/...
• MYSQL_TEST=1 REDIS_TEST=1 go test ./server/service/...
• go test ./server/fleet/...
• Run go test ./server/service/ after the datastore interface changes. Uninitialized mocks crash unrelated tests.