Recurring/cron schedule operability: run-state, occurrence attribution, and manual trigger (core surface)

[mysticmind]

Core primitives to make recurring/cron schedules operable: attribute occurrences back to their
schedule, record last-run outcome, and allow a manual "run now". These are the framework-side pieces
that a monitoring console (e.g. CritterWatch) then surfaces -- see Related issues below.

Environment

WolverineFx 6.34.0, Marten 9.33.0 on PostgreSQL, RabbitMQ, .NET 9. Observed while running a sample of
40 recurring schedules (full cron grammar + time zones) with three deliberately-failing schedules.

What exists today (verified against source)

Per-schedule surface -- RecurringScheduleDescriptor
(Wolverine/Configuration/Capabilities/RecurringScheduleDescriptor.cs):

Name, CronExpression, TimeZoneId, MessageType, NextOccurrence, Paused, PausedAt

Runtime control -- IRecurringScheduleControl
(Wolverine/Runtime/Recurring/RecurringScheduleControl.cs):

PauseAsync(name), ResumeAsync(name), StatusesAsync()   // pause state + next occurrence only

Tracking row -- RecurringMessageRecord (Wolverine/Persistence/Durability/IRecurringMessageStore.cs):

Name, CronExpression, EnvelopeIds, DeduplicationId, NextOccurrence, Paused, PausedAt

So: current + next occurrence and pause/resume exist. Nothing about the actual runs.

Gaps (core)

GAP-1 -- No per-run execution record (last run time / status / error)

No LastRunAt, LastStatus, LastError, LastDurationMs, or success/failure on the descriptor or
tracking row. Verified behaviorally: FailingHealthProbe fires every 20s and throws every time, yet
nothing on the schedule reflects that its last occurrence failed.

GAP-3 -- No manual re-trigger ("run now")

IRecurringScheduleControl exposes only Pause/Resume/query. There's no "trigger now / run once" verb.
Today you'd hand-publish the message type out of band, which also bypasses the occurrence/dedup
machinery.

GAP-4 -- Occurrences aren't attributed to their schedule

A published occurrence carries its message type but not the schedule name or the scheduled
occurrence time. So a failure/DLQ entry -- or an OTel span -- can't be tied back to which schedule /
which scheduled run produced it, especially when two schedules share a message type or a type is
also sent ad hoc. This is the prerequisite for GAP-1 and for OTel correlation.

Run history (GAP-2) and full spans are intentionally left to OpenTelemetry -- see the design
discussion. Core only needs last-run state + attribution + trigger.

Design discussion (from the thread)

The ask (paraphrased): record the state/error of each run at the cron-job level (rather than
looking it up in the DLQ), keep a history of runs, show if the last run failed, and allow manual
re-trigger -- typical schedule-job use cases.

Jeremy's responses:

• "Wonder if it would help to move them to being sagas. Hmm."
• "There is already some of that. I purposely didn't try to persist the state of the run because we
  can get that off of open-telemetry data."
• "Yup. Wanna keep a list of all of this on an issue?"

Trade-offs:

• OTel as the source of run state is great for dashboards/tracing but limited as the authoritative
  "did the last run fail?": it's often sampled, retention-limited, and lives in an external
  backend a console can't cheaply query per schedule. A small per-schedule "last run:
  {time, status, error}" record is cheap and directly drives a status badge + the re-trigger decision
  -- without persisting full history. Reasonable split: last-run = tiny durable record;
  history/spans = OTel.
• Sagas: a saga per schedule could hold last-run/next-run/pause + a bounded recent-runs ring and
  expose trigger/pause/resume as saga messages -- a natural fit for per-schedule state + control. A
  saga per occurrence is likely too heavy (churn) unless a run genuinely spans steps/time. Worth
  prototyping against the descriptor/tracking-row approach.

Suggested minimal, additive direction

1. Attribution: stamp each occurrence envelope with schedule-name + occurrence-time headers
   (small, no new storage; enables GAP-4 and OTel correlation).
2. Last-run state: add LastRunAt / LastStatus / LastError to the recurring tracking row and
   RecurringScheduleDescriptor (last-run only, not history).
3. Trigger: add TriggerAsync(name) to IRecurringScheduleControl that publishes one occurrence
   through the normal path (respecting occurrence/dedup).
4. History/spans remain in OTel.

Reproduction

Self-contained sample available: CronDemoService (40 recurring schedules across the full cron grammar

• four time zones + three deliberately-failing schedules).

Related issues

• Blocked-by / relates-to: Non-UTC durable recurring schedules crash on the Postgres timestamptz write #4436 (non-UTC durable recurring timestamptz
  write bug) -- the durable path that would carry the new LastRun* fields is currently broken for
  non-UTC schedules.
• Enables: JasperFx/CritterWatch#1246 (recurring-schedule operability drill-down, console UX) --
  the console view that surfaces last-run, history, trigger, and failure attribution depends on the
  core surface proposed here.

[jeremydmiller]

Split into three issues so each can be scheduled and argued independently:

• Recurring schedules: attribute occurrences to their schedule in metrics, and carry the occurrence time #4445 — attribution (GAP-4)
• Recurring schedules: add a manual trigger ("run now") to IRecurringScheduleControl #4446 — manual trigger (GAP-3)
• Recurring schedules: durable last-run state (LastRunAt / LastStatus / LastError) #4447 — durable last-run state (GAP-1)

Two findings from reading the current source that change the framing here:

GAP-4 is largely already shipped. The schedule name is stamped on every occurrence (RecurringMessageAgent.cs:398) and already lifted onto the handler activity as wolverine.schedule.name (Envelope.Internals.cs:476-479, WolverineTracing.cs:183). It is a non-reserved header, so it round-trips every transport and is on the span on whichever node handles the occurrence. What is genuinely missing is narrower: the occurrence time as a first-class header/tag (today recoverable only by string-parsing the dedup id), and metrics attribution — ToMetricsHeaders omits it, so no counter can be sliced by schedule.

The dead-letter claim is half-true. The schedule name is durably in every DLQ row, inside the serialized body blob, readable per-row via DeadLetterEnvelope.Envelope.Headers. What is missing is server-side filtering: DeadLetterEnvelopeQuery has no header predicate. So "which schedule produced this dead letter" is answerable today; "show me all dead letters from schedule X" is not.

GAP-1 has a structural blocker, not just a cost. The agent publishes but never observes handling, so the writer has to be a pipeline seam (IMessageTracker). But IMessageStore.RecurringMessages is Main-store-only (MessageDatabase.cs:172-176), so a node handling the occurrence against an ancillary store gets NullRecurringMessageStore and has nowhere to write — silently, in exactly the modular-monolith setups where per-schedule visibility is most wanted. That needs an answer before the feature is coherent, which is why it is split out rather than bundled.

GAP-3 has a clean precedent: dead-letter replay is a durable flag flip that the owning agent picks up, which is the house answer to the same cross-node problem that already makes PauseAsync go through the store rather than the local agent.

No PR yet — #4447 in particular turns on whether the "run state comes from OpenTelemetry" position should change, which is a product call.

Note that #4436, listed as blocking this, is fixed in #4444.