Recurring schedules: durable last-run state (LastRunAt / LastStatus / LastError)

[jeremydmiller]

Split out of #4437 (GAP-1). This is the contentious third of that issue and deliberately separated so it can be argued on its own — the attribution and trigger pieces do not depend on it.

The ask

A per-schedule record of the last run: LastRunAt, LastStatus, LastError (history explicitly not included) on the tracking row and on RecurringScheduleDescriptor, so a console can render a status badge and decide whether to offer a re-trigger without querying an external tracing backend.

The standing position

From the #4437 thread: run state is obtainable from OpenTelemetry data, and that was a deliberate choice rather than an oversight. The counter-argument in the issue is that OTel is sampled, retention-limited and lives in a backend a console cannot cheaply query per schedule, whereas "did the last run fail?" is one small durable fact.

That trade is a product decision. What follows is only what the code says it would cost.

Structural problems, beyond effort

1. The agent cannot be the writer. Its involvement ends at publish (RecurringMessageAgent.publishOccurrenceAsync / recordPublishAsync, RecurringMessageAgent.cs:382-484); afterwards it only counts whether the envelope is still Scheduled (RdbmsRecurringMessageStore.cs:179-198). The occurrence is handled later, through the normal inbox pipeline, on whatever node claims it. So the writer has to be a message-pipeline seam — IMessageTracker is the central one (src/Wolverine/Logging/IMessageLogger.cs:53,59,66,86), with three runtime implementations that would need to stay in sync if hooked below the interface. (IWireTap is opt-in per endpoint, ListenerConfiguration.cs:586, so it is not a default-on hook; IWolverineObserver is node/agent lifecycle only and not a fit.)

2. The handling node may have nowhere to write. IMessageStore.RecurringMessages is built Main-store-only (MessageDatabase.cs:172-176; interface default IMessageStore.cs:157). A node handling the occurrence against an ancillary store gets NullRecurringMessageStore, so last-run state would silently not be recorded in exactly the modular-monolith setups where per-schedule visibility is most wanted. This needs an answer before the feature is coherent.

3. It puts a write on the handling hot path. One additional store round-trip per occurrence handled, on every schedule, forever.

4. Terminal-path discipline. There must be exactly one terminal record per path — the convergence points are MessageSucceededContinuation.cs:26, MoveToErrorQueue.cs:69, NoHandlerContinuation.cs:55-57 and HandlerPipeline.cs:160, and the GH-4136 reasoning is documented at MoveToErrorQueue.cs:63-68. A naive writer double-writes.

Schema notes, if it does go ahead

Adding nullable columns is routine: the recurring tables are plain Weasel Table objects yielded from AllObjects() (PostgresqlMessageStore.cs:922-925 and siblings), with no hand-written migrations, and there is in-family precedent for conditional columns at DeadLettersTable.cs:46-48. Current columns are schedule_name, cron_expression, envelope_ids, dedup_id, next_occurrence, paused, paused_at, last_updated (DatabaseConstants.cs:91-99).

Non-schema work in RdbmsRecurringMessageStore: selectFields (:63-66), the ordinal-based reader readRecordAsync (:303-322), and the SQL at :79-103. Note that _updatePublishSql deliberately preserves pause state — a last-run write needs the mirror-image discipline of not clobbering publish bookkeeping. MessageDatabase.Admin.cs:314-317 clears the table on ClearAll.

Oracle, RavenDb and CosmosDb have no recurring table at all and inherit NullRecurringMessageStore, so they would report no last-run state.

Suggested decision order

Settle whether a small durable last-run fact is wanted at all given the OTel position, and if so how the ancillary-store hole (2) is closed. The rest is mechanical.

🤖 Filed with Claude Code from the #4437 analysis

[jeremydmiller]

Closing: run state stays in OpenTelemetry, and the operability view belongs in CritterWatch (JasperFx/CritterWatch#1246) rather than in Wolverine's durable tracking row.

Recording the reasoning so this doesn't get re-litigated from scratch:

• The agent can't be the writer. Its involvement ends at publish; the occurrence is handled later, on whatever node claims the envelope. So a last-run write would have to hang off a message-pipeline seam (IMessageTracker), not the recurring agent.
• IMessageStore.RecurringMessages is Main-store-only (MessageDatabase.cs:172-176). A node handling the occurrence against an ancillary store gets NullRecurringMessageStore and would have nowhere to write — silently, in exactly the modular-monolith setups that most want per-schedule visibility.
• It would also put an extra store round-trip on the handling hot path for every occurrence, forever.

What core still owes for the OTel path to actually work is #4445 — the occurrence-time header/tag and metrics attribution. Today ToMetricsHeaders omits the schedule, so no counter can be sliced by schedule, and the occurrence instant is recoverable only by string-parsing the dedup id. That attribution is the prerequisite for a console answering "did the last run fail?" from tracing data.

#4446 (manual trigger) is unaffected and still stands on its own.