GH-4437: recurring schedule operability — occurrence attribution and a manual trigger

[jeremydmiller]

Closes #4437
Closes #4445
Closes #4446

The two remaining halves of the recurring/cron operability issue, in one PR. (#4447, durable
last-run state, was closed as not-planned earlier: run state stays in OpenTelemetry and the
operability view belongs in CritterWatch.)

GH-4445 — attribution

Occurrences already carried their schedule's name in a recurring-schedule header, lifted onto the
handler span as wolverine.schedule.name. Two gaps remained.

The occurrence instant was not on the envelope as a first-class value. Envelope.ScheduledTime
is cleared by the scheduled machinery at fire time, so by the time a handler runs, the only record of
which firing it is serving was the deduplication id — which exists for deduplication and had to be
string-parsed to serve as attribution. Occurrences now carry a recurring-occurrence header (UTC,
round-trippable "O"), lifted onto the span as wolverine.schedule.occurrence.

Metrics carried no schedule attribution, so the success / failure / effective-time counters could
not be sliced per cron job. ToMetricsHeaders() now emits a schedule.name tag.

It reads the header rather than the SetMetricsTag seam the issue proposed, and that detail is
the point: the metric tag list is a private field that is never serialized, so an occurrence
published on one node and handled on another would reach the counters on the handling node with no
attribution at all. The header round-trips every transport — pinned by a serializer round-trip test
that asserts both tags and the metric tag still resolve on the deserialized envelope.

The occurrence instant is deliberately trace-only. One distinct value per firing would make the
metric series unbounded in cardinality; a test pins that it stays out of the tag set.

GH-4446 — a manual trigger

TriggerAsync records the request on the schedule's durable tracking row (new nullable
trigger_requested_at column) and the agent publishes one occurrence for it on its next pass —
following the dead-letter replay precedent, because the caller is usually not on the node running the
agent.

Two semantics, both deliberate:

• A manual run carries its own deduplication id, {schedule}:manual:{requested:O}. Dedup exists
  to collapse a failover double-publish of the same occurrence; a "run now" is a separate intent,
  and inheriting the occurrence id would make a trigger issued in the same instant as a scheduled
  firing silently do nothing. Still derived from the request instant rather than a random value, so
  failover idempotence is preserved.
• Triggering a paused schedule is refused with RecurringSchedulePausedException. Pause says the
  schedule must not fire. Refusing is also the future-compatible choice: relaxing it later is
  additive, restricting it later would break callers. The refusal is the and paused = false
  predicate on the update itself, so the check and the write are one atomic statement rather than a
  read-then-write race.

The trigger is honoured before the cron computation, so a fixed-date schedule whose occurrences
have run out still runs on demand, and it never touches the pending-occurrence bookkeeping — a manual
run is extra, never a replacement for the cadence. The request clears only after a successful
publish. One slot per schedule, so two triggers landing between passes coalesce into a single run.

The two new IRecurringMessageStore members are defaulted rather than abstract, matching that
interface's stated policy, so an out-of-tree store that predates the verb keeps compiling.

Verification

• Red baseline first on the Recurring schedules: attribute occurrences to their schedule in metrics, and carry the occurrence time #4445 half: three tests failing for the right reasons (header absent,
  schedule.name missing from the metrics dictionary, trace tag null), then green.
• CoreTests recurring / tracing / serialization: 72/72.
• RecurringMessageCompliance gains a durable-trigger fact, so it is proven on PostgreSQL, SQL
  Server, MySQL and SQLite at once.
• Full wolverine.slnx -c Release -f net9.0: clean.

Per repo convention CHANGELOG.md is untouched; the release notes carry this.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VDUrBeB4tTnKj4AExCS1nj