Leader assigns the wolverinedb durability agent to nodes with DurabilityAgentEnabled = false — assignment never converges and outgoing recovery never runs

[beriniwlew]

Summary

In a Balanced cluster, the leader assigns the wolverinedb://… durability agent to nodes running Durability.DurabilityAgentEnabled = false. Such a node cannot start the agent (ArgumentOutOfRangeException: Unrecognized agent scheme 'wolverinedb'), the leader re-issues the identical assignment forever, and — because no durability agent ends up running anywhere for that store — owner_id = 0 outgoing envelopes are never recovered. All queue tables read zero the whole time, so the failure is silent. We found a 9-day backlog of undelivered envelopes in a staging environment this way.

Observed on 6.17.3 and 6.24.4; from reading main, the relevant path looks unchanged.

Setup

Two-node cluster on PostgreSQL persistence + the PostgreSQL queue transport:

• Node A (worker/consumer): durability enabled, ListenToPostgresqlQueue(...).ExclusiveNodeWithParallelism(...).
• Node B (web producer): stages envelopes inside caller-owned EF transactions (so they land in wolverine_outgoing_envelopes with owner_id = 0 and rely on durability-agent recovery), no listeners, conventional discovery disabled, and Durability.DurabilityAgentEnabled = false — the intent being "this node stages but runs no recovery/scheduled agents".

Observed behavior

1. Node A happened to hold leadership. The leader assigned the wolverinedb://postgresql/<host>/<db>/wolverine agent to node B.
2. Node B throws on the StartAgent command:

   System.ArgumentOutOfRangeException: Unrecognized agent scheme 'wolverinedb' (Parameter 'uri')
      at Wolverine.Runtime.Agents.NodeAgentController.startWithRetriesAsync(Uri agentUri)
      at Wolverine.Runtime.Agents.NodeAgentController.StartAgentAsync(Uri agentUri)
      at Wolverine.Runtime.Agents.StartAgent.ExecuteAsync(...)
   

   and the envelope moves to the error queue; the leader logs Unable to confirm that agent wolverinedb://… started on node … and Giving up on a batched agent command after no progress for 00:05:00.
3. The leader re-evaluates every ~5 minutes and makes the same assignment again (wolverine_node_records shows an AssignmentChanged row every ~5m10s, indefinitely). It never converges to the capable node.
4. Result: no durability agent runs for the store → CheckRecoverableOutgoingMessagesOperation never executes → producer-staged envelopes sit in wolverine_outgoing_envelopes (owner_id = 0) forever, while the queue tables stay empty.

Why (from source)

• A node with DurabilityAgentEnabled = false never registers the store as an agent family, so it genuinely cannot start the agent (NodeAgentController ctor only does _agentFamilies[_runtime.Stores.Scheme] = _runtime.Stores when the flag is true).
• Node capabilities correctly exclude the scheme in that case (StartLocalProcessing populates WolverineNode.Capabilities from the registered families' SupportedAgentsAsync), so the leader has the data to avoid the node.
• But the assignment path for the store family doesn't consult capabilities: MessageStoreCollection.EvaluateAssignmentsAsync uses DistributeEvenly (6.x) / DistributeEvenlyWithAffinity (main, Database affinity is per agent family, not per database: 73% of shard databases have durability and projections on different nodes #3785, falling back to the even spread when there are no projection agents), and both spread across all grid nodes. The remainder pass even prefers non-leader nodes (_nodes.FirstOrDefault(x => !x.IsLeader …)), which in a two-node cluster deterministically targets the incapable producer whenever the consumer is leader.
• Capability matching (AllNodesHaveSameCapabilities / MatchAgentsToCapableNodesFor) exists in DistributeByGroupAffinity and DistributeEvenlyWithBlueGreenSemantics, but not in the plain even spread used for wolverinedb agents.

Repro sketch (deterministic, single node)

A single-node Balanced cluster with DurabilityAgentEnabled = false reproduces it without any second node: the node elects itself leader, assigns the durability agent to itself, throws Unrecognized agent scheme, and an envelope staged into wolverine_outgoing_envelopes with owner_id = 0 is never recovered into the queue. (That's how we pinned it in a regression test.)

Expected

Either of these would resolve it:

• The leader skips nodes that don't advertise the agent's scheme in their capabilities when distributing wolverinedb agents (mirroring what the blue/green and group-affinity paths already do) — related in spirit to Can a node take event-subscription agents but not durability agents? (warming a new projection version before it serves traffic) #3746 / Add Durability.MessagingEnabled for event-subscription-only nodes (GH-3746) #3747, which scoped the inverse case.
• Or DurabilityAgentEnabled = false is documented as unsupported for nodes participating in a Balanced cluster whose store has durability work.

Workaround

We removed DurabilityAgentEnabled = false from the producer nodes, so any node the leader picks can run the agent. That's safe in practice because a Main-role store's cluster-assigned agent doesn't auto-start scheduled-job polling (GH-3376/GH-3439) and incoming recovery no-ops for endpoints without a local listener — but it does mean the durability settings (e.g. KeepAfterMessageHandling) must be kept identical across all nodes, since the agent runs with whichever node hosts it.

Thanks for Wolverine — happy to provide more logs or turn the single-node repro into a PR-able failing test if useful.

[jeremydmiller]

@beriniwlew Why are you doing this? What's the point? It make sense, but it's not anything I ever expected to be done.

[beriniwlew]

Fair question! The context is a modular monolith (practice-management app) deployed as several differently-shaped hosts sharing one PostgreSQL database and one Wolverine store:

• three HTTP hosts (practitioner web app, public booking site, an API host) that only stage messages — the transactional-outbox write happens inside their own EF Core transaction (e.g. a public booking inserts the appointment and stages its notification emails atomically), and
• one worker host that owns all execution: it's the only node with listeners/handlers, and the queue listener is ExclusiveNodeWithParallelism + message partitioning so same-aggregate handling stays strictly ordered on one node.

The point of the split: web-facing processes never execute background side effects (email dispatch lives in exactly one place, and an exclusive listener keeps ordering trivial), while staging stays atomic with the business write. DurabilityAgentEnabled = false on the web hosts looked like the way to say "stages durably, runs no background agents" — which is what produced this report: those nodes still join the Balanced cluster, and the leader kept handing them the one agent they can't run.

Since filing this we hit the same distribution behavior on a second scheme: a worker restart during a deploy window (producer nodes present) had the leader assign the exclusive listener (wolverine-listener://…) to a producer node that never configured that endpoint — same no-progress loop, this time bouncing between the two producers every ~5 minutes while messages sat in the queue table. Both are worked around locally now (durability left enabled everywhere per the issue; plus an IAgentFamilySource transport that mirrors ExclusiveListenerFamily but distributes with DistributeEvenlyWithBlueGreenSemantics), so nothing is burning — the ask is just capability-aware distribution for the built-in schemes, since the capability data is already sitting on the node rows.

That said, if mixed-shape clusters are outside what you want to support, that's a fine answer too — is there a sanctioned way to run a "durable producer" host that stays out of the agent cluster entirely (Serverless? MediatorOnly with an external transport?)? Cluster membership for the producer hosts is entirely incidental for us: we need transactional staging plus something recovering owner_id = 0 envelopes, and we're happy to restructure toward whatever shape you actually intend.

[jeremydmiller]

@beriniwlew Okay. I would have recommended you making the different types of nodes to be completely separate logical services (WolverineOptions.ServiceName) and use shared configuration helpers for the database side of things to make this easier.

And just to remind you, this is a thing for consulting on this kind of thing: https://jasperfx.net/support-plans/