Agent assignment stalls at a few hundred of ~5650 agents and the leader goes idle (Wolverine 6.23.0, Marten multi-tenanted databases)

[erdtsieck]

Summary

With ~5650 distributed agents (Marten event subscriptions across 512 tenant databases + one durability agent per database), Wolverine's leader never converges on a full assignment.

It converges, but at roughly 9–10 agents per minute and heavily skewed: two and a half hours after a rollout, 1481 of ~5240 agents are assigned and 1114 of those sit on the leader. Followers go 40 minutes at a stretch without gaining a single agent while the leader keeps adding to itself, and during those stretches no AssignmentChanged record is written at all — then it bursts back to ~50/s. Nodes stay healthy and no exceptions are logged. At this rate a full assignment takes about nine hours, and any rollout restarts it from scratch, so in practice most projections never run.

Two concrete leads, both detailed below:

• 5987 dead-lettered Wolverine.Runtime.Agents.AgentsStarted messages, every one with System.UriFormatException: Invalid URI: The URI is empty. — i.e. the reply that confirms an agent start failing to parse. Dated 10–14 July, an earlier release, but on this same cluster and agent set.
• Projection version bumps as the trigger. The previous release had none and rolled out fine; every stuck agent belongs to one of the two projections whose version changed in the release that is stuck.

The per-node counts also land on exact multiples of AgentStartBatchSize early on, which is what first made us look at the chunked assignment path.

Environment

• WolverineFx 6.24.0, Marten 9.20.2, JasperFx 2.36.2, .NET 10, PostgreSQL 18 (AWS RDS) — verified from the running container image, not from configuration. The five nodes in the measurements below started 16:26–16:30 local, which is exactly the rollout of that build.
  
  (Correction: an earlier version of this issue said 6.23.0 / 9.20.0 / 2.36.1. That came from a ddpv tag inside a persisted mt_event_progression.pause_reason, which is written once and never cleared, so it named a build from earlier in the day. The stall reproduces on the current 6.24.0.)
• IntegrateWithWolverine(opts => { opts.MainDatabaseConnectionString = ...; opts.UseWolverineManagedEventSubscriptionDistribution = true; }) — always on; we never register AddAsyncDaemon(...)
• MultiTenantedWithShardedDatabases — 512 tenant databases on one server, ~854 tenants
• Events.TenancyStyle = Conjoined, Events.UseTenantPartitionedEvents = true, Events.EnableExtendedProgressionTracking = true
• 6 async projections on the main store (4 of them CompositeProjectionFor)
• Durability.Mode is Balanced; no Durability overrides at all — AgentStartBatchSize 50, MaxAgentStartParallelism 10, StaleNodeEjectionThreshold 2
• No external broker that provides a per-node control queue (AWS SQS only), so node-to-node control traffic uses the database control queue

Agent capabilities per node, from wolverine_nodes.capabilities: 5228–5240, of which

• ~5100 event-subscriptions://marten/main/<host>.<db>/<projection>/all/v<n>/<tenant> — tenants × projections
• 512 wolverinedb://postgresql/<host>/claims<N>_productie/public — one durability agent per tenant database
• ~14 on an ancillary store

Observed

Three healthy nodes, mid-rollout. wolverine_node_assignments, count(*) per node:

node 478  100
node 479   50
node 480    0

150 total of ~5650. 100 and 50 are exact multiples of AgentStartBatchSize (50). Polled over 100 s: 97 → 99. Meanwhile 22 465 AssignmentChanged records in 5 minutes (~75/s) with the assignment table not growing.

Five healthy nodes, stable topology, no deploy in flight. After ~30 minutes:

node  capabilities  assigned
481          5228        97
483          5234        39
484          5234        39
485          5234        50
486          5240        49
                       ---
                       274

AssignmentChanged in the preceding 5 minutes: 0. AgentStarted: 51. The leader has stopped assigning with ~4970 agents unassigned.

Same five nodes, sampled twice more — nothing restarted, health_check fresh on all five throughout. Node 481 is the leader; it holds the wolverine://leader/ row in wolverine_node_assignments:

node        +20 min   +40 min   delta
481 (leader)    300       350    +253
483              39        39       0
484              39        39       0
485              50        50       0
486              49        49       0
                ---       ---
                477       527

AssignmentChanged stayed at 0 across both of those windows; AgentStarted was 101 and 50. So during those 40 minutes the table grew only for the leader's own node, and without a single AssignmentChanged record being written, while the four followers sat at exactly the counts from their first chunk.

Correction to an earlier version of this issue: I first described that as the leader having stopped for good. It has not — it grinds, in long-idle-then-burst fashion. Two and a half hours after the rollout, with the same five nodes and nothing restarted:

node        16:14   17:20   18:59
481 (leader)  100     350    1114
483            50      39      89
484             0      39      39
485             -      50     145
486             -      49      94
              ---     ---    ----
              150     527    1481   of ~5240

AssignmentChanged over the last 15 minutes of that: 44 884 (~50/s), having been 0/min for the two windows before it. So the pattern is not a hard stall but very slow, bursty, and badly skewed convergence: ~9–10 agents/minute overall, 75% of everything placed on the leader, and followers gaining nothing for 40-minute stretches. At that rate a full assignment takes on the order of nine hours, and any rollout in the meantime starts it over.

0 of the 512 wolverinedb:// durability agents are assigned in any sample, so per-database outbox/scheduled-message recovery is not running anywhere.

Effect on projections. mt_event_progression across the 512 databases: 5463 of 5823 rows with agent_status = 'Stopped', 455 of 512 databases behind their high-water mark, individual (tenant, projection) shards up to 1.7M events behind. 3000 current agents have never written a heartbeat.

Not resource pressure. The database server was idle throughout: CPU 10–34%, read latency 0.9 ms, 2.5k of 12k provisioned IOPS, EBSIOBalance 99%, 134–235 of max_connections 2400. No errors in mt_event_progression.failure_category beyond 4 unrelated read timeouts out of 5823 rows, and no AgentPaused records since earlier in the day.

Node-record event types, all-time, for completeness — there is no failure-shaped event in there at all:

AssignmentChanged    36 087 221
AgentStarted             33 239
AgentStopped             14 314
AgentPaused                  96
NodeStarted                  29
DormantNodeEjected           11
LeadershipAssumed            11

AgentStarted is still being written as I write this (most recent 17:53:37), so agents do keep starting locally — it is the assignment bookkeeping that does not spread.

Startup failures: yes — 5987 dead-lettered AgentsStarted replies

Asked whether we see failures on startup. We do, on exactly the message that confirms an agent start. wolverine_dead_letters in the Wolverine main (references) database:

message_type                              exception_type            exception_message                  count
Wolverine.Runtime.Agents.AgentsStarted    System.UriFormatException  Invalid URI: The URI is empty.      5987

Every one of the 5987 dead letters is that same message type, exception type and message — nothing else has ever dead-lettered in that database. They arrived on the database control queue:

received_at                                            source            count
dbcontrol://ea02a4a5-7bb5-425f-9634-bc04f5d631f8/      Zorgdeclaraties    3714
dbcontrol://18e57cca-2198-4952-9c20-1f134cb553be/      Zorgdeclaraties    1157
dbcontrol://7e020f79-5b09-4044-ac01-f8731a0615a4/      Zorgdeclaraties    1003
dbcontrol://d800f95b-ebb0-4fab-857f-bb7e13d46ffe/      Zorgdeclaraties     113

So a node reports "I started these agents", the leader cannot deserialize the reply because some agent Uri in it is an empty string, and the reply is discarded. The leader then has no confirmation for that batch.

Two caveats, stated plainly:

• These are dated 10–14 July, i.e. an earlier release, not the window of the stall described above. There are no dead letters in the current window and no stuck envelopes in wolverine_incoming_envelopes / wolverine_outgoing_envelopes right now.
• I could not decode a body to show which URI is empty — the payload is not UTF-8 (invalid byte sequence for encoding "UTF8": 0xa1), so it is a binary serializer. Happy to dump raw bytes if that helps.

What makes it worth reporting anyway: it is the same message class that has to confirm a start, failing on empty-URI parsing, in the same cluster with the same agent set. If that parse still fails now but no longer dead-letters — swallowed instead — the leader would never see confirmations, which is exactly the state the assignment table is in, and it would explain why only the leader's own local starts (which need no reply) keep landing.

Our agent URIs may be relevant to reproducing it: the database segment contains dots, e.g.

event-subscriptions://marten/main/database-productie-1.zorgdeclaraties-productie.aws.topicus.healthcare.claims95_productie/provided-cares/all/v21/01009333
wolverinedb://postgresql/database-productie-1.zorgdeclaraties-productie.aws.topicus.healthcare/claims336_productie/public

Second failure mode: connection-slot exhaustion

The shard databases also dead-letter Npgsql.PostgresException 53300: remaining connection slots are reserved for roles with privileges of the "pg_use_reserved_connections"/"rds_reserved" role — 172 of them, most recently today:

2026-07-09   2      2026-07-16  22      2026-07-24   1
2026-07-10   7      2026-07-17   4      2026-07-27   4
2026-07-12   1      2026-07-21  35      2026-07-29   7
2026-07-14   1      2026-07-22  34
2026-07-15   9

This is our own capacity problem rather than a Wolverine bug, but not the one I first assumed. The shard pools are already capped — the connection string carries Maximum Pool Size=4, Connection Idle Lifetime=30, Connection Pruning Interval=10. The exhaustion happens anyway because the cap is per pool and the pool count is per shard database, per pod: Marten's sharded tenancy builds one NpgsqlDataSource per shard database, and while projection agents are assigned with database affinity, an API pod still queries any tenant's shard on request. So the ceiling is 512 databases × 4 × 5 replicas ≈ 10 000 against max_connections 2400, and 4 is already tight enough that lowering it further is unattractive.

It is listed here because a node that cannot open a connection to a shard database cannot start that database's agents either, so it is a plausible source of failed starts feeding into the above.

The remaining shard dead letters are all application-level handler bugs of ours (NullReferenceException, a Refit 500, ApplyEventException) and unrelated.

The trigger looks like projection version bumps

Colleagues pointed out that the previous release rolled out fine, and that it carried no projection version bumps. The release that is stuck did. That holds up against the data.

Versions in the deployed release vs the one before it:

projection	previous release	deployed release
provided-cares	V20	V21
invoices	V14	V15
claim_lines	V14	V14
invoicejournalentries	V4	V4
import-reconciliation	V2	V2

A version bump changes the agent Uri (.../provided-cares/all/v21/<tenant>), so all ~853 per-tenant shards of a bumped projection are brand-new agents that have to be assigned and started from scratch. The unbumped ones keep their URIs and their existing assignments.

How far each projection got, per mt_event_progression (a row with agent_status set means the agent ran at least once since the deploy), out of 853 advertised agents each:

projection	bumped	agents that ran	agents behind their high-water mark
provided-cares V21	yes	119 (14%)	13
invoices V15	yes	118 (14%)	9
claim_lines V14	no	642 (75%)	0
invoicejournalentries V4	no	642 (75%)	0
import-reconciliation V2	no	642 (75%)	0

Every lagging agent in the whole cluster is one of the two bumped projections; not one of the unbumped ones is behind. The unbumped projections are three-quarters started, the bumped ones one-seventh.

One candidate mechanism, which I have not proven: both bumped composites set AsyncOptions.GateSideEffectsBehindPriorVersion = true (#480). On a version bump that makes each new agent, at start, first replay up to the prior version's progression mark in Rebuild mode before going continuous. So a bump turns ~1700 agent starts into ~1700 long-running rebuild replays, each against a different tenant partition. If an agent start has to make meaningful progress before the receiving node acks its StartAgents batch, the chunk reply would time out and the leader's walk would never be confirmed — which is what the assignment table shows. That would also explain the leader-only growth: the leader's own local starts need no ack.

If that is the mechanism, the interesting question is whether a slow or long-running agent start should be able to hold up (or silently void) the rest of the assignment walk.

History — the pre-6.23.0 livelock, and what it became

wolverine_node_records had grown to 16 GB / 36M rows in five days (that table having no retention at all is filed separately as #3701):

day	AssignmentChanged	AgentStarted
24 Jul	4 756 150	0
25 Jul	12 818 392	6
26 Jul	12 818 386	0
27 Jul	5 530 373	17 392
28 Jul	12 482	9 462
29 Jul	25 995 + bursts	5 344

25–26 July: 12.8M assignment decisions per day with zero agents started — the livelock the 6.20→6.23 work targets. That stopped after we deployed 6.23.0 on 27 July ~17:00, so those fixes clearly did something. We have since moved to 6.24.0, which is what the cluster ran during every measurement above.

What is left on 6.24.0 is the quieter failure: the leader converges to a stable-but-wrong state a few hundred agents in, then only trickles agents onto itself.

Two smaller things noticed in the same table:

• NodeStarted and LeadershipAssumed rows carry nonsense node_number values (1827159984, -1841074440, -899441513) interleaved with the real ones — looks like the skeleton-node insert described in the 6.23.0 notes still leaves a trace on the record side.
• DormantNodeEjected fires while health_check on those rows is fresh.

What I would expect

With N healthy nodes and M advertised agents, the leader keeps issuing chunks until all M are assigned, or reports why it cannot. Stopping at 274–527 of ~5240 with no error and no further AssignmentChanged looks like the chunked-assignment loop losing track of the remainder — perhaps the pending-assignment ledger suppressing chunks that were never confirmed, or a per-chunk reply timeout leaving the walk in a state the next CheckAssignmentPeriod treats as complete.

The sharpest clue is the asymmetry: the leader keeps adding agents to itself while every follower stays frozen at its first chunk, and the leader's own additions produce no AssignmentChanged records at all. That reads as the local assignment path still making progress while the remote AssignAgent path has stopped being driven — two code paths that have diverged rather than one loop that stalled.

Happy to run queries against the production tables, dump wolverine_nodes / wolverine_node_assignments / wolverine_node_records slices, or turn on debug logging on the leader — say what would help.

What we are doing meanwhile

Only making the assignment settings configurable per environment (AgentStartBatchSize, MaxAgentStartParallelism, StaleNodeEjectionThreshold, the assignment cadence) so we can try larger chunks and fewer rebalances, and looking at moving the control queue off the database onto a broker with per-node control queues. We are staying on Wolverine-managed distribution throughout — handing the daemon back to Marten's own coordination is not an option for us.

[erdtsieck]

[erdtsieck]

We shipped the ability to configure the assignment knobs and ran a controlled experiment on the same production cluster. Summary: raising AgentStartBatchSize makes the first burst roughly 25× faster, but it does not make the assignment converge — it moves the plateau, then stops in the same way.

Setup

Same 512 tenant databases, ~854 tenants, six async projections. Node capabilities are now 5366 agents each. Five replicas, Durability.Mode = Balanced, Wolverine 6.24.0.

Deployed a build with only these changed from the defaults:

Daemon:AgentStartBatchSize            1500   (default 50)
Daemon:MaxAgentStartParallelism         25   (default 10)
Daemon:MaxAgentStopParallelism          25   (default 10)
Daemon:StaleNodeEjectionThreshold        3   (default 2)
Daemon:CheckAssignmentPeriodSeconds     15   (default 30)

Then rolled the deployment so all five nodes registered fresh (nodes 491–495, 19:19–19:22), and sampled every 2.5 minutes.

Result

wolverine_node_assignments, total and per node, plus records written in the preceding 2 minutes:

time	assigned	per node (491/492/493/494/495)	AssignmentChanged/2min	AgentStarted/2min
19:23	1	0/0/0/1/0	5857	0
19:25	500	499/0/0/1/0	0	147
19:28	507	506/0/0/1/0	0	6
19:30	511	510/0/0/1/0	0	3
19:33	514	513/0/0/1/0	0	3
19:35	517	516/0/0/1/0	0	2
19:38	517	516/0/0/1/0	0	0

Node 494 is the leader (it holds the wolverine://leader/ row).

Against the untuned run reported above: 500 agents placed within 2.5 minutes, versus 67 minutes to reach 527. That part is a real, large improvement and we are keeping the settings.

But then it flatlines at 517 of 5366, and this time everything lands on a single follower while three nodes get nothing at all. AssignmentChanged is 5857 in the first two minutes and then exactly 0 for the next fifteen.

What that suggests

Untuned, the counts were 100/50/0 — multiples of AgentStartBatchSize, spread over more than one node. Tuned, it is 499 on one node and zero on three others. So the limiter is not "N agents then stop": the leader appears to complete one node's share and then stop iterating over the remaining nodes, and a bigger batch simply means that one node's share is bigger.

499 is also suggestive with database-affine assignment: agents are grouped per tenant database and whole groups are placed together, so 499 is presumably the agents belonging to some set of databases. It looks like the leader places group after group onto the first node until the batch is exhausted, and never advances to the second node.

Projection state after the experiment

Measured directly from mt_event_progression across all 512 databases, because our own admin aggregate cannot see per-tenant shards:

• Running agents: 145 claim_lines:V14, 145 InvoiceJournalEntries:V4, 144 provided-cares:V21 — i.e. the tenants living on the one node that got assigned.
• 1580 agents behind, 22 587 071 events of backlog in total. Worst single shards: provided-cares:V21 at 1 413 515 and 1 387 864 events, invoices:V15 at 1 286 365.

So the four idle nodes are not idle because there is no work — there is a 22.6M-event backlog they could be chewing through.

Happy to try other combinations if you want a specific one tested; we can roll a config change and sample it in about 20 minutes.

[erdtsieck]

Correction to my previous comment, and I think this is the actual mechanism.

I posted that too early — it does not flatline at 517. It advances, one node per evaluation pass, roughly every 20 minutes. Continuing the same run:

time	assigned	491 / 492 / 493 / 494 / 495
19:25	500	499/0/0/1/0
19:38	517	516/0/0/1/0
19:43	518	517/0/0/1/0
19:46	959	517/441/0/1/0
19:51	969	517/451/0/1/0

(494 is the leader, holding only the wolverine://leader/ row.)

The per-minute record log is the giveaway

19:22  AssignmentChanged  5857
19:23  AssignmentChanged  5884     AgentStarted 465     <- node 491 gets 517
19:24  AgentStarted 30
19:25  AgentStarted 5
19:26  AgentStarted 4
19:27  AgentStarted 1
19:28  AgentStarted 3
19:29  AgentStarted 2
19:32  AgentStarted 2
19:33  AgentStarted 2
19:34  AgentStarted 2
19:38  AgentStarted 1                                   <- last straggler of node 491
19:43  AssignmentChanged  5367     AgentStarted 189     <- node 492 gets 451
19:44  AgentStarted 246

Two things fall out of that:

1. Each evaluation pass writes ~5366 AssignmentChanged rows — exactly one per advertised agent. The node advertises 5366 capabilities, and the pass emits 5857/5884/5367 records. So the leader re-decides every single agent on every pass, and only ~500 of those decisions actually result in an assignment. That is also where the 12.8M records/day in the original report came from: a pass per interval × one row per agent.

2. A new pass does not start while the previous batch is still starting. Node 491's stragglers trickle in until 19:38 (1–5 AgentStarted per minute), and the next pass fires at 19:43. Same shape for the first pass. That would explain the ~20 minute cadence far better than any configured interval — note CheckAssignmentPeriod was set to 15 seconds for this run, so the gap is not the poll interval.

If that is right, the convergence time is agents / agents-placed-per-pass × pass-duration, and pass duration is dominated by how long the slowest agent in the previous batch takes to report started. With per-tenant projection shards on a version bump — where each new agent's start kicks off a rebuild replay (GateSideEffectsBehindPriorVersion) — those starts are slow, so the whole fleet serialises behind them.

Which also explains why the tuning helps as much as it does

AgentStartBatchSize does not change the number of passes needed per se, but it changes how many agents each pass places:

	untuned (batch 50)	tuned (batch 1500)
agents placed per pass	~50–150	~450–520
time to ~500 assigned	67 min	2.5 min
projected full convergence	~9 h	~3.5 h

So the knob is worth having — we are keeping it — but the real cost driver is "one node per pass, and a pass waits for the previous batch to finish starting". Raising the batch just buys a bigger bite per pass.

What would help us most

Either of these would take this from hours to minutes for a cluster this size:

• let a pass place work on all eligible nodes rather than filling one and stopping, and
• do not gate the next pass on the previous batch having fully started, or at least bound that wait.

And separately: one AssignmentChanged row per agent per pass is what turns this into a 16 GB table (#3701) — a per-pass summary row would carry the same information.

Environment unchanged: Wolverine 6.24.0, Marten 9.20.2, JasperFx 2.36.2, 512 tenant databases, ~854 tenants, 5366 agents, 5 replicas, Durability.Mode = Balanced, database control queue (no broker with per-node control queues).

[jeremydmiller]

Closing this out. Every mechanism reported here is now accounted for — three fixes merged, and the two "leads" both resolved, one of them already shipped. Full accounting, with the caveat about releases at the bottom.

Your two asks

1. let a pass place work on all eligible nodes instead of filling one and stopping
2. do not gate the next pass on the previous batch fully starting, or bound that wait

• (1) → Fan the agent command drain out per destination so a pass places work on every node (GH-3698) #3723 (d12b0a487). executeAgentCommandsAsync executed one command at a time across the whole cluster, and because a wave's AssignAgent commands are grouped by destination and appended contiguously, the drain was destination-at-a-time: every chunk for node A completed before node B got its first. Cluster-wide start concurrency was capped at a single node's MaxAgentStartParallelism regardless of node count — which is exactly your "one node per evaluation pass, roughly every 20 minutes."

• (2) → Decouple the assignment evaluation from the command drain, with pending assignment as grid state (GH-3698) #3719 (5a642b222). The health-check loop no longer executes the commands it produces; they go to an independent per-destination dispatcher. This also answers the question you raised directly:

  If that is the mechanism, the interesting question is whether a slow or long-running agent start should be able to hold up (or silently void) the rest of the assignment walk.

  It should not, and now it does not. Your GateSideEffectsBehindPriorVersion hypothesis was a good one — a version bump turning ~1700 agent starts into ~1700 long-running rebuild replays is precisely the load that exposed this.

Your AssignmentChanged measurement

Each evaluation pass writes ~5366 AssignmentChanged rows — exactly one per advertised agent. So the leader re-decides every single agent on every pass, and only ~500 of those decisions actually result in an assignment.

That was the sharpest diagnostic in the whole report, and it is fixed by the pending-assignment ledger in #3719. A dispatched-but-unconfirmed start now shows on the grid as Agent.PendingNode, so the leader stops re-deciding placements it has already dispatched. Crucially, the ledger is held until the dispatch is confirmed or failed rather than for a fixed TTL — 2 × CheckAssignmentPeriod is seconds, and your rebuild waves run for minutes, which is why a TTL would not have been enough.

Your two smaller observations are covered too: the nonsense node_number values on NodeStarted / LeadershipAssumed rows by #3724 (57b79a90d — NodeRecord.For read AssignedNodeNumber one line before the Balanced start path adopted the value PersistAsync returned, so every row carried DurabilitySettings' random default — your 1827159984 / -1841074440), and DormantNodeEjected firing against a fresh health_check by #3624 plus #3724.

Your empty-URI dead letters — already fixed, and it explains your own caveat

Wolverine.Runtime.Agents.AgentsStarted / System.UriFormatException / Invalid URI: The URI is empty. / 5987

Ran this down. That exact message comes from new Uri(""), and AgentsStarted.Read gained StringSplitOptions.RemoveEmptyEntries in 8da0bffe8 (GH-3460 / #3484), released in 6.21.0 on 20 July — immediately after your 10–14 July dead letters. So this was fixed before the cluster you measured, which is exactly why you found "no dead letters in the current window." Your instinct to report it anyway was right, but it is not implicated in the 6.24.0 stall.

Auditing it did turn up a new, still-open defect in the same code, filed as #3733: that round-trip is comma-delimited, and a comma is a legal URI character, so an agent URI containing one shatters the batch and voids the confirmation for every agent in it. Reachable through a tenant id or projection name. Not your symptom, but adjacent to it, and tracked separately now.

The connection-slot exhaustion you described is, as you said, capacity rather than a Wolverine bug; wolverine_node_records retention stays tracked on #3701.

The one thing to be aware of

None of this is released yet. 6.24.0 (28 July) is the latest published build and is the one you measured — all three fixes are on main only, targeting the next release. So please do re-measure when it ships, and reopen or file fresh if the leader still plateaus. The convergence assertion added alongside #3719 (agent_assignment_at_scale) covers 900 agents over 3 nodes and asserts the leader keeps evaluating while starts are in flight, which is a much smaller cluster than your ~5366 — your scale is still the real test.

Thank you for this report. The measurement tables, the controlled AgentStartBatchSize experiment, and particularly your two mid-thread corrections turned a vague "assignment is slow" into three separately-fixable defects. That is a genuinely unusual amount of work to put into an issue and it is the reason this got diagnosed rather than guessed at.