Agent distribution has no per-node capacity ceiling: one node dying pushes its whole share onto the survivors

[erdtsieck]

Summary

AssignmentGrid distributes agents across every live node without any notion of how much a node can actually host. When a node dies, its share is redistributed over the survivors — which can push them past their own limits, killing them in turn. There is no floor to that process, so a cluster under memory pressure cannot converge, and adding replicas makes it worse rather than better.

We hit this on production last weekend: 34 hours of intermittent unavailability, restart counters up to 334, and no self-recovery.

What already works

To be clear about what I'm not asking for: EventSubscriptionAgentFamily.EvaluateAssignmentsAsync already does the right thing for sharded stores. Multi-database stores get DistributeByGroupAffinity(SchemeName, DatabaseKeyOf, …), keeping all of a shard database's per-tenant agents together on one node so pools scale with databases rather than nodes × databases (#486 / marten#4806). That behaviour is exactly right for our topology and it worked.

The gap

Neither distributor bounds what a node takes on.

AssignmentGrid.Distribution.cs:

var spread = (double)agents.Count / _nodes.Count;
var minimum = (int)Math.Floor(spread);
var maximum = (int)Math.Ceiling(spread);

and group affinity places groups "largest-first onto the least-loaded node". Both balance, neither caps. The implicit assumption is that any live node can host whatever share it is handed.

That assumption breaks whenever an agent's cost is not uniform over time.

What happened

Topology: Marten with a sharded store, 512 shard databases, per-tenant event partitioning. Six projections × 1.463 tenants = 8.792 agents across 6 nodes (one pod per node, enforced by pod anti-affinity).

A release bumped two projection versions, so those read models rebuilt from event 0 — about 381M events. We measured that an agent working off a backlog holds roughly 4.4 MB, against roughly 1 MB for an agent that is caught up. Same agents, same code, four times the memory, purely as a function of having work to do.

At ~1,465 agents per node that is ~6.4 GB during catch-up, against a .NET GC hard limit of 9,216 MiB. Tight but survivable — until one node went over and died. Its ~1,465 agents were then spread over five survivors, taking each of them to ~1,760 agents, over the limit. Then four. Then three.

Two properties made it unrecoverable:

1. Adding capacity does not help. Distribution is over live nodes, so new pods join the same doomed rebalance and die before they can relieve anyone. Measured: scaling 6 → 12 replicas raised agents-per-surviving-pod from 1,196 to 1,290.
2. Every restart triggers another rebalance. With pods failing every few minutes the grid never settles, and each rebalance is itself the expensive moment.

What finally recovered it was replacing all pods simultaneously (maxUnavailable: 100%) so the grid distributed once across six live nodes instead of six times across a shrinking set. That is a Kubernetes-side workaround for a distribution-side problem.

Proposal

An optional capacity ceiling per node:

// DurabilitySettings
public int? MaxAgentsPerNode { get; set; } = null;   // null = today's behaviour

advertised alongside the node's capabilities, and respected by both DistributeEvenly and DistributeByGroupAffinity: when no node can take an agent (or, under group affinity, a group) without exceeding its ceiling, leave it unassigned rather than overload a node. Unassigned agents get picked up on a later evaluation once capacity appears — which it does, as soon as an operator adds a node or the backlog that made agents expensive is worked off.

That converts a cascading failure into visible, bounded degradation: some shards stop progressing, the rest keep running, and the cluster stays up. For our case that is strictly better — a projection falling behind is an internal problem, a fleet that cannot stay up is a customer-facing one.

Two smaller things that would have helped independently:

• Expose the assigned-agent count per node as a metric. We had to infer it from log lines; there was no way to see "this node is carrying 1,760 agents" while it was happening.
• Consider a hysteresis on redistribution. A node that just died is very likely to be replaced within seconds; immediately redistributing its full share is what turned one failure into six.

Willing to help

Happy to prototype this against a fork if the direction is agreeable — we have a reproduction with real scale (8,792 agents, 512 databases) and the production measurements above to validate against.

[erdtsieck]

I prototyped this against 6.25.5 to check the idea holds up in the code rather than just on paper. Sharing it in case it's useful — happy to open a PR, or to drop it if you'd rather solve it another way.

What I found while implementing it. Two things surprised me:

1. EventSubscriptionAgentFamily already places by DistributeByGroupAffinity, so my original framing ("distribution is naive") was wrong. The distribution is group-aware and well thought through — it is only capacity-blind. Every placement path ends in a fallback (?? ordered.FirstOrDefault(x => !x.IsLeader) ?? ordered.First(), ?? candidates.…First()) that assigns regardless of the target's load.

2. FindDelta already does the right thing with an unassigned agent — StopRemoteAgent when it is running, nothing when it is not. So "leave it unassigned" needs no new machinery anywhere; the shape is already there.

The change. DurabilitySettings.MaxAgentsPerNode, default null = today's behaviour exactly. When set, an agent that fits nowhere stays unassigned instead of being forced onto a full node.

One detail that turned out to matter more than expected: capacity has to be read from the node's whole load, not the pass being distributed. countOn() in DistributeEvenly only counts the current pass's agents — correct for a fairness tie-break, but a node already carrying a thousand agents of another scheme looks empty to the next family that distributes. Capacity is a property of the node, not of the pass.

Applied to DistributeEvenly, DistributeByGroupAffinity (whole groups — indivisible by design), the remainder pass of DistributeEvenlyWithAffinity, and DistributeEvenlyWithBlueGreenSemantics. Affinity's preferred placements I left unbounded, since the doc comment says that is deliberate and they follow another family's distribution which is itself now bounded.

The leader reads the ceiling from its own settings rather than per node — every node in a deployment runs the same config, so one value is right in practice and it costs no new column on wolverine_nodes. A genuinely heterogeneous fleet would need it persisted with the node record.

Tests. 7 new, and the existing 2 388 CoreTests still pass unchanged. The one that matters:

[Fact]
public void losing_a_node_stops_at_the_ceiling_instead_of_cascading()
{
    // Three nodes at four agents each; one dies and its four come up for redistribution.
    // Unbounded, the two survivors go to six -- a 50% jump in held state, which is what
    // takes the second node down and then the third. Bounded at five, they take two of
    // the four and the remaining two stay unassigned. The cascade has a floor.
    var agents = Agents("blue", 12);

    var grid = new AssignmentGrid { MaxAgentsPerNode = 5 };
    grid.WithNode(1, Guid.NewGuid()).Running(agents.Take(4).ToArray());
    grid.WithNode(2, Guid.NewGuid()).Running(agents.Skip(4).Take(4).ToArray());
    grid.WithAgents(agents);

    grid.DistributeEvenly("blue");

    grid.Nodes.Each(node => node.Agents.Count.ShouldBe(5));
    grid.UnassignedAgents.Count.ShouldBe(2);
}

The open question, which is yours to make a call on. Silently not running a shard is a real cost, and it trades a loud failure for a quiet one. In our incident that trade was obviously worth it — 34 hours of nothing running at all versus a subset lagging — but that is our situation, not a general one. If this is worth pursuing, unassigned-due-to-capacity probably wants to be visible: a log at warning level and/or a metric, so a misdimensioned ceiling cannot leave shards parked forever without anyone noticing. I did not build that, since where it belongs depends on how you would want it surfaced.

[jeremydmiller]

Scope note: this is not in 6.32, and neither is #3987.

Not a dismissal — the opposite. The two are the same subsystem and, we think, the same design gap: AssignmentGrid has no notion of what a node can host, and no damping across a deploy window. Fixing either one alone risks making the other worse, and a partial answer to "one node dying pushes its whole share onto the survivors" is exactly the shape of change that turns a bad weekend into a worse one.

So they are being taken together, starting from a written design proposal rather than from code. 6.32 would have forced them through on a release schedule, which is the wrong pressure for a change to cluster convergence behaviour.

Nothing here is blocked on more information from you. The report is unusually complete — 34 hours of intermittent unavailability, restart counters to 334, and the explicit note about what already works so we do not go re-solving it — and it is what makes clear that a capacity ceiling alone is not the whole answer.