GH-3959 follow-ups: require a real load monitor, stop the shed draining the cluster, hold pins across every distribution path

[jeremydmiller]

Immediate follow-ups to #4297, closing #4589, #4590 and #4591.

#4589 — the default load monitor measured the wrong thing

MemoryPressureLoadMonitor divided process RSS by GCMemoryInfo.TotalAvailableMemoryBytes — the GC's budget, not a memory limit. The mismatch broke the reading in opposite directions on the two shapes we actually deploy:

	denominator	a healthy service reads
No cgroup limit	the whole machine's RAM	0.09 (116 MB on a 128 GB box)
Inside a cgroup	~75% of the limit, vs full RSS on top	80 at 60% of the limit — already the receive line

So: inert on one, trigger-happy on the other, and inert silently — you set the flag, nothing happens, nothing says why.

There is now no default. CapacityAwareAssignment requires an explicit INodeLoadMonitor, and the runtime refuses to start without one. What "load" means is specific to what an application does, and a built-in guess looks authoritative while being wrong for most deployments. Falling back silently was the worst option on the table: a node advertising nothing is treated as having unlimited headroom, so the fallback would turn the feature on and then quietly make that node the cluster's preferred placement target.

MemoryPressureLoadMonitor stays as an opt-in class, now measuring against the cgroup limit the process actually runs under (v2 memory.max, then v1, then a configured GC hard limit) and returning null when there is no limit to measure against. A monitor that cannot see a ceiling says so instead of inventing one.

#4590 — shedding drained an overloaded cluster to zero

The shed pass ran before the "nobody has headroom" early return, so with every node over the line each evaluation detached a batch and placed none of it:

single node over the shed line:   3 → 2 → 1 → 0 → 0
two nodes, both overloaded:       6 → 4 → 2 → 0 → 0

One batch per tick, taking the durability agents and daemon shards with it, and never restarting while the pressure held. On a single-node cluster it was unconditional. Shedding is only ever a move, so it now happens only once a destination is known to exist — an overloaded node still running its work beats an idle one.

Also in this area: a node advertising no load sorted into the lowest band, which made a node mid-rolling-upgrade — or one whose sampler was throwing — the preferred target for every placement precisely when least was known about it. It sorts mid-range now: still eligible, no longer privileged.

#4591 — pinned agents were detached by the ceiling pass

ApplyRestrictions runs before the families distribute, so a ceiling pass that detached whatever sat above the line undid an operator's pin — and ApplyRestrictions put it back next evaluation, and the pass took it off again. A churn loop that emits commands forever and never converges.

#4297 fixed this for DistributeEvenly. DistributeEvenlyWithBlueGreenSemantics and DistributeEvenlyWithAffinity still had the bare Skip(maximum). All three now share Node.ExtrasAboveCeiling, which counts pins toward the ceiling without ever detaching one — a node carrying pins gives up more of its unpinned agents instead.

The capability detach in the affinity path deliberately still ignores pins. A pin to a node that cannot run the agent is an instruction that cannot be carried out, and honoring it would park the agent somewhere that only throws "Unrecognized agent scheme".

Notes on the tests

Every new test was confirmed red against the pre-fix code before being kept. Two needed correcting before they bit:

• The pin tests only reproduce the bug when the pin sits at the tail of the node's agent list — which is where ApplyRestrictions actually leaves it, since TryAssign appends — and has somewhere else it could go. A pinned agent whose only capable node is the one it is pinned to gets detached and re-placed on that same node within the pass, so the final assignment looks untouched even though the evaluation churned.
• TryAssign only succeeds for an agent the node declares, so a grid built without capabilities silently never applies the pin at all. My first version of the fixed-point test was green for that reason rather than for the right one.
• The monitor's percentage test originally re-read a live Environment.WorkingSet, which passed alone and failed inside the full suite. MemoryPressureLoadMonitor now has a protected virtual CurrentWorkingSet seam so the arithmetic and the smoothing are driven deterministically.

The drain regressions need consecutive evaluations to fail — a single-tick assertion passed throughout the bug's life.

Verification

• dotnet build wolverine.slnx -c Release -f net9.0 — 0 warnings, 0 errors
• CoreTests — 3062 passed, 0 failed (+16 new)
• PostgresqlTests.Agents.capacity_aware_column_is_opt_in — 3 passed against the docker Postgres

Remaining follow-ups from the #4297 review are tracked in #4592 (group-affinity / blue-green coverage — the multi-database shape this feature was asked for), #4593 (the other message stores), #4594 (docs), JasperFx/CritterWatch#1343 (alert category) and JasperFx/ai-skills#233.

🤖 Generated with Claude Code

[mlh758]

Checks out on the tools I used to reproduce it. Sorry for the oversight on the lost assignment. I added hooks to detect agents that were both unassigned and not running for a later experiment for testing ungraceful shutdown on nodes but didn't then apply that check to the overload thing I used for testing #4297.