Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,52 @@ changes; what changed is that a caller who *knows* the answer no longer has to w
the heuristic to guess it. Measured on `NodeTypeRecompileAlcLeakTest`: 6.5 → 5.0 retained
hubs per compile activity, with the `cache/`-side mirror hubs going 3 → 0.

### Once the mirror is released, the retention is BOUNDED — and the ceiling is 15 minutes

Releasing the mirror does not itself reclaim the activity's own node hub; it removes the
thing that was **stopping** the reclaim. What finishes the job is already there, in the
monolith as well as on Orleans: an **Activity** node hub is one of the few hub kinds that
carries an idle reaper of its own —

```
ActivityNodeType.CreateMeshNode HubConfiguration = … .AddKernelSubHubHandlers()
→ KernelHubConfigurationExtensions ParentHub.GetService<IKernelHubConfigurator>()
→ KernelContainer.ConfigureSubHub .WithInitialization(hub => DisposeOnTimeout(hub))
→ KernelContainer.DisposeOnTimeout one-shot Timer, KernelHubOptions.IdleDisconnectTimeout
```

registered by `KernelNodeType.AddKernel()`, which `AddGraph()` includes. The timer is
**one-shot and re-armed by every inbound message**, which is exactly why the 45 s heartbeat
above kept it at bay forever — and why, with the terminal release in place, nothing re-arms
it and the hub disposes one window later, taking its `sync/` sub-hubs with it.

So the per-compile `_Activity/compile-<ts>` residual that `NodeTypeRecompileAlcLeakTest`
counts is a **transient with a 15-minute ceiling, not a leak** — that test simply runs for
seconds and cannot watch it expire. Measured unscaled, at production timer values, with the
real 45 s heartbeat present:

| t | per-compile activity hubs | mesh total hubs |
|---:|---:|---:|
| 0 – 885 s | 15 | 51 → 47 |
| 900 s | 5 | 37 |
| **915 s** | **0** | **32** |

13 hubs reclaimed in **915 s = 15.25 min** — one idle window plus the 15 s poll granularity.
(The mesh total's earlier 51 → 47 step near t+660s is the cache's own 10-minute idle sweep,
a different reaper doing a different job.) `CompileActivityHubRetentionTest` is the
CI-affordable guard on the same property, and it measures it directly: the activity hubs
disappear after **exactly one idle window**, with nothing re-arming the timer.

🚨 That test compresses the window **and the heartbeat together**, at production's 1:20
ratio, and the pairing is load-bearing. Shortening only the window would put the 45 s
heartbeat *outside* it: the timer would fire between two heartbeats, the test would pass,
and production — where 45 s sits far inside 15 min — would still reclaim nothing. A guard
that can only pass is not a guard.

🚨 **Do not shorten `IdleDisconnectTimeout` to make a memory figure look better.** The whole
lesson of this section is that the clocks were correct and something was resetting them; a
shorter window would have hidden that defect rather than cured it.

### The idle sweep is not the only reaper — an evicted upstream dies with its last holder

The idle sweep answers *"this path went quiet"*. It cannot answer *"this write is finished
Expand Down
14 changes: 11 additions & 3 deletions src/MeshWeaver.Kernel.Hub/KernelContainer.cs
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,14 @@ public class KernelContainer(IServiceProvider serviceProvider)
private IMessageHub? executorHub;
private readonly object executorHubLock = new();

private static readonly TimeSpan DisconnectTimeout = TimeSpan.FromMinutes(15);
/// <summary>
/// Idle window after which a kernel-hosting hub disposes itself. 15 minutes by default; a host
/// (or a test that needs to observe the reclamation inside its budget) can override it by
/// registering a <see cref="KernelHubOptions"/> singleton. 🚨 NOT a memory tuning knob — see
/// <see cref="KernelHubOptions.IdleDisconnectTimeout"/>.
/// </summary>
private readonly TimeSpan disconnectTimeout =
(serviceProvider.GetService<KernelHubOptions>() ?? new KernelHubOptions()).IdleDisconnectTimeout;

/// <summary>
/// Hub configuration for the standalone kernel-hub (full mesh-types + routes).
Expand Down Expand Up @@ -179,13 +186,14 @@ private void DisposeOnTimeout(IMessageHub hub)
{
if (((WeakReference<IMessageHub>)state!).TryGetTarget(out var h))
h.Dispose();
}, weakHub, DisconnectTimeout, Timeout.InfiniteTimeSpan);
}, weakHub, disconnectTimeout, Timeout.InfiniteTimeSpan);
// Dispose the timer WITH the hub for the normal (activated → disposed) path so
// the queue drops it promptly rather than waiting on a GC.
hub.RegisterForDisposal((IDisposable)timer);
var idle = disconnectTimeout;
hub.Register<object>(d =>
{
timer.Change(DisconnectTimeout, Timeout.InfiniteTimeSpan);
timer.Change(idle, Timeout.InfiniteTimeSpan);
return d;
});
}
Expand Down
31 changes: 31 additions & 0 deletions src/MeshWeaver.Kernel/KernelHubOptions.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
namespace MeshWeaver.Kernel;

/// <summary>
/// Tuning knobs for hubs that host a kernel — which, via
/// <c>ActivityNodeType.CreateMeshNode</c>'s <c>.AddKernelSubHubHandlers()</c>, means every
/// <b>Activity</b> node hub. Optional: when not registered in DI the defaults below apply.
/// </summary>
public sealed record KernelHubOptions
{
/// <summary>
/// How long a kernel-hosting hub may sit with NO inbound message before it disposes itself
/// (<c>KernelContainer.DisposeOnTimeout</c>). A one-shot timer, re-armed by every message the
/// hub receives, so it fires only after a genuinely idle window.
///
/// <para>🚨 <b>This is the ceiling on how long a finished <c>_Activity/compile-&lt;ts&gt;</c>
/// node hub is retained</b> — the residual #1324 measures. It is the one reclaimer that reaches
/// an Activity hub in the monolith (the host has no other idle collection for node hubs), and it
/// is why that residual is a BOUNDED retention rather than a leak. It never fired before #1435
/// because a finished activity's still-warm mirror posted a <c>HeartBeatEvent</c> every 45 s
/// (<c>SyncStreamOptions.HeartbeatInterval</c>) expressly to keep the hub alive, re-arming this
/// timer forever; releasing that mirror on the terminal write is what let the clock run.</para>
///
/// <para>🚨 <b>Do NOT shorten this to make a memory number look better.</b> #1324 prohibits it
/// by name, and #1435's finding is exactly why: the clocks were correct and something was
/// resetting them, so a shorter window would have hidden the defect instead of curing it. It is
/// settable so a TEST can assert the reclamation property inside a test budget
/// (<c>CompileActivityHubRetentionTest</c>), and for a host that genuinely wants a different
/// idle policy — not as a memory tuning knob. Default: 15 minutes.</para>
/// </summary>
public TimeSpan IdleDisconnectTimeout { get; init; } = TimeSpan.FromMinutes(15);
}
Loading
Loading