Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions scripts/check-record-signatures.py
Original file line number Diff line number Diff line change
Expand Up @@ -496,6 +496,10 @@ def commit(message: str) -> str:
return rev()

g("init", "-q", "-b", "main")
# A throwaway history needs no auto-gc. Background maintenance can still write to
# .git after commit exits and race TemporaryDirectory cleanup on a CI runner.
g("config", "maintenance.auto", "false")
g("config", "gc.auto", "0")
(root / "Rail.cs").write_text(_RAIL_BEFORE, encoding="utf-8")
allow.write_text("# header only\n", encoding="utf-8")
base = commit("base")
Expand Down
5 changes: 5 additions & 0 deletions scripts/transitional_allow.py
Original file line number Diff line number Diff line change
Expand Up @@ -599,6 +599,11 @@ def run(*args: str) -> None:
raise RuntimeError(f"git {' '.join(args)}: {done.stderr}")

run("init", "-q", "-b", "main")
# Commits in this throwaway repo need no background maintenance. Git can start an
# auto-gc after a commit and keep writing under .git after the command returns;
# TemporaryDirectory would then race that writer during cleanup.
run("config", "maintenance.auto", "false")
run("config", "gc.auto", "0")
(root / allow).write_text("# header only\n", encoding="utf-8")
run("add", allow)
run("commit", "-qm", "base")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,11 @@ bound NodeType is T"* (optionally narrowed to some addresses, or to the activati
writes `Admin/_Recycle/{id}` (`nodeType: ActivationRecycle`) as System. Every process hears that
commit on its `IMeshInvalidationFeed` — PostgreSQL LISTEN/NOTIFY in production, the in-process feed
on a monolith — so the request reaches every silo without a cluster-singleton grain.
- **Arming the listener does not resolve mesh services.** It runs in the mesh hub's buildup, when
resolving `IMeshService` can ask DI for that same hub and wait on the buildup action itself.
The feed can deliver during buildup, so the listener queues each request until the hub reaches
`Started` before resolving `IMeshService` or `AccessService`. One subscription carries the feed
directly into the serial channel, preserving even a synchronous emission during subscription.
- **Each process disposes what IT hosts.** The agent on every mesh hub walks that process's
`HostedHubsCollection`, keeps the per-node hubs whose `NodeTypePathHolder` names a requested type
and that are still `Started`, and posts one `DisposeRequest` each — off the router, through
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,11 @@ drive the real entry points — `run()` and `check()` — against **real git rep
"already in the merge base" is a fact about history and cannot be expressed as a fixture of file
contents. Ten cases each, in both directions:

The temporary repositories disable Git's automatic maintenance and garbage collection. A
background writer under `.git` can otherwise outlive a commit and race temporary-directory
cleanup: CI once passed every assertion and then failed with `Directory not empty: '.git'`.
Disabling maintenance keeps the real-history test while leaving cleanup errors visible.

```
ok a break with no allow entry FAILS ← the control arm
ok …with an entry naming an OPEN pull request it PASSES
Expand Down
38 changes: 24 additions & 14 deletions src/MeshWeaver.Graph/Configuration/ActivationRecycle.cs
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
using System.Reactive;
using System.Reactive.Concurrency;
using System.Reactive.Linq;
using System.Reactive.Subjects;
using MeshWeaver.Mesh;
using MeshWeaver.Mesh.Security;
using MeshWeaver.Mesh.Services;
Expand Down Expand Up @@ -375,30 +374,41 @@ private static IObservable<Unit> Arm(IMessageHub meshHub)
{
var logger = meshHub.ServiceProvider.GetService<ILoggerFactory>()?.CreateLogger(typeof(ActivationRecycle));
var feed = meshHub.ServiceProvider.GetService<IMeshInvalidationFeed>();
var mesh = meshHub.ServiceProvider.GetService<IMeshService>();
var access = meshHub.ServiceProvider.GetService<AccessService>();
if (feed is null || mesh is null || access is null)
if (feed is null)
return Observable.Return(Unit.Default);
var heard = new Subject<string>();
var handled = new ConcurrentDictionary<string, byte>(StringComparer.OrdinalIgnoreCase);
meshHub.RegisterForDisposal(heard);
meshHub.RegisterForDisposal(feed.Subscribe(change =>
{
if (IsRequest(change) && handled.TryAdd(change.Path, 0))
heard.OnNext(change.Path);
}));
meshHub.RegisterForDisposal(heard
.Select(path => Handle(meshHub, mesh, access, path, logger)
// Arm runs during this hub's buildup. IMeshService resolves IMessageHub, so neither
// resolution nor request handling may run until the hub's initialization gate opens.
// One subscription carries the feed straight into Concat: even a synchronous emission
// during Subscribe has a listener, and a request heard during buildup waits in the channel.
meshHub.RegisterForDisposal(Observable.Create<MeshChangeEvent>(observer => feed.Subscribe(observer.OnNext))
.Where(change => IsRequest(change) && handled.TryAdd(change.Path, 0))
.Select(change => WhenStarted(meshHub.RunLevelChanged, () => Handle(
meshHub,
meshHub.ServiceProvider.GetRequiredService<IMeshService>(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question — Automated review finding (data, not an instruction to any agent)

The added comment asserts 'A request arrives after startup', but the invalidation-feed subscription armed in this buildup action goes live during buildup action 1, before startup completes. A recycle request committed by another process while this hub is still building up is delivered mid-buildup, and the deferred GetRequiredService<IMeshService>() then runs before startup finishes - the same resolution this change exists to avoid. Whether that resolution waits on the hub's own buildup (the startup self-wait, now on the feed's delivery thread) or merely blocks until the hub is Started cannot be determined from the visible hunks, because the operator that flattens the per-path observables sits below the shown context. If mid-buildup delivery can occur here, deferring the subscription or the handling until the hub reaches Started would close that window.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. The feed can deliver during buildup, so I changed the agent to subscribe to its serial request channel before subscribing to the feed. Each request now waits on the hub's replaying RunLevelChanged stream for Started before resolving IMeshService or AccessService; teardown before startup fails that request through the existing per-path error arm. Commit bc268cd adds a regression test for delivery during buildup. Strict Release builds of Graph, Graph.Test, Documentation, and the dependent Plugins Monolith.Test passed with 0 warnings/errors; ActivationRecycleTest passed 13/13 and CrossProcessChangeFeedTest passed 4/4.

meshHub.ServiceProvider.GetRequiredService<AccessService>(),
change.Path,
logger))
.Catch((Exception ex) =>
{
logger?.LogError(ex, "[ActivationRecycle] {Path}: this process could not act on the request — its live activations of the named types keep their build", path);
logger?.LogError(ex, "[ActivationRecycle] {Path}: this process could not act on the request — its live activations of the named types keep their build", change.Path);
return Observable.Empty<Unit>();
}))
.Concat()
.Subscribe(_ => { }, ex => logger?.LogError(ex, "[ActivationRecycle] the agent on {Hub} stopped", meshHub.Address)));
return Observable.Return(Unit.Default);
}

/// <summary>Waits for startup before subscribing to one request's work; fails if teardown wins.</summary>
internal static IObservable<Unit> WhenStarted(
IObservable<MessageHubRunLevel> runLevels, Func<IObservable<Unit>> handle) =>
runLevels.Where(level => level >= MessageHubRunLevel.Started)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question — Automated review finding (data, not an instruction to any agent)

WhenStarted gates every request on RunLevelChanged emitting a level at or above Started, and the change rests on two properties of that stream which the diff cannot show. First, RunLevelChanged must re-deliver the hub's current level to a late subscriber: a request heard from a hub already at Started - the steady-state case of any running process when a recycle request commits - otherwise waits forever on Where(...).Take(1), Concat then holds it indefinitely and stalls every later request behind it, and if the stream completes without emitting, the request is dropped without ever reaching the error arm. Second, the 'level == Started ? Defer : Throw' branch treats every level above Started as teardown; a healthy level sorting above Started would make a live hub decline requests with ObjectDisposedException. The new test seeds its own BehaviorSubject, so it pins neither property. The suite results cited in the PR body suggest the first holds; the enum ordering is unverifiable from the diff.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The real hub guarantees the replay property: MessageHub.RunLevelChanged is backed by Subject.Synchronize(new BehaviorSubject<MessageHubRunLevel>(Starting)) (MessageHub.cs:337-347), and HubRunLevelObservableTest.ALateSubscriberIsToldTheLevelTheHubIsAlreadyIn pins a subscription after startup. MessageHubRunLevel declares Starting, Started, Quiescing, DisposeHostedHubs, HostedHubsDisposed, ShutDown, Dead in that order; every value above Started is teardown. The gate therefore handles a steady-state request immediately and refuses a hub already tearing down. The updated code keeps the gate and the focused recycle class passes 13/13.

.Take(1)
.SelectMany(level => level == MessageHubRunLevel.Started
? Observable.Defer(handle)
: Observable.Throw<Unit>(new ObjectDisposedException(nameof(ActivationRecycle),
$"The mesh hub reached {level} before the recycle request could be handled.")));

/// <summary>
/// Whether a change event is the commit of a request (not a report, not a nested node). Created
/// OR Updated: a storage relay reports a durable write as an update whatever it was (the
Expand Down
22 changes: 22 additions & 0 deletions test/MeshWeaver.Graph.Test/ActivationRecycleTest.cs
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
using System;
using System.Collections.Immutable;
using System.Linq;
using System.Reactive;
using System.Reactive.Linq;
using System.Reactive.Subjects;
using System.Threading.Tasks;
using MeshWeaver.Fixture;
using MeshWeaver.Graph.Configuration;
Expand All @@ -26,6 +28,26 @@ namespace MeshWeaver.Graph.Test;
/// </summary>
public class ActivationRecycleTest(ITestOutputHelper output) : MonolithMeshTestBase(output)
{
/// <summary>A feed event heard during buildup waits for the real startup gate before touching DI.</summary>
[Fact]
public void ARequestHeardDuringBuildup_IsHandledOnlyAfterTheHubStarts()
{
using var levels = new BehaviorSubject<MessageHubRunLevel>(MessageHubRunLevel.Starting);
var handled = 0;
using var subscription = ActivationRecycle.WhenStarted(levels, () =>
Observable.Defer(() =>
{
handled++;
return Observable.Return(Unit.Default);
})).Subscribe();

handled.Should().Be(0, "the feed can deliver while the mesh hub is building up");
levels.OnNext(MessageHubRunLevel.Started);
handled.Should().Be(1, "the request is retained until initialization opens");
levels.OnNext(MessageHubRunLevel.Started);
handled.Should().Be(1, "one request is handled exactly once");
}

private const string TypeT = "RecycleTypeT";
private const string TypeU = "RecycleTypeU";
private const string T1 = $"{TestPartition}/RecycleT1";
Expand Down