You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(modules): the reload agent reads the commit the feed announced, never a trailing mirror - #6226
ModuleReloadFaultedBesideActivationTest.OneModulesCrashedAdopt_BesideAnotherModulesLiveActivation_IsFaulted times out on core main (push run 37569688010, shard 3) and on #6210 (run 37570022272, shard 4). #6210's diff does not touch this code; the defect was already on main. The test is not in .github/known-flakes.json.
The per-process ModuleReloadAgent hears a request on the invalidation feed, which fires post-commit. This process's mirror of the node gets the owner's echo by a separate path, and under load that echo arrives after the feed event. Handle read the mirror with a bare Take(1), so for the event that announced Activating it saw the previous state, Landing. It swapped nothing and reported nothing. The executor writes nothing more until a replica reports, so no later event came to wake the agent. The request stayed in Activating, logged nothing, and the test's AwaitRequest ran out its time (108 s in CI, with no log line after the crashed download).
InstanceRebootAgent.Handle had the same read and gets the same floor.
If the mirror never reaches the announced version, the read now times out and logs a Warning that includes the announced version. The old behaviour was silence.
Evidence
New ModuleReloadAgentReadsTheAnnouncedCommitTest makes the "mirror behind the announcement" moment deterministic: both handlers subscribe before the commit is written.
The handler with the floor waits for the commit and reports.
Its negative control, the same handler without a floor (the old read), answers from the earlier state and reports nothing.
Local Release run: 12/12 passed across the new test, ModuleReloadFaultedBesideActivationTest, ModuleReloadLiveTest, ModuleReloadByRestartTest and InstanceRebootTest. Built -c Release -warnaserror with 0 errors.
Not established: CI logs only record Warning level, so the exact interleaving in the CI failure was not captured in a log. The mechanism is inferred from where it stopped (after the crashed download, before any report) and pinned deterministically by the test above.
Doc: Doc/Architecture/ModuleReload step 5.
Pairs-with: none — no public surface removed (internal Handle gains an optional parameter)
Mirror-sync: none — no catalog change
Recycle: none — process-level agents, picked up by the roll
…ever a mirror that trails it
The invalidation feed fires post-commit while this process's mirror receives the
owner's echo on its own path - under load AFTER the feed. The agent's bare Take(1)
then read Landing for the event that announced Activating, swapped nothing,
reported nothing, and nothing later woke it: the request sat in Activating until
ModuleReloadFaultedBesideActivationTest timed out (main run 37569688010, #6210's
run 37570022272). The read now waits for a state at or past the event's Version
(the RebaseSource floor, #1174); the boot listing reads unfloored. The instance
reboot agent had the same read and gets the same floor.
Pinned by ModuleReloadAgentReadsTheAnnouncedCommitTest, whose unfloored control
reports nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…mit is an error, not a silent empty
Both agents' floored read used Take(1): a node stream that completes before reaching
the announced version (a disposed synchronization stream forwards OnCompleted) left
it empty, Timeout passed the completion through, and Arm's warning never ran — no
report, no diagnostic. FirstAsync turns that completion into an error on the existing
warning path.
Adds InstanceRebootAgentReadsTheAnnouncedCommitTest, the reboot twin of the module
reload regression: a handler subscribed below a nonzero floor waits for the
AwaitingRestart commit and reports, while the unfloored control reads the earlier
state and reports nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Use an ImmutableHashSet for this read-only fixture, as required by the collection policy in AGENTS.md:413. It satisfies LoadedNames's IReadOnlySet<string> return type and preserves the case-insensitive comparer without introducing a mutable collection.
…instead of reading an unfloored mirror in silence
A path-only invalidation (StorageChangeFeedRelay.PathOnlyFallback) keeps the node type
but carries version 0, which the agents could not tell from the boot listing's no-floor
read. Each agent now warns, by request path, that the announced commit is unknown and
its read may trail it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Document the intentional best-effort exception for versionless feed events. Both agents log a warning for Version <= 0 but still read unfloored, so they can miss the announced phase when the mirror trails it. As written, this paragraph implies that only boot-listing reads lack a version floor.
This fixture does not mutate the set, but the new HashSet conflicts with the immutable-collection requirement in AGENTS.md:413. Use ImmutableHashSet to preserve the case-insensitive lookup and satisfy LoadedNames's IReadOnlySet<string> contract.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
ModuleReloadFaultedBesideActivationTest.OneModulesCrashedAdopt_BesideAnotherModulesLiveActivation_IsFaultedtimes out on core main (push run 37569688010, shard 3) and on #6210 (run 37570022272, shard 4). #6210's diff does not touch this code; the defect was already on main. The test is not in.github/known-flakes.json.The per-process
ModuleReloadAgenthears a request on the invalidation feed, which fires post-commit. This process's mirror of the node gets the owner's echo by a separate path, and under load that echo arrives after the feed event.Handleread the mirror with a bareTake(1), so for the event that announcedActivatingit saw the previous state,Landing. It swapped nothing and reported nothing. The executor writes nothing more until a replica reports, so no later event came to wake the agent. The request stayed inActivating, logged nothing, and the test'sAwaitRequestran out its time (108 s in CI, with no log line after the crashed download).Fix
ModuleReloadAgent.Handlenow takes the event'sVersionas a floor:GetMeshNodeStream(path).Where(n => n.Version >= committed).Take(1).Timeout(ReadBudget). This is the same floorMeshNodeStreamExtensions.RebaseSourceapplies to an announced version (A CreateNodeRequest addressed to the sending hub itself is never answered (nodeops, empty queue, RunLevel=Started) — activity tracking and every downstream read then time out #1174). Paths from the boot listing announce no commit, so they read with no floor (0).InstanceRebootAgent.Handlehad the same read and gets the same floor.Evidence
ModuleReloadAgentReadsTheAnnouncedCommitTestmakes the "mirror behind the announcement" moment deterministic: both handlers subscribe before the commit is written.ModuleReloadFaultedBesideActivationTest,ModuleReloadLiveTest,ModuleReloadByRestartTestandInstanceRebootTest. Built-c Release -warnaserrorwith 0 errors.Not established: CI logs only record Warning level, so the exact interleaving in the CI failure was not captured in a log. The mechanism is inferred from where it stopped (after the crashed download, before any report) and pinned deterministically by the test above.
Doc:
Doc/Architecture/ModuleReloadstep 5.Pairs-with: none — no public surface removed (internal
Handlegains an optional parameter)Mirror-sync: none — no catalog change
Recycle: none — process-level agents, picked up by the roll
🤖 Generated with Claude Code