Skip to content

Three terminal _Activity writers bypass ActivityLogAppender.Append, so their hubs wait for the 10-minute idle sweep #3117

Closed

Description

@rbuergi

Found while fixing #3110, and deliberately not folded into that PR — it is a different failure mode with a different severity.

The invariant

ActivityLogAppender.Append fires IMeshNodeStreamCache.ReleaseIfUnwatched on the terminal write (#1435/#1324). That is what retires an activity's per-node hub promptly when the activity finishes, rather than leaving it for the 10-minute idle sweep.

The gap

Several terminal _Activity writers bypass ActivityLogAppender.Append entirely, so they never fire ReleaseIfUnwatched:

writer volume
ActivityLogLogger.PublishSnapshotLocked the highest — every kernel run, script, markdown execution and test run
CodeNodeType.FailActivity every failed code-node activity
BuildProtocolDriver.FinishActivity every build-protocol completion

Why this is bounded, not a leak

Unlike #3110 — where a faulted entry was orphaned from both reclaimers and stayed alive indefinitely with a 45 s heartbeat — these hubs are still reachable by the idle sweep. So the cost is a retention window (up to ~10 minutes per activity) rather than unbounded growth.

That is why it did not belong in #3110's PR: fixing it there would have widened a leak fix into a throughput change, and the two want different evidence. It is worth doing on its own because the highest-volume writer is on the list, and a 10-minute retention multiplied by every kernel and test run is a real steady-state footprint on a busy portal.

Suggested shape

Route those three through ActivityLogAppender.Append, or give them the same ReleaseIfUnwatched call on their terminal path — whichever keeps one implementation rather than a fourth copy.

🚨 That last point is not incidental. #3110's root cause was exactly this: EvictFaultedEntry documents itself as "the single teardown … so they cannot drift", and a fourth site had a hand-rolled copy that disposed only half the state. Adding a parallel release call to three more writers would set up the same drift. Prefer the shared seam.

Provenance

Measured during the #3110 investigation. Re-verify the three call sites before acting — activity writers move.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions