Repository navigation
Recompile-set derivation times out and abandons the affected recompiles with no retry or failure detail #2887
Description
What is failing
When code lands on the mesh — a GitSync update, a plugin delta-install, a boot sweep — the platform first derives the recompile set: the affected closure of the owning NodeType, every shared= sharer anywhere on the mesh that pulls the changed sources in, ordered dependencies-first (Release to Production §(B)). In this event that derivation failed outright with System.TimeoutException, so the recompiles it was computing never ran. Under the delivery contract the mesh then keeps serving the last good assemblies — silent staleness, no user-visible error.
Probable cause — medium confidence
The bare The operation has timed out. — with no stack, no failing query, and no elapsed time recorded — points at an underlying mesh/grain call exceeding its timeout budget during source discovery / closure computation, not a deterministic defect in the closure math. Corroborating, within the same night on the same instance: an Orleans memory-stream dequeue timed out at its 30-second message budget roughly two minutes earlier (suppressed incident 17309c2fb8611d33), and the same component logged release-request timeouts hours before (incident Admin/_LogIncident/19be79b3588b8152, filed as #1549). Which specific hop timed out cannot be confirmed from the log — the log site records none of the detail needed to say, which is itself part of the defect.
Impact
Low by the numbers (see evidence footer) — a transient, self-limited event; the direct cost is triage noise. The deferred cost is the real one: whatever change triggered the derivation may still not be live, and this path has no documented auto-retry, so a recurrence strands stale assemblies silently. Timeout-class events clustered on this instance overnight — watch for recurrence before closing.
Where to look
MeshWeaver.Graph.NodeTypeRecompileExtensions— the[Recompile]derivation failure log site. It emits neither the failing query nor the elapsed time nor a stack; add them, and decide whether a transient timeout should retry rather than abandon the derivation (abandoning = stale assemblies with no retry).- The derivation runs cross-partition source-discovery queries over hub/grain calls (30s budget class) — correlate with Orleans message timeouts in the same window.
- A sequencing change in this exact machinery shipped the day before (Recycle's hub dispose now waits for the release-request stamp — What's New 2026-08-30). Confirm which build the instance was running and whether the new ordering is in scope for reproduction.
- Known related, not duplicates: #1549 (same component, but the release step of the pipeline, different fingerprint) and incident
Admin/_LogIncident/39d3318bdc030838(same log line, but a pool-teardownOperationCanceledExceptionmisclassified as failure; drafted as a log-classification fix with no issue number recorded on the node). One log-site cleanup can reasonably cover that classification as well.
Evidence
| Fingerprint | ef6b38e71ee0561f |
| Category | MeshWeaver.Graph.NodeTypeRecompileExtensions |
| Severity | Error |
| Exception | System.TimeoutException |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-9bf79b774-tk88b |
| Occurrences | 1 |
| First seen | 2026-08-31 05:57:01Z |
| Last seen | 2026-08-31 05:57:01Z |
Recent log lines
2026-08-31 05:57:01Z memex-portal-deployment-9bf79b774-tk88b fail: MeshWeaver.Graph.NodeTypeRecompileExtensions[0]
[Recompile] Deriving the recompile set failed.
System.TimeoutException: The operation has timed out.
Opened automatically from Admin/_LogIncident/ef6b38e71ee0561f. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Aug 31, 2026
Same incident as #2885 — one pod exhausting memory, filed four times
All four reports share a pod, a namespace and a five-minute window, each with
Occurrences: 1:One pod ran out of memory. The OOM surfaced first on the write path, then on the read path three minutes later; the two timeouts follow because a thrashing pod misses every bound it is held to.
🚨 The two
TimeoutExceptions are consequences, not defects. A sync hub that cannot completeBuildupActioninside 120 s, and a recompile-set derivation that gives up, are what a memory-starved process looks like from the inside — nothing is wrong with either bound. Filed separately they send four people into four subsystems (Orleans messaging, serialization, hub startup, the recompile lane) for one event, and each would find its own code behaving correctly.Consolidating into #2885 — chronologically first, and the closest to the root condition — where the actual question belongs: what allocated enough to exhaust the pod? Both OOM sites are
Utf8JsonWriter.TranscodeAndWriteRawValue/JsonReaderHelper.TranscodeHelper, i.e. a single oversized JSON token, and both name the import path. An oversized import payload is the hypothesis worth testing first.(Filer note: fingerprinting by exception site is right for dedup but blind to correlation. Four reports from one pod inside five minutes, each
Occurrences: 1, is a signal in itself — the same pod+window grouping that makes this obvious to a reader could suppress the follow-on reports automatically.)