Repository navigation
ContentRoute times out fetching static content when target hub is unreachable #1563
Description
What is failing
The MeshWeaver.ContentRoute component fails with a TimeoutException when it dispatches a GetDataRequest to a target hub (e.g., Skill) to serve static content (e.g., an Open Graph image at Skill/content/og-card.png). The request waits the full 1-minute timeout and then logs the failure.
Probable cause
Confidence: Medium. The exception message states "the target hub was not found" — the Skill hub was unreachable or not initialized when the request arrived. Given this is a single occurrence on a single pod, never repeated, the most likely explanation is a transient hub routing glitch (hub recycling, initialization race, or a brief network partition). The code does not handle the unreachable-hub case gracefully and instead lets the full 1-minute timeout expire before surfacing a fail:-level log.
Impact
Low. One occurrence on one pod (memex-portal-deployment-79867669f4-f2zzl), never seen again. A single user's request for the Open Graph image (og-card.png) would have failed, potentially affecting link preview rendering on that one page load. No cascading or widespread effect.
Where to look
- Component:
MeshWeaver.ContentRoute— the handler that receives content-read requests, dispatchesGetDataRequestmessages to target hubs, and awaits responses. - The timeout for static content reads should be reviewed (1 minute is excessive for a small image).
- Consider adding a fallback (e.g., serving a default placeholder or returning a 404 fast) when the target hub is unreachable, rather than blocking for a full timeout period.
- Related architecture: MeshWeaver Messaging — MessageHub, Hub Initialization Failure.
Evidence
| Fingerprint | ade144cc74e8454b |
| Category | MeshWeaver.ContentRoute |
| Severity | Error |
| Exception | System.TimeoutException |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-79867669f4-f2zzl |
| Occurrences | 1 |
| First seen | 2026-08-14 12:32:54Z |
| Last seen | 2026-08-14 12:32:54Z |
Recent log lines
2026-08-14 12:32:54Z memex-portal-deployment-79867669f4-f2zzl fail: MeshWeaver.ContentRoute[0]
Content read failed for Skill/content/og-card.png
System.TimeoutException: No response received in hub mesh/ib4ky_C_f0OTpLJYI50_MA within 00:01:00 for request GetDataRequest (id=jD7z-rAqBEqb7Y0QoxkRkg) → target Skill. The request may have been undeliverable or the target hub was not found.
Opened automatically from Admin/_LogIncident/ade144cc74e8454b. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Aug 14, 2026
The mechanism here is almost certainly #1486, now fixed in #1612 — recording the reasoning rather than closing this on it.
Why this is the failure leg, not a missing timeout
The read waits its full 60 s and then surfaces "the target hub was not found". That wording is the router's own NotFound verdict — so the verdict was reached; what did not happen is its delivery.
RoutingGrain.PostFailurereturned every NACK by publishing to the sender's Orleans stream, unconditionally, while every forward delivery to a co-hosted hub short-circuits on the local route table and never touches a stream. And a publish to a memory stream with no live subscriber succeeds — nothing faults, so the router logs a successful send and the caller waits out its timeout for an answer that was discarded.The sender here is the portal's own hub (
ContentFileResolvercallshub.Observe(new GetDataRequest(...), o => o.WithTarget(targetAddress))), i.e. co-hosted with the router. So its NACK went down exactly that path, and #1612 makes it take the local route instead — the answer arrives immediately and the 60 s bound is never reached.Why I am not "fixing the timeout"
The issue's own Where to look suggests reviewing the 1-minute bound and adding a fallback. With the NACK delivered, the bound stops being load-bearing: an unreachable hub is answered promptly, so nothing waits it out. Shortening it instead would have converted a silent minute into a shorter silent failure while leaving the lost answer in place — and AGENTS.md rules that out.
Left open deliberately
One occurrence, no repro. The honest position is that the mechanism is identified and its cause is fixed, not that this specific symptom is proven gone. If a 60 s content read recurs after #1612 ships, that is informative rather than a mystery: it would mean the sender was NOT co-hosted, which is #1486's remaining open question — a publish with no subscriber still cannot distinguish delivered from discarded for a genuinely remote sender.
Worth closing once a week of clean logs has passed, or reopening with the new evidence.
Merge pull request #1824 from Systemorph/fix/unreachable-hub-degradation
systemorph-com commentedon Sep 17, 2026
Reopened — this fault is happening again.
Still occurring: 26 more since the last update (27 total).
memex-portal-deployment-79867669f4-f2zzl,memex-portal-deployment-66b79847f8-zdbtz,memex-portal-deployment-d976fd57-ct22g,memex-portal-deployment-648c756f74-smlsb,memex-portal-deployment-f96d46ddf-zn9nl,memex-portal-deployment-f96d46ddf-v6zw2,memex-portal-deployment-7d54845c86-brdng,memex-portal-deployment-5d6f77b68c-vsmvd,memex-portal-deployment-7cb5549c57-8c9gr,memex-portal-deployment-7cb5549c57-xxg55,memex-portal-deployment-89d7df89f-lrbc4,memex-portal-deployment-76b967b5fb-h8kzgRecent log lines
rbuergi commentedon Sep 17, 2026
Re-closed — the reopen above was on stale evidence (#4577).
The recurrence bot reopened this at
2026-09-17T07:52:34Zunder the headline "this fault is happening again", but the occurrence its own comment cites is OLDER than the close it reverted:Last seen:2026-08-17 18:29:25Z2026-08-18T13:33:28ZSo the counter moved, not the clock — the fault's newest occurrence predates the close, which is the one thing "still happening" has to mean. Root cause and fix:
LogIncidentFiler.Reopenreopened on the sole condition that the issue was closed, while the comment-debt clock pays a final count for incidents whose fault has stopped (the fix, Systemorph/MeshWeaver.Plugins#2014, is written but NOT yet merged — it makes a reopen requireincident.LastSeen > issue.ClosedAt).This restores the close exactly as it stood; nothing is asserted about the defect itself. If the fault happens again, the next occurrence postdates this close and reopens the issue on evidence that means what it says.
One of 53 issues whose reopen was measured stale (129 bot reopens on 2026-09-17, 125 of them inside the 07:51-07:53Z sweep; 76 postdate their close and were left open). Measured and re-closed 2026-09-17 by Claude Opus 5.