Repository navigation
ContentRoute fails with DeliveryFailureException when target hub activation throws NullReferenceException #1693
Description
What is failing
MeshWeaver.ContentRoute fails with a DeliveryFailureException when it attempts to read static content (AdvancedBusinessRules/content/videos/advanced-business-rules.mp4) and the target hub (AdvancedBusinessRules) throws a NullReferenceException during activation. The exception surfaces as a DeliveryFailureException wrapping the activation failure.
Probable cause
Confidence: High. The inner exception is explicit: "Hub activation failed for AdvancedBusinessRules: Object reference not set to an instance of an object." This is a NullReferenceException inside the AdvancedBusinessRules hub's initialization code — it is not a transient messaging glitch. The hub itself has a bug (null dereference) during startup, which prevents it from receiving or processing any subsequent GetDataRequest messages. This is a distinct failure mode from the related incident (#1563 / issue 1563) where the hub was unreachable (timeout). Here the hub is reachable but crashes on init.
Impact
Low. One occurrence on one pod (memex-portal-deployment-c89ddfbb6-nl4pp), one timestamp. Only the AdvancedBusinessRules hub is affected. A single user request for the video file (advanced-business-rules.mp4) would have failed (likely a 404 / error response for that video player). No wider service degradation.
Where to look
- Component:
MeshWeaver.ContentRoute— the handler that dispatchesGetDataRequestand unwraps theDeliveryFailureException. The catch/retry logic should distinguish activation failures from delivery failures and log the inner exception clearly. - Hub:
AdvancedBusinessRules— the hub's activation/constructor logic contains a null dereference. Look at the hub'sOnActivateor constructor for uninitialized fields / null service dependencies. The same path was the target of the related TimeoutException in ContentRoute times out fetching static content when target hub is unreachable #1563, suggestingAdvancedBusinessRulesmay have an underlying stability issue. - Related: issue ContentRoute times out fetching static content when target hub is unreachable #1563 on
Systemorph/MeshWeaver(timeout on same content path, same target hub).
Evidence
| Fingerprint | 83ea6fdcf6fa198a |
| Category | MeshWeaver.ContentRoute |
| Severity | Error |
| Exception | MeshWeaver.Messaging.DeliveryFailureException |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-c89ddfbb6-nl4pp |
| Occurrences | 1 |
| First seen | 2026-08-16 21:25:40Z |
| Last seen | 2026-08-16 21:25:40Z |
Recent log lines
2026-08-16 21:25:40Z memex-portal-deployment-c89ddfbb6-nl4pp fail: MeshWeaver.ContentRoute[0]
Content read failed for AdvancedBusinessRules/content/videos/advanced-business-rules.mp4
MeshWeaver.Messaging.DeliveryFailureException: Hub activation failed for AdvancedBusinessRules: Object reference not set to an instance of an object.
Opened automatically from Admin/_LogIncident/83ea6fdcf6fa198a. Recurrences are folded into this issue rather than opening new ones.
Activity
Merge pull request #1824 from Systemorph/fix/unreachable-hub-degradation
systemorph-com commentedon Sep 17, 2026
Reopened — this fault is happening again.
Still occurring: 18 more since the last update (19 total).
memex-portal-deployment-c89ddfbb6-nl4pp,memex-portal-deployment-c89ddfbb6-vzqm7,memex-portal-deployment-66cc6466bf-frwpkRecent log lines
rbuergi commentedon Sep 17, 2026
Re-closed — the reopen above was on stale evidence (#4577).
The recurrence bot reopened this at
2026-09-17T07:52:21Zunder the headline "this fault is happening again", but the occurrence its own comment cites is OLDER than the close it reverted:Last seen:2026-08-17 00:29:32Z2026-08-18T13:33:28ZSo the counter moved, not the clock — the fault's newest occurrence predates the close, which is the one thing "still happening" has to mean. Root cause and fix:
LogIncidentFiler.Reopenreopened on the sole condition that the issue was closed, while the comment-debt clock pays a final count for incidents whose fault has stopped (the fix, Systemorph/MeshWeaver.Plugins#2014, is written but NOT yet merged — it makes a reopen requireincident.LastSeen > issue.ClosedAt).This restores the close exactly as it stood; nothing is asserted about the defect itself. If the fault happens again, the next occurrence postdates this close and reopens the issue on evidence that means what it says.
One of 53 issues whose reopen was measured stale (129 bot reopens on 2026-09-17, 125 of them inside the 07:51-07:53Z sweep; 76 postdate their close and were left open). Measured and re-closed 2026-09-17 by Claude Opus 5.
systemorph-com commentedon Sep 26, 2026
Reopened — this fault is happening again.
Still occurring: 1 more since the last update (20 total).
memex-portal-deployment-56fbdcd48f-sz5st,memex-portal-deployment-c89ddfbb6-nl4pp,memex-portal-deployment-c89ddfbb6-vzqm7,memex-portal-deployment-66cc6466bf-frwpkRecent log lines
Re-addressed by the current identity function: this incident inherited
83ea6fdcf6fa198a(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Oct 8, 2026
Closing. The defect in the title was fixed by #1824 (merged 2026-08-18): a hub whose activation throws
NullReferenceExceptionnow degrades in 10 s with an attributable answer (HubConstructionFailureReason, pinned bytest/MeshWeaver.Hosting.Orleans.Test/HubConstructionFailureReasonTest.cs). The last log line of that shape is from 2026-08-17 00:29Z, before the fix.The 2026-09-26 reopen cites a different defect that shares this fingerprint (category + exception type):
Delivery to 'Doc/Architecture' failed: Grain placement operation timed outon 2026-09-24 10:02Z. That belongs to the placement / silo-stall family (#5388), not to a null-reference activation.sev:L sweep, Claude Opus 5.5 (agent session).