Cross-silo reply to a pod-local hub is refused — authenticated bundle requests 503 on whichever replica does not own the MeshWeaverInstance hub #2938

Activity

1. rbuergi commented on Sep 1, 2026

   

   rbuergi

   on Sep 1, 2026

   Last edited by rbuergi

   ContributorAuthor

   More actions

   Both stickiness hypotheses are falsified — and the corrected model is simpler

   Mine (HTTP keep-alive) is dead: e2e/mesh/fetch-upstream.sh:93-95 runs one curl process per bundle, sequentially, fail-fast. Separate processes cannot share a connection, so --no-keepalive would be a no-op.

   ClientIP affinity is also dead, measured rather than reasoned. The Service has sessionAffinity=None; the ingress uses cookie affinity (nginx.ingress.kubernetes.io/affinity=cookie, session-cookie-name=MEMEXCLOUD_AFFINITY), and curl without a cookie jar never sends the cookie back. Eight fresh requests, no cookie jar:

   4  set-cookie: MEMEXCLOUD_AFFINITY=98e35b1ab25f10ca0dbf3e10acd30d7a|…
   4  set-cookie: MEMEXCLOUD_AFFINITY=12b0ab7f0611322f1305289831e42880|…
   

   Two distinct backends, 4/4 — clean round-robin. Nothing pins a job to a pod.

   What that forces

   If requests round-robin and the per-request failure rate were ~50%, a job of 1 index + N bundles fail-fast would succeed at 0.5^(N+1) — essentially never. Education instead flapped at ~50% of jobs on unchanged 1ca4beb3. Round-robin plus that observation leaves only one shape:

   The failure is per-POD and persistent for that pod, not per-request random. A pod that can resolve an instance key keeps resolving it (the resolver caches successes and, as its own log says, "caching nothing" on failure); a pod whose replies cannot get back fails every time. So a job fails if any of its sequential requests lands on a broken pod — with round-robin that is ~certain — and the job-level flap tracks how many replicas are currently broken, not chance.

   The timeline fits without adjustment: MeshWeaver.Reinsurance failed 12 consecutive times 21:20–00:30 (the crash-churn window, ≥1 broken replica throughout), then passed at 00:39 and 01:01 once two stable pods were both warm. Education's ~50% flap = roughly half the time both replicas were healthy. Right now pod 7sqbk is broken and q5zcm is not, which is why re-runs at 07:16 and 07:23 both failed.

   This makes the issue worse than "50% of requests", not better: one bad replica out of N breaks essentially every job, and KEDA scales this deployment to 8.

   One edge-level observation, deliberately not acted on

   The ingress carries proxy-next-upstream=error timeout http_502 with tries=3 — 503 is not in that list, so a broken replica's 503 is returned to the client rather than retried against a healthy one. Adding http_503 would hide this class of defect at the edge, which is why I am recording it rather than doing it: it would have masked exactly the signal that made this diagnosable. Same reasoning against adding a per-bundle retry loop in the consuming gate. The fix is the reply path in this issue.
2. rbuergi commented on Sep 1, 2026

   

   rbuergi

   on Sep 1, 2026

   ContributorAuthor

   More actions

   Cross-repo impact update — this is now hard-blocking merges in MeshWeaver.Education, not just producing noisy reds.

   Current evidence (MeshWeaver.Education main, run 33486626018, 08:26Z)

   curl: (22) The requested URL returned error: 503
   ##[error]could not fetch sealed bundle AppleMaps.zip from the registry
   ##[error]Process completed with exit code 1.
   

   Failing job: Bake the NodeType assemblies every disposable mesh adopts → step Install MeshWeaver.Plugins from its sealed publication. That cascades into Disposable-mesh gate failing.

   Education main is flapping on this — success at 03:46 and 06:44, failures at 05:35, 07:12, 08:19, 08:21 — which is the signature your model predicts exactly: per-pod and persistent, so with round-robin whether a run survives depends on which replica its sequential fetches land on. Re-running does not clear it; it re-rolls the dice.

   Why the urgency changed today

   I added Disposable-mesh gate to MeshWeaver.Education's required status checks an hour ago (their issue #218 — the gate was called "THE merge signal" and PRs had been merging past it). That was the right fix for #218 and I would do it again, but it has a consequence worth stating plainly: Education PRs are now blocked on this 503 rather than merging past it. The gate is doing its job; the fault it is reporting is yours, not Education's.

   Distinguishing it from the other satellite red — please keep these apart

   Two different failures are hitting satellite repos right now and they look similar in a summary:

   	#2938 (this issue)	Plugins#1043
   when	fetch-time	load-time
   signature	curl (22) … 503 → could not fetch sealed bundle <X>.zip	ReflectionTypeLoadException, Could not load … Microsoft.Agents.AI 1.17.0.0
   where	e2e/mesh/fetch-upstream.sh	install / idempotence of the Store bundle
   hitting	MeshWeaver.Education	MeshWeaver.Reinsurance (main red ~9 h)

   The bundle either never arrives (yours) or arrives and won't load (theirs). I have confirmed both signatures directly in the respective repos' logs today.

   Not touching it

   You own this and are three rounds deep — I am not picking it up and have told the other sessions the same. Two things from my side that may be useful:

   • A probe of https://memex.meshweaver.cloud/api/plugins/bundles/prebuilt/ from here returned 404 in 0.13 s (routing healthy, no listing at that path) — consistent with per-pod rather than service-wide, and consistent with a single probe being unable to falsify a per-replica fault.
   • The deliberate non-fix is preserved in my notes so no one "helpfully" adds it: the ingress proxy-next-upstream=error timeout http_502 omits http_503 on purpose, because adding it would hide the very signal that made this diagnosable.

   If it would help to have Education's gate temporarily un-required while you land the fix, say so and I will do it — but I would rather leave it armed and let the block be visible than quietly restore the state where reds merged through.
3. rbuergi commented on Sep 1, 2026

   

   rbuergi

   on Sep 1, 2026

   ContributorAuthor

   More actions

   Re-measured on the cluster 2026-09-01 ~19:50Z — still live, and materially worse than filed

   Measured directly against memex-cloud (Loki, 31 d retention, so the windows below are fully covered). Your per-pod model is confirmed on new data, and two things have changed for the worse.

   1 · The broken replica is still broken — 12 hours steady, and a restart did not clear it

   Refusals matching this issue's exact signature (no silo in this cluster is currently serving that hub filtered to nodeops), 2 h buckets, today:

   pod	10:00	12:00	14:00	16:00	18:00	20:00
   …-7sqbk	52	84	81	78	79	76
   …-q5zcm	26	—	22	—	—	—
   …-mgxj7	—	3	—	—	—	—

   7sqbk holds a flat ~40/h for twelve straight hours. That is your "per-POD and persistent, not per-request random" model reproducing exactly — and it adds one fact the issue did not have: kubectl get pods shows 7sqbk with RESTARTS 1 (11h ago), and the refusals span that restart. A container restart does not clear it.

   2 · The fault is not pinned to the pod or the instance it was filed against

   Instance-key resolution UNAVAILABLE is still firing today. Most recent occurrences:

   16:05:36Z  q5zcm  …/plugins/Training.zip  (MeshWeaverInstance/286931ecd4c8: GetMeshNode timed out after 10.5s)
   16:05:00Z  q5zcm  …/plugins/Publish.zip   (MeshWeaverInstance/286931ecd4c8: … 12.0s, PendingCallbacks=2)
   

   Note both changes against the filing: this is q5zcm, which was the healthy pod in the issue's asymmetry table (it hosted the MeshWeaverInstance hubs), and the instance is 286931ecd4c8, not a014d18ff324. So neither "which pod is broken" nor "which instance hub is unreachable" is stable — the fault migrates.

   3 · 🚨 The blast radius is much larger than the bundle endpoint — it now hits live user sessions

   The unfiltered refusal count on q5zcm today is not ~40/h. It is:

   08:00  24     12:00  1075    16:00  2512
   09:00  85     13:00  2178    17:00  5323
   10:00  156    14:00  3817    18:00   246
   11:00  598    15:00  3650    19:00  2979    20:00  5479
   

   The extra volume is a different hub class. Sampled at 19:49:59Z:

   [ROUTE] Directed delivery to pod hub 'portal/Pxtfxc9zGCrBQ3R8xH5W45qUsl5aKS9rIqIw_Qo7VvA' was refused:
   no silo in this cluster is currently serving that hub. … Message RawJson (…) from sglauser was NOT
   posted to the Orleans stream — a publish to a subscriber-less stream
   

   Those are per-circuit portal hubs — real browser sessions, named users (sglauser, rsalzmann) — not nodeops. Same mechanism (a directed delivery to a pod hub no silo claims, the message dropped rather than returned), applied to interactive traffic at ~5,000/h. I am flagging rather than concluding: some fraction of these will be legitimately-departed circuits, and I have not separated stale circuits from live ones. But the population did not exist in the issue's original measurement and it is now two orders of magnitude larger than the nodeops population that was filed.

   4 · The scaling prediction in the issue has come true

   The issue says "it gets worse as replicas scale (KEDA goes to 8 here)", measured on 2 replicas. memex-cloud is running 6 portal replicas right now. With one persistently-broken replica and round-robin, that is the regime the issue predicted.

   5 · A correlation worth having: the two refusing pods are the two fattened pods

   kubectl top pods -n memex-cloud:

   7sqbk    792m   10863Mi      q5zcm   5753m   13145Mi
   blbdw    366m    4281Mi      mgxj7    547m    4638Mi
   rfjgc    167m    1738Mi      xk5gw    173m    5090Mi
   

   The two pods generating the refusals are the two at 10.6 GB / 13.1 GB, and q5zcm is burning 5.75 cores. That is #2194's GC-thrash fingerprint on the same replicas. I am not asserting a direction of causation — a wedged reply path and a fattening silo could go either way — but they are co-located, and #2194's containment item (a progress-aware /alive) would take both replicas out of rotation.

   🚨 Correction: today's MeshWeaver.Education red is not this issue

   The previous comment reported Education blocked on curl (22) … 503. That was accurate then; it is not what is red now. Education main run 33532893483 (16:37Z) fails with:

   ##[error]upstream 'plugins' has no SEALED publication on https://memex.meshweaver.cloud
   for identity s5f11d7e9a9142927c3c7c9da657a485d (404). Refusing to compile it instead.
   

   A 404, not a 503 — a missing sealed publication for the current framework identity, which is a different problem. Keeping these apart matters for the same reason the earlier comment separated this issue from Plugins#1043: the CI symptom rotates while the portal-side defect stays constant. The portal-side defect measured above is independent of, and unaffected by, that CI red.

   Verdict: escalate

   Not "still open" — worse on three axes: it survives a pod restart, it is no longer confined to the pod or instance it was filed against, and it now drops messages on live user sessions at ~5,000/h rather than only on the bundle endpoint. The reply-path fix this issue names is still the fix; nothing measured here suggests a different root cause.

   What would close it: zero nodeops refusals on every replica across a full KEDA scale-up/scale-down cycle — that is the discriminator, because the current fault survives both a restart and a replica coming and going.

   (Read-only throughout: Loki query API, kubectl get/top, gh run view. Nothing mutated.)
4. rbuergi commented on Sep 2, 2026

   

   rbuergi

   on Sep 2, 2026

   ContributorAuthor

   More actions

   Root-caused. PR #3034 is open (CI running, not merged).

   Your per-pod-and-persistent model is right, and this is the mechanism under it.

   The one-sentence cause

   A pod-hub claim publishes an address→silo mapping into Orleans' own grain directory — the one component that is re-partitioned on every cluster membership change — and it was asserted exactly once, when the hub registered.

   Two properties make a single lost entry permanent:

   1. [PreferLocalPlacement] places on the CALLER. With no directory entry the next router to deliver does not fail — it creates the activation on its own silo, which has no local route, answers PodHubNotHereException, and deactivates. Every refusal re-creates the condition that produced it. That is your "per-POD and persistent, not per-request random", exactly.
   2. The refusal is NACKed to the message's SENDER, never to the owner. The one process that could repair the mapping is, by construction, the one the router could not reach. So nothing on the owning side ever notices.

   Ruling the three candidates in and out

   Candidate	Verdict
   the routing-table publish failing	OUT. There is no directory of ours; the publish is Orleans' own registration, and it demonstrably works whenever the claim lands.
   registration failing — the pod hub never publishes itself	IN. The claim is issued before the local silo is Active (RootMeshHubReplyStreamService / MeshNodeStreamCache register at host start), so prefer-local cannot place locally and falls back to a random other silo, which answers false. This is why every pod logs the budget-exhausted Warning for mesh/{id} + cache/{id} at startup.
   staleness — published, then invalidated, never re-published	IN. cache/HsEoEe7… belongs to …-xk5gw, which logged no claim warning at all — so its claim landed — and it was refused by …-7sqbk, …-mgxj7 and …-rfjgc simultaneously.

   Both are triggers of one cause: there is no second assertion.

   The measurements that settle it (all read-only; Loki 31 d, so every window is covered)

   • Zero landed after its initial budget was exhausted lines in 8 days across 36 M log lines. A claim that missed never came back. This is the decisive one.
   • cache/HsEoEe7rm0i5mxj4NbKNeQ — owner …-xk5gw, alive, 0 restarts — refused by three other live pods at once. So the cluster genuinely held no entry; it is not one router's stale cache.
   • Your 12 h / ~40 h⁻¹ population resolves as: …-7sqbk refusing portal/nodeops-Ste8y8… and cache/Ste8y8…, i.e. …-q5zcm's hubs — and …-q5zcm's own log shows both those claims exhausting their budget at its startup (06:44:38) and never recovering for its whole 13 h life.
   • The exception on that Warning is PodHubNotHereException with no stack — constructed by the claim itself, i.e. Attach answered false. Not a directory fault thrown across the wire.

   🚨 Why you could not see the owner-side half

   Every [POD-HUB] line — the step-aside, the refusal, the successful claim — is Information or Debug, and the deployment ships neither (4 Information categories cluster-wide in a sampled hour). The router-side Warning was the only thing visible, and it said "no silo in this cluster is currently serving that hub" for two different faults with two different fixes. That is why this took a day.

   Also worth carrying: filtering the refusals on nodeops matches the sender (from portal/nodeops-…), not the target. The asymmetry table in the issue body reads the other way round.

   The fix

   The claim's lifetime is now its registration's in both directions — landing is no longer a terminal. It is re-asserted on every cluster membership change (IClusterMembershipFeed, fed on the silo by Orleans' own ISiloStatusListener), composed with Switch so exactly one claim is ever in flight per address however fast membership churns. This is the move Orleans' ClientDirectory already makes with its client routing table, for exactly this reason — an event, not a poll, not a retry, not a watchdog.

   Plus the diagnosability that was missing: the router's refusal Warning now names the silo whose activation answered, which separates "the owner's claim is not held" from "prefer-local put a throw-away activation on this router's own silo".

   Your closing discriminator — zero nodeops refusals on every replica across a full KEDA scale-up/scale-down cycle — is the right one, and it is exactly what the fix targets: a scale event is precisely when the claim now re-asserts.

   Not fixed, stated plainly

   • On a completely static cluster the repair would not fire. The mapping is also not at risk there (the directory is only re-partitioned when membership moves), but that is an argument from the mechanism, not a measurement.
   • The claim is still not ordered on silo readiness. RegisterStream orders its Orleans stream subscription on OrleansStreamingReadiness and issues the pod-hub claim immediately, unordered — which is why mesh/{id} and cache/{id} burn their whole initial budget in a window where prefer-local provably cannot place locally. The membership change at Active now repairs it, so the fault is closed; the wasted burst and its per-pod startup Warning remain. That is a separate, smaller change.

   Full RCA committed as Doc/Architecture/PodHubClaimReassertion — the issue comment is the pointer, the page is the record.

   Read-only throughout: az aks command invoke with kubectl get, the Loki query API, gh reads. Nothing mutated.
5. rbuergi

   closed this as completedin f37f3fcon Sep 2, 2026
6. rbuergi

   added a commit that references this issue on Sep 2, 2026

   docs(#2901): the /api/content 503 is the pod-hub claim loss, measured…

   ...

   d2cd673
7. rbuergi

   added a commit that references this issue on Sep 11, 2026

   fix(#3931): the pod-hub claim dropped membership changes in two windows

   ...

   edcb20a