Repository navigation
Plugin registry's instance-key resolution outlasted consumers' boot reconcile budget, deferring package reconciliation to the PENDING ledger #3119
Description
What is failing. The memex plugin registry's catalog surface (GET /api/plugins) answered 503 — instance-key resolution temporarily unavailable — for a consumer pod's entire boot-reconcile budget (4 attempts, ~26 s), so the startup package reconcile was deferred. The consumer is NOT broken: RegistryUpdateReconciler behaved as designed (#2888) — Pending recorded on Plugins/_RegistryReconcileLedger, admins notified, reconcile deferred to the next feed read. The real question is registry-side: why was memex.meshweaver.cloud's key resolution unavailable that long at 2026-09-02T13:37Z?
Probable cause. High confidence the consumer's deferral worked as documented in Plugin Update on Green Build — this is that designed failure mode, not a crash. Registry-side cause unclear from one occurrence: most plausibly a registry pod rollout or storage hiccup; no registry-side logs in evidence, so that half is a hypothesis, not a root cause.
Impact. Low: one occurrence, one pod, self-healing — the ledger drains on the next successful feed read. Residual risk is by-design: consumers booting inside such a window defer reconciliation until the registry answers again, so severity tracks how often the registry's key resolution goes unavailable.
Where to look. Consumer (as designed): RegistryPackageSource.ListPackages (src/MeshWeaver.PluginCatalog/RegistryPackageSource.cs:91), RegistryUpdateReconciler — check the ledger drains to feed-read once the registry is back. Registry (the actual event): InstanceRegistryAuthenticator on memex.meshweaver.cloud at 13:37Z. Context: Plugin Registry, Instance Identity and Setup.
Evidence
| Fingerprint | 0f779d9103a3366a |
| Category | MeshWeaver.PluginCatalog.RegistryUpdateReconciler |
| Severity | Error |
| Exception | MeshWeaver.PluginCatalog.RegistryResponseException |
| Top frame | MeshWeaver.PluginCatalog.RegistryPackageSource.<>c__DisplayClass16_0.<<ListPackages>b__0>d.MoveNext() |
| Namespace | memex |
| Pods | memex-portal-deployment-58c88885f8-h5hfs |
| Occurrences | 1 |
| First seen | 2026-09-02 13:37:12Z |
| Last seen | 2026-09-02 13:37:12Z |
Recent log lines
2026-09-02 13:37:12Z memex-portal-deployment-58c88885f8-h5hfs fail: MeshWeaver.PluginCatalog.RegistryUpdateReconciler[0]
[RegistryUpdate] could not read the package feed of Plugins (https://memex.meshweaver.cloud) after 4 attempt(s) — installed packages from it are NOT reconciled this boot. Recorded as PENDING on Plugins/_RegistryReconcileLedger; the next successful feed read against this registry (a catalog open, an install) runs the reconcile. Platform admins have been notified.
MeshWeaver.PluginCatalog.RegistryResponseException: Registry catalog list failed (503): {"error":"Instance-key resolution is temporarily unavailable — retry shortly. This is NOT a statement about your key or your grant."}
at MeshWeaver.PluginCatalog.RegistryPackageSource.<>c__DisplayClass16_0.<<ListPackages>b__0>d.MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.PluginCatalog/RegistryPackageSource.cs:line 91
--- End of stack trace from previous location ---
at MeshWeaver.Mesh.Threading.IoPool.<>c__DisplayClass31_0`1.<<InvokeCore>b__0>d.MoveNext()
Opened automatically from Admin/_LogIncident/0f779d9103a3366a. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 2, 2026
Registry-side cause, measured on memex-cloud 2026-09-02 14:05–15:25Z (Loki, namespace
memex-cloud+ ingressapp-routing-system):503responses onbundles/prebuiltin the same window, all from the CI runner IP.[CrossSchema] SLOW810 / 160 / 552 / 783 per 10 min on four pods (the unanchored fan-outs inventoried on RCA 2026-08-25: sync/bake recompile waves fatten type-hosting silos to 20 GB — superseded builds stay rooted behind the Recycle banner, and a degraded pod never leaves rotation #2194); pod xk5gw at 11.7 GB with 104 GC stalls in 30 min.MeshWeaverInstance/<id>node with a 10 s budget, and the owning per-node hub on a fan-out-starved pod does not answer in 10 s → 503. It is not a rollout or a storage hiccup.Consequences today: MeshWeaver.Reinsurance run 33641740836 failed twice (14:28Z compile-check, 15:24Z test-repos) on exactly this 503 while fetching the sealed set → publish-bake skipped → no Underwriting/Crm bundles for the live identity → memex.systemorph.com compiles 12 Underwriting types from source after every roll (#3129 context). Every satellite gate and bake depends on this endpoint.
Fix shape (registry side): the instance-key → instance mapping must not be a fresh
GetMeshNodewith a 10 s budget on the request hot path. Hold the instance node through the process-wideIMeshNodeStreamCache(GetMeshNodeStream(path), an instance singleton owned by the registry module, live via the change feed — no TTL cache), so a request reads the current value in memory and only the FIRST read pays the owner round-trip; an unresolved key answers 404/401 as today, a not-yet-hydrated stream answers 503 only until the first frame arrives.Merge pull request #3137 from Systemorph/fix/3119-instance-key-resolu…
rbuergi commentedon Sep 2, 2026
Registry side fixed by #3137 (merged 2026-09-02 18:05Z, first entry through the new merge queue): instance-key resolution reads the index/instance/grant legs through the process-wide stream cache (live children listing for existence, owner mirror for content), so a request answers from memory and only a cold first frame pays the owner round-trip; the verdict cache is gone with it. Reaches memex.meshweaver.cloud with its next roll. Plugins #1201 adapts the consumer-side test and lands once Plugins main is green (#1188).