Recycle jobs for the Hosting platform's own NodeTypes are enqueued without their declaration seal and refused

What is failing

After a platform build releases a new build of the Hosting module's own NodeTypes (Hosting/Job, Hosting/Queue, OperationsBoard, PlatformBuildInbox), the recycle-on-new-build rule enqueues the corresponding recycle jobs on the instance's execution queue (Hosting/Queues/instances/memex), but no declaration seal is written at {job}/seal. The JobExecution executor refuses each job — "a job runs as System only with the declaration the control plane wrote when it enqueued it" — and the recycle never runs, so live activations of those NodeTypes are never re-bound to the released build.

Probable cause

The seal write on enqueue is skipped or lost specifically when the job targets the Hosting module's own NodeTypes — i.e. the component that writes the seal is part of the very module whose build just released and whose hubs are about to be recycled. Medium confidence. The evidence:

All 4 failures in the burst target Hosting-native NodeTypes released in one platform build (…v92-c003e001…), while every recycle job in the same window targeting other NodeTypes (Radzen/Gallery, GoogleMaps/Gallery, Approvals/…) on the same queue and pod succeeded and logged request … written as System.
The seal nodes genuinely do not exist: no descendants under any of the four failed job paths.
Routing was identical for both groups (enqueued by node-type-build, received from Hosting/Queues/operator/ops, admitted) — the jobs die at the System-authorization check, before running.
A refused job is marked terminal after attempt 1 despite maxAttempts: 2, so it never self-heals — the re-binding is silently skipped until someone recycles by hand.

Impact

4 occurrences over ~26 seconds, one pod (memex-portal-deployment-…), the memex instance's execution queue. Consequence: automatic re-bind of the platform's own NodeTypes after a platform update silently does not happen on that instance — a broken secondary maintenance path with an easy manual workaround (recycle the instance). Not user-visible beyond stale Hosting hubs.

Where to look

The JobExecution System-authorization for the recycle executor and the enqueue-side seal writer, in the Hosting/Queues / Hosting/Job plugin (MeshWeaver.Plugins, landed via chore: retire the #2721 type-forwards.allow entries (stale the moment #2721 merged) #2730 — see the jobs architecture, decision 1).
The recycle-on-new-build enqueue path when the released NodeType belongs to the Hosting module itself.
Failing job nodes (state Failed, reason recorded, seal absent): recycle-hosting-job-156a5a50, recycle-hosting-queue-7b60ea4f, recycle-hosting-operationsboard-57ac3f4d, recycle-hosting-platformbuildinbox-b5f4f427, plus their dispatch mirrors under Hosting/Queues/operator/ops.
A succeeding sibling for contrast: recycle-radzen-gallery-b4cee495.

Evidence


Fingerprint	`65b3605d1ad0597b`
Category	`JobExecution`
Severity	Error
Namespace	`memex`
Pods	`memex-portal-deployment-6bcdbb54fc-g9gh4`
Occurrences	4
First seen	2026-10-06 22:02:17Z
Last seen	2026-10-06 22:02:43Z
Routing	not determined — no configured route matches the category `JobExecution`. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.

Recent log lines

2026-10-06 22:02:17Z memex-portal-deployment-6bcdbb54fc-g9gh4 fail: JobExecution[0]
      [Job] Hosting/Queues/instances/memex/recycle-hosting-job-156a5a50: refused: this job has no declaration seal at Hosting/Queues/instances/memex/recycle-hosting-job-156a5a50/seal — a job runs as System only with the declaration the control plane wrote when it enqueued it
2026-10-06 22:02:19Z memex-portal-deployment-6bcdbb54fc-g9gh4 fail: JobExecution[0]
      [Job] Hosting/Queues/instances/memex/recycle-hosting-queue-7b60ea4f: refused: this job has no declaration seal at Hosting/Queues/instances/memex/recycle-hosting-queue-7b60ea4f/seal — a job runs as System only with the declaration the control plane wrote when it enqueued it
2026-10-06 22:02:28Z memex-portal-deployment-6bcdbb54fc-g9gh4 fail: JobExecution[0]
      [Job] Hosting/Queues/instances/memex/recycle-hosting-operationsboard-57ac3f4d: refused: this job has no declaration seal at Hosting/Queues/instances/memex/recycle-hosting-operationsboard-57ac3f4d/seal — a job runs as System only with the declaration the control plane wrote when it enqueued it
2026-10-06 22:02:43Z memex-portal-deployment-6bcdbb54fc-g9gh4 fail: JobExecution[0]
      [Job] Hosting/Queues/instances/memex/recycle-hosting-platformbuildinbox-b5f4f427: refused: this job has no declaration seal at Hosting/Queues/instances/memex/recycle-hosting-platformbuildinbox-b5f4f427/seal — a job runs as System only with the declaration the control plane wrote when it enqueued it

_{Opened automatically from Admin/_LogIncident/65b3605d1ad0597b. Recurrences are folded into this issue rather than opening new ones.}
_{It also stands for the whole log site edf965fea70ca564: other fingerprints of this site fold in here as comments rather than opening tickets of their own.}

Still occurring: 1 more since the last update (5 total).

Last seen: 2026-10-07 19:13:29Z
Pods: memex-portal-deployment-6bcdbb54fc-g9gh4, memex-portal-deployment-dcf6f66d5-5khwq

Recent log lines

2026-10-06 22:02:28Z memex-portal-deployment-6bcdbb54fc-g9gh4 fail: JobExecution[0]
      [Job] Hosting/Queues/instances/memex/recycle-hosting-operationsboard-57ac3f4d: refused: this job has no declaration seal at Hosting/Queues/instances/memex/recycle-hosting-operationsboard-57ac3f4d/seal — a job runs as System only with the declaration the control plane wrote when it enqueued it
2026-10-06 22:02:43Z memex-portal-deployment-6bcdbb54fc-g9gh4 fail: JobExecution[0]
      [Job] Hosting/Queues/instances/memex/recycle-hosting-platformbuildinbox-b5f4f427: refused: this job has no declaration seal at Hosting/Queues/instances/memex/recycle-hosting-platformbuildinbox-b5f4f427/seal — a job runs as System only with the declaration the control plane wrote when it enqueued it
2026-10-07 19:13:29Z memex-portal-deployment-dcf6f66d5-5khwq fail: JobExecution[0]
      [Job] Hosting/Queues/instances/memex/recycle-mcp-20261007191325844-8de991: refused: this job has no declaration seal at Hosting/Queues/instances/memex/recycle-mcp-20261007191325844-8de991/seal — a job runs as System only with the declaration the control plane wrote when it enqueued it

Closing as fixed. I verified this on the deployed control instance (memex.systemorph.com) on 2026-10-08. Every read below is get/search over the systemorph MCP.

Both bursts came from writers that lacked the seal at the time. Neither writer does now.

The 2026-10-06 22:02Z burst (recycle-hosting-{job,queue,operationsboard,platformbuildinbox}-…) happened in the same minute the seal mechanism went live. Plugins c12e7664e (merged 2026-10-06 06:03Z) made the executor require a seal. The Hosting/JobSeal NodeType node on the control instance has createdDate: 2026-10-06T22:02:05Z, 12 s before the first refusal. So the recycle-on-new-build jobs that the Hosting release enqueued were the first jobs judged against a seal. They were enqueued while the seal's own type was still being installed. This is a one-time transition, not a property of Hosting's own types.
- Since then, Hosting's own types recycle cleanly. For example, Hosting/Queues/instances/memex/recycle-hosting-job-922dc1b3 (2026-10-07 13:33Z, rule recycle-on-new-build for Hosting/Job) is state: Succeeded, attempts 1, and 363 live activation(s) disposed over 5 process(es). The same holds for recycle-hosting-queue-7cfec05d, …-jobseal-7b847056, …-platformbuildinbox-5800d45f, …-stuckcondition-* and …-moduleinventory-* (13:29–14:06Z, versions 7–8, all routed). Their seals exist: search namespace:Hosting/Queues scope:descendants nodeType:Hosting/JobSeal lists a seal at both operator/ops/{job}/seal and instances/memex/{job}/seal for every job up to 2026-10-07 14:53Z.
The 2026-10-07 19:13Z recurrence (recycle-mcp-20261007191325844-8de991) was an enqueue_recycle MCP job. At that time the MCP intake (src/MeshWeaver.Mcp/RecycleJobIntake.cs) wrote no seal at all, and get …/operator/ops/recycle-mcp-20261007191325844-8de991/seal answers Not found. Plugins 08154f20c ("Seal MCP recycle jobs before queue admission", 2026-10-07 19:27Z, 14 minutes later) makes it write the seal first, exactly as JobQueues.EnqueueJob does.

What I did NOT establish:

I did not fold the 22:02Z logs to say which of the two in-flight builds the enqueuer ran. The transition reading rests on the timestamps above.
No recycle job has been enqueued on the control instance since 2026-10-07 19:13Z (the newest Hosting/Job under Hosting/Queues). So the MCP fix has not yet been exercised on the deployed portal by a fresh job. Its merge is the evidence for it, not a live run.
"A refused job is terminal after attempt 1 despite maxAttempts: 2" is the documented behaviour of a seal refusal (JobSeals.Refusal: final). I left it unchanged.

Navigation Menu

Repository navigation

Recycle jobs for the Hosting platform's own NodeTypes are enqueued without their declaration seal and refused #6217

What is failing

Probable cause

Impact

Where to look

Evidence

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Recycle jobs for the Hosting platform's own NodeTypes are enqueued without their declaration seal and refused #6217

Description

What is failing

Probable cause

Impact

Where to look

Evidence

Activity

systemorph-com commented on Oct 7, 2026

rbuergi commented on Oct 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions