SelfUpdateRouting source node contains a literal NUL byte, making it unstorable in PostgreSQL and aborting CreateNodes batches mid-persist

What is failing

A CreateNodes batch in the memex namespace partially persists and then fails while storing the node Hosting/Deployment/Source/SelfUpdateRouting: its serialized content's code property contains a literal NUL character (U+0000), which PostgreSQL's jsonb cannot represent (server error 22P05). The batch is aborted mid-landing, leaving a partial write.

Probable cause

High confidence: a literal NUL byte was typed or pasted into the source file backing SelfUpdateRouting's code property, most likely meant as a field/key separator. The adapter's BuildUpsertAsync detects this and throws UnstorableContentException before the write, so the PostgreSQL adapter is behaving as designed — the defect is in the data, not the storage layer. The message itself points to the fix: replace the NUL with an explicit escape such as U+001F UNIT SEPARATOR. A secondary, framework-level gap: the NUL is only rejected mid-persist, after earlier nodes in the batch have already landed, so validation at Create/Patch time would fail the whole batch atomically instead.

Impact

Low so far: 2 occurrences across 2 pods within ~90 minutes (2026-09-15 19:01–20:31 UTC). But the SelfUpdateRouting node is effectively unwritable until the byte is removed, and any retry of the batch keeps failing at the same node — so a stuck deploy/update path if this node is part of an automated pipeline.

Where to look

Offending data: memex namespace, node Hosting/Deployment/Source/SelfUpdateRouting, content property code — strip the literal NUL byte from the underlying source file (use U+001F UNIT SEPARATOR as the key separator instead).
Detection site: PostgreSqlStorageAdapter.BuildUpsertAsync in the MeshWeaver.Hosting.PostgreSql plugin, which throws UnstorableContentException pre-write.
Follow-up worth considering: reject NUL-bearing content at Create/Patch time so bad data fails before any nodes land, rather than as a partial batch.

Evidence


Fingerprint	`6a83b33b0e74b4ed`
Category	`MeshWeaver.Mesh.CreateNodes`
Severity	Error
Exception	`MeshWeaver.Hosting.PostgreSql.UnstorableContentException`
Top frame	`MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.BuildUpsertAsync(MeshNode node, JsonSerializerOptions options, Nullable`1 expectedVersion)`
Namespace	`memex`
Pods	`memex-portal-deployment-6546c779d4-4ngqv`, `memex-portal-deployment-854cd8cd6d-r4cn5`
Occurrences	2
First seen	2026-09-15 19:01:10Z
Last seen	2026-09-15 20:31:54Z

Recent log lines

2026-09-15 19:01:10Z memex-portal-deployment-6546c779d4-4ngqv fail: MeshWeaver.Mesh.CreateNodes[0]
      [CreateNodes] failed AFTER 3 node(s) were persisted — reporting the partial landing
      MeshWeaver.Hosting.PostgreSql.UnstorableContentException: Node 'Hosting/Deployment/Source/SelfUpdateRouting' cannot be persisted: its serialized content contains 1 NUL character(s) (U+0000), which PostgreSQL's jsonb type cannot represent — first occurrence in content property 'code'. JSON permits the character but jsonb stores DECODED text, and PostgreSQL text cannot hold a NUL byte, so the server rejects it with 22P05 (unsupported Unicode escape sequence). This is a property of the DATA, not of its size and not of the column: no widening makes it storable, so the NUL has to go at the source. A literal NUL typed into a source file is the usual cause — write a key separator as an explicit escape (U+001F UNIT SEPARATOR) instead.
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.BuildUpsertAsync(MeshNode node, JsonSerializerOptions options, Nullable`1 expectedVersion) in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 744
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct, Nullable`1 expectedVersion) in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 636
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.<>c__DisplayClass43_0.<<Write>b__0>d.MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 482
      --- End of stack trace from previous location ---
         at MeshWeaver.Mesh.Threading.IoPool.<>c__DisplayClass43_0`1.<<InvokeCore>b__0>d.MoveNext()
2026-09-15 20:31:54Z memex-portal-deployment-854cd8cd6d-r4cn5 fail: MeshWeaver.Mesh.CreateNodes[0]
      [CreateNodes] failed AFTER 2 node(s) were persisted — reporting the partial landing
      MeshWeaver.Hosting.PostgreSql.UnstorableContentException: Node 'Hosting/Deployment/Source/SelfUpdateRouting' cannot be persisted: its serialized content contains 1 NUL character(s) (U+0000), which PostgreSQL's jsonb type cannot represent — first occurrence in content property 'code'. JSON permits the character but jsonb stores DECODED text, and PostgreSQL text cannot hold a NUL byte, so the server rejects it with 22P05 (unsupported Unicode escape sequence). This is a property of the DATA, not of its size and not of the column: no widening makes it storable, so the NUL has to go at the source. A literal NUL typed into a source file is the usual cause — write a key separator as an explicit escape (U+001F UNIT SEPARATOR) instead.
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.BuildUpsertAsync(MeshNode node, JsonSerializerOptions options, Nullable`1 expectedVersion) in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 744
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct, Nullable`1 expectedVersion) in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 636
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.<>c__DisplayClass43_0.<<Write>b__0>d.MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 482
      --- End of stack trace from previous location ---
         at MeshWeaver.Mesh.Threading.IoPool.<>c__DisplayClass43_0`1.<<InvokeCore>b__0>d.MoveNext()

_{Opened automatically from Admin/_LogIncident/6a83b33b0e74b4ed. Recurrences are folded into this issue rather than opening new ones.}

Fixed: Systemorph/MeshWeaver.Plugins#1955. Confirming the byte and adding the two things the fix needed beyond removing it.

Located. Hosting/Deployment/Source/SelfUpdateRouting.cs, line 522, byte 30818:

Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(string.Join('<NUL>', parts))))

A literal 0x00 inside a char literal, where '\0' was meant — a deliberate NUL separator for a fingerprint join, written as the byte instead of the escape. Exactly one in the file.

The fix is the escape, which is the same character to the compiler and storable as text everywhere else. Behaviour is unchanged; the separator is still a NUL.

Why it could not be caught. Nothing in the pipeline can see this: the C# compiler treats it as an ordinary character, review renders it as nothing at all, and no test exercises storage of that node. It is invisible at every stage until PostgreSQL refuses the write — and by then the import has already recorded the fingerprint (#4459) and will skip forever.

So the fix ships with a guard: scripts/check-no-nul-in-mesh-sources.py refuses a raw 0x00 in any tracked file the installer maps to a node. Scoped to exactly that population — src/ files are module sources compiled into the bundle and never written to the mesh, so a NUL there is a style problem rather than an outage. One such case exists today (src/MeshWeaver.Blazor.Views/Components/Launcher.cs, raw NULs as key prefixes); it is named in the script's docstring rather than silently swept in, because it would become an outage the day that file moves into a node tree.

Measured: self-test of 4 cases wired to run before the gate; and reverting the one-character fix makes the real gate red, naming the file, line and byte. A repo-wide sweep found 106 tracked files containing 0x00 — all but two are legitimately binary (xlsx fixtures, an .icns, UTF-16 CSV encoding fixtures).

Scope of the unblock. This is why the control instance has no action surface, and therefore why the Roll that #1130, #4443, #3545 and #1823 are all waiting on could not be executed. Landing it does not by itself heal the mesh — #4459's re-import half still has to run, since the recorded fingerprint means the corrected file will otherwise be skipped.

_{Measured 2026-09-15 by Claude Opus 5.}

Still occurring: 1 more since the last update (3 total).

Last seen: 2026-09-15 22:12:18Z
Pods: memex-portal-deployment-6546c779d4-4ngqv, memex-portal-deployment-854cd8cd6d-r4cn5, memex-portal-deployment-6546c779d4-2dd5f

Recent log lines

2026-09-15 19:01:10Z memex-portal-deployment-6546c779d4-4ngqv fail: MeshWeaver.Mesh.CreateNodes[0]
      [CreateNodes] failed AFTER 3 node(s) were persisted — reporting the partial landing
      MeshWeaver.Hosting.PostgreSql.UnstorableContentException: Node 'Hosting/Deployment/Source/SelfUpdateRouting' cannot be persisted: its serialized content contains 1 NUL character(s) (U+0000), which PostgreSQL's jsonb type cannot represent — first occurrence in content property 'code'. JSON permits the character but jsonb stores DECODED text, and PostgreSQL text cannot hold a NUL byte, so the server rejects it with 22P05 (unsupported Unicode escape sequence). This is a property of the DATA, not of its size and not of the column: no widening makes it storable, so the NUL has to go at the source. A literal NUL typed into a source file is the usual cause — write a key separator as an explicit escape (U+001F UNIT SEPARATOR) instead.
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.BuildUpsertAsync(MeshNode node, JsonSerializerOptions options, Nullable`1 expectedVersion) in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 744
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct, Nullable`1 expectedVersion) in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 636
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.<>c__DisplayClass43_0.<<Write>b__0>d.MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 482
      --- End of stack trace from previous location ---
         at MeshWeaver.Mesh.Threading.IoPool.<>c__DisplayClass43_0`1.<<InvokeCore>b__0>d.MoveNext()
2026-09-15 20:31:54Z memex-portal-deployment-854cd8cd6d-r4cn5 fail: MeshWeaver.Mesh.CreateNodes[0]
      [CreateNodes] failed AFTER 2 node(s) were persisted — reporting the partial landing
      MeshWeaver.Hosting.PostgreSql.UnstorableContentException: Node 'Hosting/Deployment/Source/SelfUpdateRouting' cannot be persisted: its serialized content contains 1 NUL character(s) (U+0000), which PostgreSQL's jsonb type cannot represent — first occurrence in content property 'code'. JSON permits the character but jsonb stores DECODED text, and PostgreSQL text cannot hold a NUL byte, so the server rejects it with 22P05 (unsupported Unicode escape sequence). This is a property of the DATA, not of its size and not of the column: no widening makes it storable, so the NUL has to go at the source. A literal NUL typed into a source file is the usual cause — write a key separator as an explicit escape (U+001F UNIT SEPARATOR) instead.
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.BuildUpsertAsync(MeshNode node, JsonSerializerOptions options, Nullable`1 expectedVersion) in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 744
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct, Nullable`1 expectedVersion) in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 636
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.<>c__DisplayClass43_0.<<Write>b__0>d.MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 482
      --- End of stack trace from previous location ---
         at MeshWeaver.Mesh.Threading.IoPool.<>c__DisplayClass43_0`1.<<InvokeCore>b__0>d.MoveNext()
2026-09-15 22:12:18Z memex-portal-deployment-6546c779d4-2dd5f fail: MeshWeaver.Mesh.CreateNodes[0]
      [CreateNodes] failed AFTER 3 node(s) were persisted — reporting the partial landing
      MeshWeaver.Hosting.PostgreSql.UnstorableContentException: Node 'Hosting/Deployment/Source/SelfUpdateRouting' cannot be persisted: its serialized content contains 1 NUL character(s) (U+0000), which PostgreSQL's jsonb type cannot represent — first occurrence in content property 'code'. JSON permits the character but jsonb stores DECODED text, and PostgreSQL text cannot hold a NUL byte, so the server rejects it with 22P05 (unsupported Unicode escape sequence). This is a property of the DATA, not of its size and not of the column: no widening makes it storable, so the NUL has to go at the source. A literal NUL typed into a source file is the usual cause — write a key separator as an explicit escape (U+001F UNIT SEPARATOR) instead.
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.BuildUpsertAsync(MeshNode node, JsonSerializerOptions options, Nullable`1 expectedVersion) in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 744
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct, Nullable`1 expectedVersion) in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 636
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.<>c__DisplayClass43_0.<<Write>b__0>d.MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 482
      --- End of stack trace from previous location ---
         at MeshWeaver.Mesh.Threading.IoPool.<>c__DisplayClass43_0`1.<<InvokeCore>b__0>d.MoveNext()

Still live after the 8710 roll, and it is currently costing the portal its own operations API

Measured 2026-09-16 04:4x–05:3xZ on the control instance (memex.systemorph.com), read-only.

Ops/Status/memex (written by the 00:42Z Roll to 3.0.0-ci.8710, finished 00:51:12Z, converged: true, 2/2 ready) reports health: Degraded on BOTH replicas, and the reason is a content-type census, not a bake failure:

memex-portal-deployment-7cb6684584-jdw7v  Degraded content-types: 2 node type(s) whose content this
  replica cannot type … Hosting/InstanceAction ×2 (last Deployments/memex-roll-8710-agent-collab-20260916);
  Hosting/ModuleInventory ×1 (last Ops/Modules/memex)

memex-portal-deployment-7cb6684584-ztqz8  Degraded content-types: … Hosting/Deployment ×2
  (last Deployments/memex); Hosting/LogEntry ×2 (last Ops/Logs/memex-…-jdw7v)

Four Hosting types, split across the two replicas, and bake-report is Healthy on both (total=218 baked=217). So this is the module's assembly not being loaded on the replica, which is the shape this issue describes.

The consequence, measured: two READ-ONLY Logs Hosting/InstanceAction nodes I filed for MeshWeaver#1174 —

Deployments/memexcloud-logs-20260916-nodeops-1174 (created 04:49:32Z)
Deployments/memexcloud-logs-20260916-nodeops-b (created 04:51:50Z)

— are still at version 1 with log: [] and no state stamp 40+ minutes later. Neither was ever picked up. A Roll created at 00:42Z, on the previous image, ran end to end. So the control instance's operations API is presently unable to answer a log query, which makes every production read that AGENTS.md routes through Logs/Sample unavailable — and cluster access is break-glass, so there is no sanctioned substitute.

This is the concrete cost of the NUL byte beyond the failed CreateNodes batch: an absent Hosting source node ⇒ types that cannot be typed on a replica ⇒ InstanceAction content reads as empty ⇒ the watcher never fires.

_{Reads: get @Ops/Status/memex, get @Deployments/*, get on the two action nodes. Nothing mutated except the two read-only action requests themselves.}

🤖 Generated with Claude Code

Fixed and verified on the live instance. Closing — with the second finding on this thread redirected, not buried.

The titled defect

Hosting/Deployment/Source/SelfUpdateRouting.cs:522 carried a literal 0x00 inside a char literal (string.Join('<NUL>', parts) where '\0' was meant). Fixed in Systemorph/MeshWeaver.Plugins#1955, merged 2026-09-15 21:34:57Z.

Verified on memex.systemorph.com, not inferred from the merge:

Hosting/Deployment/Source/SelfUpdateRouting — Active, version 1, 0 NUL bytes, carrying string.Join('\0', parts). Imported 2026-09-15 23:46:54Z.
Hosting/_GitSync — Imported, now at 061976bc.
search 'nodeType:NodeType content.compilationStatus:Error partitions:all' → 0, against nodeType:NodeType partitions:all → 147, coverage.partitions → 98. All five formerly-parked types (Hosting/{Backup, Deployment, InstanceAction, InstanceRequest, PlatformBuildInbox}) are out of the Error set.

The class is closed too: scripts/check-no-nul-in-mesh-sources.py refuses a raw 0x00 in any tracked file the installer maps to a node, self-test runs before the gate, and reverting the one-character fix makes the real gate red naming file, line and byte. A repo sweep found 106 tracked files containing 0x00 — all but two legitimately binary; the other is under src/ (a module source, never written to the mesh) and is named in the script's docstring rather than silently swept in.

The downstream half — a partial import recording the content fingerprint anyway, which is what made one unstorable file permanent — was #4459, fixed in core #4467 (merged).

🚨 The other finding on this thread is a DIFFERENT defect and is NOT fixed

A session recorded here that Hosting/InstanceAction is untypable on a control-instance replica (3.0.0-ci.8710), so Logs/Sample do not execute. I re-measured /health on memex.systemorph.com across 8 samples (repeated calls sample different replicas) — it reports Degraded, union:

Crm/Stage · Hosting/BackupStore · Hosting/ConfigAudit · Hosting/LogEntry
Hosting/ModuleInventory · Store/Catalog · Store/Install · Store/Maintenance · Store/Tier

with the payload's own reason: "the module that declares the type is not loaded here (its prebuilt bundle was declined or its compiled assembly is not on this replica)."

That is not a NUL byte and not a compile failure — the shared compilationStatus field is clean for every one of those types. It is a module bundle not loaded on a replica, i.e. the shape #3583 already tracks ("bundles are baked once per framework identity"). Operationally it matters because Hosting/LogEntry appears in every sample and a Logs action lands its window as LogEntry nodes — so the action runs and still yields an empty log, which is exactly what was observed.

I am carrying that measurement to #3583 rather than holding this issue open for it, because keeping a fixed-and-verified defect open for an unrelated one is how a fingerprint becomes a bucket nobody can ever close.

_{Verified 2026-09-16 by Claude Opus 5.}

fix(ci): the NUL guard caught this change writing literal NULs, and t…

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-09-16)

Verified healed on the control instance — closing

The one-character fix landed as b884699c5 (MeshWeaver.Plugins#1955). What was still owed was the
re-import, since #4459's recorded fingerprint meant the corrected file would otherwise be skipped.
It has run.

Measured 2026-09-16 07:0xZ on memex.systemorph.com, read-only:

Hosting/_GitSync — lastSyncOutcome: Imported, lastSyncCommitSha: 061976bc88, at
04:31:29Z. That commit contains b884699c5 (git merge-base --is-ancestor), and the file at it
carries zero NUL bytes.
Hosting/Deployment/Source/SelfUpdateRouting is present again in the Hosting/Deployment/Source
listing — it is the node the import used to drop.
search 'namespace:Hosting scope:descendants nodeType:NodeType content.compilationStatus:Error'
→ 0, against a denominator of 19 NodeTypes over 1 readable partition (hosting), both
from the same instrument. The five that were parked on the missing symbol — Hosting/Backup,
Hosting/Deployment, Hosting/InstanceAction, Hosting/InstanceRequest,
Hosting/PlatformBuildInbox — are all in that 19 and none is at Error.
The control instance's action surface works again, which is the consequence this issue was
costing: four Hosting/InstanceAction Logs nodes filed this morning (07:57–08:00Z under
Ops/Actions/logs-client-b-20260916-*) each ran to state: Done and landed their entries. The two
from 04:49/04:51Z that never moved were created before the import.
No recurrence since 2026-09-15 22:12:18Z, which precedes the import.

Not covered by this closure — the framework gap the original report named as secondary: the NUL
was rejected mid-persist, after three nodes of the batch had already landed, so CreateNodes
reported a partial landing rather than failing atomically. Validating unstorable content at
Create/Patch time (or validating the whole batch before any node lands) is a separate change with a
real design question in it — whether "unstorable" is a Postgres property the contract should know
about at all — and it is deliberately NOT filed here rather than smuggled into a data fix.

_{Measured 2026-09-16 by Claude Opus 5.}

Navigation Menu

Repository navigation

SelfUpdateRouting source node contains a literal NUL byte, making it unstorable in PostgreSQL and aborting CreateNodes batches mid-persist #4458

Evidence

Still live after the 8710 roll, and it is currently costing the portal its own operations API

The titled defect

🚨 The other finding on this thread is a DIFFERENT defect and is NOT fixed

Verified healed on the control instance — closing

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

SelfUpdateRouting source node contains a literal NUL byte, making it unstorable in PostgreSQL and aborting CreateNodes batches mid-persist #4458

Description

Evidence

Activity

rbuergi commented on Sep 15, 2026

systemorph commented on Sep 15, 2026

rbuergi commented on Sep 16, 2026

Still live after the 8710 roll, and it is currently costing the portal its own operations API

rbuergi commented on Sep 16, 2026

The titled defect

🚨 The other finding on this thread is a DIFFERENT defect and is NOT fixed

rbuergi commented on Oct 11, 2026

Verified healed on the control instance — closing

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions