Repository navigation
PostgreSQL rejects node creation when serialized content contains invalid Unicode escape sequences #1449
Description
What is failing
Node creation in MeshWeaver.Mesh.CreateNode fails with a PostgreSQL 22P05: unsupported Unicode escape sequence exception when the PostgreSqlStorageAdapter.WriteAsyncCore serializes a node whose content or path produces an invalid \uXXXX sequence in JSON.
Probable cause
The failure occurred while persisting a node at path MeshWeaver/src/MeshWeaver.Markdown.Export/Html/DocumentAreaResolution. The JSON produced by System.Text.Json serialization (via JsonSerializerOptions) likely contains a string with a character that is being escaped as an invalid Unicode escape (e.g., a high surrogate without its pair, or a malformed escape from the node's content). PostgreSQL's standard_conforming_strings setting rejects \u sequences that don't represent valid Unicode code points. Confidence: Moderate — the stack trace and error code are clear, but the exact offending content within the node's data is not visible.
Impact
Low — a single occurrence on one pod (memex-portal-deployment-599884944c-29dkk) at one point in time. This prevented creation of a single node under the Markdown Export/Html/DocumentAreaResolution subtree. No cascading failure observed.
Where to look
PostgreSqlStorageAdapter.WriteAsyncCoreat line 631 inMeshWeaver.Hosting.PostgreSql— where the Npgsql command is executed.- Investigate what data the
DocumentAreaResolutionnode carried that would serialize to an invalid Unicode escape. Check the node's JSON content before it hits theNpgsqlCommand.ExecuteNonQuerycall. - Verify whether the issue is reproducible by creating a node with a similar path or content containing surrogates or non-BMP characters.
Evidence
| Fingerprint | 3d329d5ef2237493 |
| Category | MeshWeaver.Mesh.CreateNode |
| Severity | Error |
| Top frame | MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct, Nullable1 expectedVersion)` |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-599884944c-29dkk |
| Occurrences | 1 |
| First seen | 2026-08-13 16:50:13Z |
| Last seen | 2026-08-13 16:50:13Z |
Recent log lines
2026-08-13 16:50:13Z memex-portal-deployment-599884944c-29dkk fail: MeshWeaver.Mesh.CreateNode[0]
Unexpected error during node creation at MeshWeaver/src/MeshWeaver.Markdown.Export/Html/DocumentAreaResolution
Npgsql.PostgresException (0x80004005): 22P05: unsupported Unicode escape sequence
DETAIL: Detail redacted as it may contain sensitive data. Specify 'Include Error Detail' in the connection string to include this information.
at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
at Npgsql.NpgsqlCommand.ExecuteNonQuery(Boolean async, CancellationToken cancellationToken)
at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct, Nullable`1 expectedVersion) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 631
at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode n…[truncated]
Opened automatically from Admin/_LogIncident/3d329d5ef2237493. Recurrences are folded into this issue rather than opening new ones.
Activity
Merge pull request #1460 from Systemorph/fix/1449-nul-content-unstorable
systemorph-com commentedon Sep 17, 2026
Reopened — this fault is happening again.
Still occurring: 293 more since the last update (294 total).
memex-portal-deployment-599884944c-29dkk,memex-portal-deployment-c46c94d87-fqfrm,memex-portal-deployment-5fc766ff6c-48djv,memex-portal-deployment-55d8df74f6-jm8ff,memex-portal-deployment-68bdfb97d6-h5sxf,memex-portal-deployment-69f7bbd6f8-fvd46,memex-portal-deployment-6db74b7759-8qdgc,memex-portal-deployment-5bbcf746b8-2q6gb,memex-portal-deployment-6cc847bd4d-lq2rh,memex-portal-deployment-fbbc4c684-jsg9j,memex-portal-deployment-797df55dfb-cn4p8,memex-portal-deployment-c6f9b9c4-lj85m,memex-portal-deployment-c6f9b9c4-gphss,memex-portal-deployment-59ccff47c5-rjs77,memex-portal-deployment-7db656d9d9-nbfzk,memex-portal-deployment-565b67998c-zzjl4,memex-portal-deployment-565b67998c-rbsd7,memex-portal-deployment-df7ddfdf4-dtp44,memex-portal-deployment-df7ddfdf4-xb9bk,memex-portal-deployment-569544576b-q6nvd,memex-portal-deployment-569544576b-89r5k,memex-portal-deployment-569544576b-9h52b,memex-portal-deployment-858f57dc6-ff5kn,memex-portal-deployment-558d86bb48-vl44h,memex-portal-deployment-58b5fbf775-fjbvn,memex-portal-deployment-5678bbc9c8-4rvjb,memex-portal-deployment-6c74bf4c66-5xvlt,memex-portal-deployment-767d8d6684-vrc78,memex-portal-deployment-76c67b6b45-v4hdf,memex-portal-deployment-66cfbf759-svxh9,memex-portal-deployment-9bf79b774-jwrng,memex-portal-deployment-c7f5ff965-vk9zq,memex-portal-deployment-58c88885f8-h5hfs,memex-portal-deployment-58c88885f8-lhgp7,memex-portal-deployment-77cfc55bfc-txpk7,memex-portal-deployment-84d46fdb95-sd7p4,memex-portal-deployment-584db8fc55-vtslpRecent log lines
rbuergi commentedon Sep 18, 2026
Closing — the defect this issue reports (
22P05: unsupported Unicode escape sequence) is fixed inmain, and the reopen is the fingerprint bucket firing on unrelated faults.Fixed in current
origin/main. The root cause was a literal U+0000 (NUL) typed as a composite-key separator inMeshWeaver.Markdown.Export/Html/DocumentAreaResolution.cs— jsonb stores decoded text and cannot hold a NUL, so the write died with22P05and a DETAIL the connection policy redacts, naming neither the file nor the character. Closed by PR #1460 (fix/1449-nul-content-unstorable). In a read-only checkout at/Users/roland/code/MW-current-main:→
test/MeshWeaver.Documentation.Test/NoLiteralNulInSourceGuard.cs— a standing governance guard inmainthat scans the working tree undersrc,test,samples,content,memex,clientsand fails on any literal NUL, and whose own doc comment names this issue:So the offending byte is gone and re-introducing one is a red test, not a silent production write failure.
The reopen evidence is not this fault. The incident
Admin/_LogIncident/3d329d5ef2237493normalizes to"Unexpected error during node creation at {path}"— the path is templated away, so everyMeshWeaver.Mesh.CreateNodeerror folds into this one bucket. Read read-only on memex.systemorph.com (get @Admin/_LogIncident/3d329d5ef2237493), all 10 stored samples (2026-09-02 → 2026-09-12) are two entirely different faults:[CreateNode] STORE UNREACHABLE at …: NpgsqlException: The operation has timed out/Exception while reading from stream— a data-store availability fault, explicitly classified by the adapter as "the fault is the data store, not this create"; andUnexpected error during node creation at WhatsNew/_Activity/…: PostgresException: 42P01: relation "whatsnew.activities" does not exist— an unprovisionedwhatsnewactivities partition table.Not one of the 294 counted occurrences in the sample window carries
22P05. The42P01is a real live fault and deserves its own issue against partition provisioning; it is not what this issue describes, and tracking it here would mean neither gets fixed.Closing a dormant incident is safe by design: the incident record stays live, and the recurrence bot reopens this issue automatically with fresh
Last seenevidence the moment the fingerprint fires again. Nothing is lost by closing; what is lost by leaving it open is the meaning of "open".Read-only triage, 2026-09-18. Nothing was run, re-run or changed on any deployment.
systemorph-com commentedon Sep 18, 2026
Reopened — this fault is happening again.
Still occurring: 1 more since the last update (295 total).
memex-portal-deployment-599884944c-29dkk,memex-portal-deployment-c46c94d87-fqfrm,memex-portal-deployment-5fc766ff6c-48djv,memex-portal-deployment-55d8df74f6-jm8ff,memex-portal-deployment-68bdfb97d6-h5sxf,memex-portal-deployment-69f7bbd6f8-fvd46,memex-portal-deployment-6db74b7759-8qdgc,memex-portal-deployment-5bbcf746b8-2q6gb,memex-portal-deployment-6cc847bd4d-lq2rh,memex-portal-deployment-fbbc4c684-jsg9j,memex-portal-deployment-797df55dfb-cn4p8,memex-portal-deployment-c6f9b9c4-lj85m,memex-portal-deployment-c6f9b9c4-gphss,memex-portal-deployment-59ccff47c5-rjs77,memex-portal-deployment-7db656d9d9-nbfzk,memex-portal-deployment-565b67998c-zzjl4,memex-portal-deployment-565b67998c-rbsd7,memex-portal-deployment-df7ddfdf4-dtp44,memex-portal-deployment-df7ddfdf4-xb9bk,memex-portal-deployment-569544576b-q6nvd,memex-portal-deployment-569544576b-89r5k,memex-portal-deployment-569544576b-9h52b,memex-portal-deployment-858f57dc6-ff5kn,memex-portal-deployment-558d86bb48-vl44h,memex-portal-deployment-58b5fbf775-fjbvn,memex-portal-deployment-5678bbc9c8-4rvjb,memex-portal-deployment-6c74bf4c66-5xvlt,memex-portal-deployment-767d8d6684-vrc78,memex-portal-deployment-76c67b6b45-v4hdf,memex-portal-deployment-66cfbf759-svxh9,memex-portal-deployment-9bf79b774-jwrng,memex-portal-deployment-c7f5ff965-vk9zq,memex-portal-deployment-58c88885f8-h5hfs,memex-portal-deployment-58c88885f8-lhgp7,memex-portal-deployment-77cfc55bfc-txpk7,memex-portal-deployment-84d46fdb95-sd7p4,memex-portal-deployment-584db8fc55-vtslp,memex-portal-deployment-6b879996c7-lk6fdRecent log lines
rbuergi commentedon Sep 19, 2026
Cross-reference: the close/reopen cycle on this issue is explained on #1540 — #1540 (comment)
Short version: the reopen was correct. Its newest occurrence postdates the close, which is what
Reopenhas required since #4577. What went wrong is the close: these are teardown faults, and the dormancy they were closed on was measured across a period with no rolls. All five of #1422, #1449, #1540, #1547 and #1548 fired inside the 05:05–05:36Z window on 2026-09-19 — the roll storm. Fifteen quiet days meant fifteen days without a roll, not fifteen days of health.Closing again on quiet between rolls will reopen again on the next roll. What would settle it is a roll on an image provably carrying the fix, producing no occurrence.
docs: rest the section on the three teardown faults, not five issues
rbuergi commentedon Sep 19, 2026
🚨 This fingerprint is collecting four unrelated faults, and the one it is titled for has not fired since 2026-08-13
Two corrections, both from reading
Admin/_LogIncident/3d329d5ef2237493directly.1. The 18:39Z comment characterises this as a teardown fault. It is not.
That comment says: "these are teardown faults… All five of #1422, #1449, #1540, #1547 and #1548 fired inside the 05:05–05:36Z window on 2026-09-19 — the roll storm."
This issue's
lastSeenis2026-09-18T15:35:29.421Z. It did not fire on 2026-09-19 at all, and no sample it retains is a teardown fault. This is the same over-claim that was caught and corrected on #4911, where the doc section was narrowed to rest on #1540/#1547/#1548 only — the three that really are teardown faults — precisely because #1449 sits at 09-18 15:35:29Z. It has now been restated here; it should not be relied on.2. The identity is too weak to be one issue
The normalized message is
"Unexpected error during node creation at {path}"— the path is masked, and the node carries noexceptionTypefield at all (compare32768bfc4b1eeb3f, which carries"exceptionType": "System.OutOfMemoryException"). So the fingerprint is effectively "CreateNode failed somewhere", and its 295 occurrences are at least four different defects:22P05: unsupported Unicode escape sequence← the title of this issue[CreateNode] STORE UNREACHABLE … NpgsqlException: The operation has timed out— connect timeoutsNpgsqlException: Exception while reading from streamlastSeen)42P01: relation "whatsnew.activities" does not existThose are a serialization bug, a data-store outage, a read timeout, and a missing table. Nothing links them except the log site.
What follows
42P01is real, live, and now filed on its own: WhatsNew namespace was never provisioned: every WhatsNew/_Activity write fails 42P01 (whatsnew.activities does not exist) — and satellite tables are never reconciled when the mapping set grows #4933. TheWhatsNewnamespace has noPartitionDefinitionat all on memex-cloud (get @Admin/Partition/WhatsNew→Not found, against a working negative control:@Admin/Partition/Crmreturns in full with its complete 12-entry satellite map). Satellite tables are created only at provisioning time from that map, sowhatsnew.activitieswas never created and every WhatsNew activity write fails. WhatsNew namespace was never provisioned: every WhatsNew/_Activity write fails 42P01 (whatsnew.activities does not exist) — and satellite tables are never reconciled when the mapping set grows #4933 also names the wider exposure — nothing reconciles an already-provisioned partition whenSatelliteTableMapping.Defaultsgrows.22P05this issue is titled for has not been seen since 2026-08-13 — 37 days, one occurrence ever. But I will not call that fixed: dormancy is only evidence when the trigger recurred, and a malformed-escape payload is by nature rare input, not something a roll exercises.STORE UNREACHABLEburst is a data-store availability event, already answeredUnavailableto callers by design (the message says so), and is not a code defect inCreateNode.Recommendation: re-scope this issue to the
22P05serialization fault it was filed for and stop reading itslastSeen/occurrence count as evidence about that fault — those numbers belong to three other defects. The fingerprint needsexceptionTypein its identity, which is the same class of problem as Systemorph/MeshWeaver.Plugins#2164 (an identity that folds things it should not) rather than its mirror.Nothing closed. No labels, state or assignees changed.
Read 2026-09-19 by Claude Opus 5 — incident node and live partition reads, both quoted above.
rbuergi commentedon Sep 22, 2026
The guard that holds this closed could pass having checked nothing — hardened in PR #5194
Core's line of defence for the titled
22P05isNoLiteralNulInSourceGuard(the adapter that dies on it now lives in MeshWeaver.Plugins — every trace above readsplugins-repo/src/MeshWeaver.Hosting.PostgreSql/…). Its predicate, "no scanned text file contains a NUL", was satisfiable by an empty universe, measured:content/, does not exist in this repository and was skipped silently viaWhere(Directory.Exists)since the guard was written;IndexOf(NUL)leaves the guard green over the whole tree.Files examined today:
src=3319 test=1374 samples=676 memex=119 clients=51; never scanned:.github=146 tools=49 scripts=12 deploy=216hand-authored text files (all clean, so widening reds nothing).PR #5194: an absent root is named, not skipped (
contentdropped;.github,tools,scripts,deployadded); per-root counts are asserted non-zero and printed with the verdict; and a planted-NUL test must report exactlysub/node.json:3over two examined files beside a NUL in a.pngand one underbin/. Controls: detector neutered → the planted-NUL test fails (Assert.Single() Failure: The collection was empty) while the main guard stays green;contentre-declared → the main guard fails namingcontent.Not changed, and named so nobody reads this as closure of the whole thread: a NUL arriving from runtime content still dies with a redacted
22P05— that diagnosis belongs to the adapter in Plugins; and the incident fingerprint still folds four unrelated faults into this bucket (the live42P01half is #4933), which is why the recurrence bot reopens this on faults that are not this fault. Re-scoping the issue to the22P05it was filed for, as the 19:39Z comment recommends, is the only thing that stops the churn.rbuergi commentedon Sep 23, 2026
Closing — the
22P05this issue is titled for is fixed at the root and deployed, and the one class the title names that had not been measured (lone surrogates) cannot reach PostgreSQL at allThe thread asked for this issue to be scoped to the
22P05it was filed for. On that scope, every half is now settled with evidence:1. The production occurrence — fixed at the source. The one
22P05ever recorded (2026-08-13 16:50:13Z,MeshWeaver/src/MeshWeaver.Markdown.Export/Html/DocumentAreaResolution) was a literal U+0000 typed as a key separator. PR #1460 replaced it. The file now lives in MeshWeaver.Plugins and readsprivate const char KeySeparator = '\u001F';(src/MeshWeaver.Markdown.Export/Html/DocumentAreaResolution.cs:151at Pluginse9d9283abe), andgit grep -P "\x00"over that project finds nothing.2. Reintroduction — guarded in both repos. Core:
NoLiteralNulInSourceGuard, hardened by #5194 (merge3ad2db1496), is an ancestor of both deployed commits — memex.meshweaver.cloud39f49494eband memex.systemorph.com3da3423c4b(git merge-base --is-ancestorexit 0 for each; both read from/api/versiontoday). Plugins:scripts/check-no-nul-in-mesh-sources.pyruns inci.yml.3. A NUL arriving from runtime content — named, not redacted. The 2026-09-22 comment said such a NUL "still dies with a redacted
22P05". On current Pluginsmainit does not:PostgreSqlStorageAdapterchecks the serialized content before the round-trip (lines 819–820) and throwsUnstorableContentException, which carries the node path and the exact content property. It covers the single-write and batch paths (UnstorableContentTests:WriteRefusesNulContentAndNamesTheNodeAndProperty,NulNestedInAnArrayIsLocatedByIndexedPath,BatchWriteNamesThePoisonedNodeAndCommitsNothing, plus a storable-replacement and a clean-content control). The type came in with the storage move on 2026-08-27 (69a37b7ec), so every current image carries it.4. Lone surrogates, the other case the title names — measured today and they cannot produce
22P05. I ranSystem.Text.Jsonon .NET 10:So a lone surrogate either becomes U+FFFD at serialization (storable) or throws a
JsonExceptionin-process, before any SQL. The only character that reaches jsonb and gets refused there is U+0000, and (3) handles that.Why this may be reopened anyway, and why that won't be this fault.
Admin/_LogIncident/3d329d5ef2237493keys on"Unexpected error during node creation at {path}"and has no exception type, so everyCreateNodeerror lands in it. None of its occurrences since 2026-08-13 is a22P05(the thread lists them:STORE UNREACHABLEtimeouts, a stream read fault, and42P01 whatsnew.activities, which is #4933 and now closed). A recurrence reopen will need its sample checked for22P05before anyone reads it as this defect.What I did not establish: I did not read the incident node myself; the control instance (memex.systemorph.com MCP) answers 503. I did not run
UnstorableContentTestsagainst a live Postgres; I'm citing them from source.systemorph-com commentedon Sep 23, 2026
Reopened — this fault is happening again.
Still occurring: 1 more since the last update (296 total).
memex-portal-deployment-5dd79dc746-45vzd,memex-portal-deployment-599884944c-29dkk,memex-portal-deployment-c46c94d87-fqfrm,memex-portal-deployment-5fc766ff6c-48djv,memex-portal-deployment-55d8df74f6-jm8ff,memex-portal-deployment-68bdfb97d6-h5sxf,memex-portal-deployment-69f7bbd6f8-fvd46,memex-portal-deployment-6db74b7759-8qdgc,memex-portal-deployment-5bbcf746b8-2q6gb,memex-portal-deployment-6cc847bd4d-lq2rh,memex-portal-deployment-fbbc4c684-jsg9j,memex-portal-deployment-797df55dfb-cn4p8,memex-portal-deployment-c6f9b9c4-lj85m,memex-portal-deployment-c6f9b9c4-gphss,memex-portal-deployment-59ccff47c5-rjs77,memex-portal-deployment-7db656d9d9-nbfzk,memex-portal-deployment-565b67998c-zzjl4,memex-portal-deployment-565b67998c-rbsd7,memex-portal-deployment-df7ddfdf4-dtp44,memex-portal-deployment-df7ddfdf4-xb9bk,memex-portal-deployment-569544576b-q6nvd,memex-portal-deployment-569544576b-89r5k,memex-portal-deployment-569544576b-9h52b,memex-portal-deployment-858f57dc6-ff5kn,memex-portal-deployment-558d86bb48-vl44h,memex-portal-deployment-58b5fbf775-fjbvn,memex-portal-deployment-5678bbc9c8-4rvjb,memex-portal-deployment-6c74bf4c66-5xvlt,memex-portal-deployment-767d8d6684-vrc78,memex-portal-deployment-76c67b6b45-v4hdf,memex-portal-deployment-66cfbf759-svxh9,memex-portal-deployment-9bf79b774-jwrng,memex-portal-deployment-c7f5ff965-vk9zq,memex-portal-deployment-58c88885f8-h5hfs,memex-portal-deployment-58c88885f8-lhgp7,memex-portal-deployment-77cfc55bfc-txpk7,memex-portal-deployment-84d46fdb95-sd7p4,memex-portal-deployment-584db8fc55-vtslp,memex-portal-deployment-6b879996c7-lk6fdRecent log lines
Re-addressed by the current identity function: this incident inherited
3d329d5ef2237493(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Sep 23, 2026
The 12:08:41Z reopen is not this fault. The new sample reads:
Unexpected error during node creation at Hosting/ATestFakeIsScopedToItsSubject/_Access/Anonymous_Access: PostgresException: 40P01: deadlock detected(pod5dd79dc746-45vzd,PostgreSqlStorageAdapter.WriteAsyncCore)That is a deadlock on an
_Accesswrite. The fault this issue is titled for is22P05(a Unicode escape in content). As the 09-19 and 09-23 comments already explain, the fingerprint3d329d5ef2237493keys on"Unexpected error during node creation at {path}"with no exception type, so it folds everyCreateNodefailure into this issue.The
22P05itself stays fixed, for the reasons in the 04:35Z comment:UnstorableContentException;Closing again as completed on that scope.
Open, and not filed by me: the
40P01on…/_Access/Anonymous_Access. The path looks like an in-mesh test (ATestFakeIsScopedToItsSubject) creating an access assignment on memex. I did not investigate it. The control instance (memex.systemorph.com) was unreachable from this session, so I could not file it into triage. It needs its own ticket with the deadlock DETAIL, which is redacted here (Include Error Detailis off). The same incident-identity gap (noexceptionTypein the fingerprint) will reopen this issue again on the next unrelatedCreateNodefailure.rbuergi commentedon Sep 24, 2026
The
40P01onHosting/ATestFakeIsScopedToItsSubject/_Access/Anonymous_Accessthat caused the 12:08:41Z reopen now has its own ticket: Systemorph/MeshWeaver.Plugins#2345. It is reproduced deterministically.public.ensure_partition_schemare-runs on a live partition and locksuser_effective_permissionsbeforeaccess, while a grant write locksaccessand thenuser_effective_permissions. The fix is in Systemorph/MeshWeaver.Plugins#2346 (lock order, no retry). The incident fingerprint gap is unchanged: everyCreateNodefailure still folds into this issue.