PostgreSQL schema 'user.mesh_nodes' missing — node creation fails on all pods

What is failing

Node creation (via PostgreSqlStorageAdapter.WriteAsyncCore) fails with a Npgsql.PostgresException for every attempt to persist a node under the User/ namespace. The operation hits the database and PostgreSQL returns error 42P01: relation "user.mesh_nodes" does not exist.

All 27 occurrences target the same path — User/_Memex/AiSettings — suggesting this fires when a user's AI settings node is first created or saved.

Probable cause — high confidence

The PostgreSQL storage adapter (MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter) writes nodes to schema-qualified tables. It expects the schema user with a table mesh_nodes to exist, but that relation is absent from the database. The most likely explanations:

EF Core / SQL migration was never run for the user schema — the adapter auto-creates schemas on write, or the migration script for the user partition was missed during deployment.
A schema name mismatch — the adapter is configured to look for user.mesh_nodes but migrations created the table under a different schema (e.g. public.mesh_nodes or a differently-cased schema name).

The error is deterministic (every retry on every pod hits it) and affects a single schema — other node operations may succeed for non-User/ paths.

Impact

Moderate. 27 failures over ~10.5 hours on all 3 memex-cloud pods. Any attempt to create or update a node in the User/_Memex/AiSettings path is silently lost (the exception is caught and logged, but the node never persists). Users trying to save or update their AI settings will see no effect — settings changes appear to succeed in the UI but are discarded on the server. Since the affected path is specifically User/_Memex/AiSettings, this likely prevents users from configuring AI/agent behaviour.

Where to look

MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore at line 595 — the call site that fails.
The EF Core migration / SQL initialisation script for the PostgreSQL adapter — verify that the user schema and user.mesh_nodes table are created.
The adapter's schema resolution logic — check whether schema names are lowercased by Npgsql vs. how they were created in the database (PostgreSQL folds unquoted identifiers to lowercase; quoted identifiers are case-sensitive).

Existing reports

No existing incident or issue found for this fingerprint.

Evidence


Fingerprint	`4a05ce8a3e7af5ed`
Category	`MeshWeaver.Mesh.CreateNode`
Severity	Error
Top frame	`MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct)`
Namespace	`memex-cloud`
Pods	`memex-portal-deployment-6956d54bb5-l6mxv`, `memex-portal-deployment-8476467cb6-nkbhh`, `memex-portal-deployment-5bbfc97c64-lq7lw`
Occurrences	27
First seen	2026-08-10 01:09:41Z
Last seen	2026-08-10 11:44:37Z

Recent log lines

2026-08-10 07:09:47Z memex-portal-deployment-8476467cb6-nkbhh fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at User/_Memex/AiSettings
      Npgsql.PostgresException (0x80004005): 42P01: relation "user.mesh_nodes" does not exist
      
      POSITION: 13
         at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteNonQuery(Boolean async, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 595
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 595
         at MeshWeaver.Host…[truncated]
2026-08-10 07:09:47Z memex-portal-deployment-8476467cb6-nkbhh fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at User/_Memex/AiSettings
      Npgsql.PostgresException (0x80004005): 42P01: relation "user.mesh_nodes" does not exist
      
      POSITION: 13
         at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteNonQuery(Boolean async, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 595
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 595
         at MeshWeaver.Host…[truncated]
2026-08-10 07:09:47Z memex-portal-deployment-8476467cb6-nkbhh fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at User/_Memex/AiSettings
      Npgsql.PostgresException (0x80004005): 42P01: relation "user.mesh_nodes" does not exist
      
      POSITION: 13
         at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteNonQuery(Boolean async, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 595
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 595
         at MeshWeaver.Host…[truncated]
2026-08-10 07:09:47Z memex-portal-deployment-8476467cb6-nkbhh fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at User/_Memex/AiSettings
      Npgsql.PostgresException (0x80004005): 42P01: relation "user.mesh_nodes" does not exist
      
      POSITION: 13
         at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteNonQuery(Boolean async, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 595
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 595
         at MeshWeaver.Host…[truncated]
2026-08-10 07:09:47Z memex-portal-deployment-8476467cb6-nkbhh fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at User/_Memex/AiSettings
      Npgsql.PostgresException (0x80004005): 42P01: relation "user.mesh_nodes" does not exist
      
      POSITION: 13
         at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteNonQuery(Boolean async, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 595
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 595
         at MeshWeaver.Host…[truncated]
2026-08-10 11:44:37Z memex-portal-deployment-5bbfc97c64-lq7lw fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at User/_Memex/AiSettings
      Npgsql.PostgresException (0x80004005): 42P01: relation "user.mesh_nodes" does not exist
      
      POSITION: 13
         at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteNonQuery(Boolean async, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 595
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 595
         at MeshWeaver.Host…[truncated]
2026-08-10 11:44:37Z memex-portal-deployment-5bbfc97c64-lq7lw fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at User/_Memex/AiSettings
      Npgsql.PostgresException (0x80004005): 42P01: relation "user.mesh_nodes" does not exist
      
      POSITION: 13
         at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteNonQuery(Boolean async, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 595
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 595
         at MeshWeaver.Host…[truncated]
2026-08-10 11:44:37Z memex-portal-deployment-5bbfc97c64-lq7lw fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at User/_Memex/AiSettings
      Npgsql.PostgresException (0x80004005): 42P01: relation "user.mesh_nodes" does not exist
      
      POSITION: 13
         at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteNonQuery(Boolean async, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 595
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 595
         at MeshWeaver.Host…[truncated]
2026-08-10 11:44:37Z memex-portal-deployment-5bbfc97c64-lq7lw fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at User/_Memex/AiSettings
      Npgsql.PostgresException (0x80004005): 42P01: relation "user.mesh_nodes" does not exist
      
      POSITION: 13
         at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteNonQuery(Boolean async, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 595
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 595
         at MeshWeaver.Host…[truncated]
2026-08-10 11:44:37Z memex-portal-deployment-5bbfc97c64-lq7lw fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at User/_Memex/AiSettings
      Npgsql.PostgresException (0x80004005): 42P01: relation "user.mesh_nodes" does not exist
      
      POSITION: 13
         at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteNonQuery(Boolean async, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 595
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 595
         at MeshWeaver.Host…[truncated]

_{Opened automatically from Admin/_LogIncident/4a05ce8a3e7af5ed. Recurrences are folded into this issue rather than opening new ones.}

Root-caused — the issue's "probable cause" (missing migration / schema-case mismatch) is not it. The path was wrong, not the schema.

User is the self-typed NodeType declaration node (UserNodeType.CreateMeshNode() sets NodeType = "User" on the node at path User), so AiSourcesInstallHook's pathless nodeType:User user enumeration returned the declaration alongside real accounts and reconciled its Id ("User") as an account — writing User/_Memex/AiSettings into a partition no instance provisions. Once per installed package, per pod boot repair pass: 27 = 3 pod boots × ~9 installed packages, bursts within the same second per pod.

Nothing persisted, nothing lost: path:User/_Memex/AiSettings returns 0 nodes on the live portal; real users' settings (e.g. rbuergi/_Memex/AiSettings) were reconciled correctly by the same pass, lastModifiedBy: system-security today. The failed writes carried only machine-generated package-source rows for a nonexistent account.
Fix: the enumeration drops the declaration id — landed on main in Installed plugins stayed invisible on any instance set up before #902 #1074 (540852750). Occurrences after ~06:24Z today are the pre-fix image still running on the pods; the noise stops on the next image roll.
Regression pin: Pin #1127: the User NodeType declaration is never treated as an account #1157 adds the end-to-end test (falsified red with the filter removed — the phantom node appears on stores without partition schemas — green with it in place), plus the What's New entry for this fix.
Sweep: no other writer constructs a literal User/... path; all other nodeType:User consumers are content-filtered or read-only, and declaration nodes never pass through the CreateNode pipeline, so the seed handlers can't reproduce this.

Traced the write path. The schema is not missing by accident — "User" is being used as a user id.

The chain

AiSettingsNodeType.PathFor(user) builds {user}/_Memex/AiSettings. The reported path is literally User/_Memex/AiSettings, so {user} resolved to the string "User". Its first segment then routes to schema user (the router lowercases the first segment), and that schema does not exist — hence 42P01.

It does not exist by design. IncludedPartitionStaticProvider records it: "Auth was historically User before V27 renamed the central auth-lookup partition." So user is a schema that was renamed away; nothing provisions it now.

Where `"User"` comes from

AiSourcesInstallHook.Users():

// No namespace filter: on a partitioned Postgres backend a `namespace:User` query
// matches nothing (the users live in their own partitions).
.GetQuery("ai-sources-users", $"nodeType:{UserNodeType} select:path,id,name,nodeType")
.Select(nodes => nodes.Select(n => n.Id) …)

It takes n.Id verbatim for every nodeType:User node, with the namespace filter deliberately removed. So a User-typed node whose Id is "User" — a legacy User partition root, or the typeless stub a satellite write auto-anchors (LegacyUserPartitionRepair describes exactly that shape) — is treated as a user id.

This matches the evidence precisely: all 27 occurrences target the same path, i.e. one bogus "user" retried, not 27 users failing.

What I could not confirm

Which node carries Id = "User" — that needs a read against the live store, e.g.

search "nodeType:User" and inspect for an entry whose id/path is exactly `User`

I did not want to ship a fix on an unverified premise, so this is analysis rather than a patch.

Suggested fix, once confirmed

Users() should not accept a reserved partition name as a user id. IncludedPartitionStaticProvider.ReservedNames already carries the exact set (Admin, Auth, User, Portal, Kernel, plus the global satellite namespaces) and already keeps "User" in it "so any live deployment that still references the old name is filtered out alongside the new Auth name" — the same reasoning applies here.

🚨 Worth stating plainly: filtering the id is the right fix only if the offending node is a legacy artefact. If a real user genuinely has Id = "User", the filter would silently stop serving them, and the correct fix is repairing that node instead. Please confirm which before patching.

Related: the same "first segment → seg.ToLowerInvariant()" rule is what makes Acme and ACME the same partition (seen in #983), and a PartitionDefinition with Schema left null provisions the schema verbatim while writes hit the lowercased name — another 42P01 shape worth ruling out if the above does not hold.

Closing out the one open question from the analysis above — "which node carries Id = "User", and is it a legacy artefact or a real account?" — with a read-only check against the live memex-cloud portal (the namespace in the evidence footer).

It is the NodeType declaration. No real account is affected.

nodeType:User select:path,id,name,nodeType returns exactly two rows:

path	name	nodeType	version	lastModified
`User`	User	User	(none)	(none)
`rbuergi`	Roland Buergi	User	6	2026-08-04

The decisive detail is the absent version / lastModified on the User row: every persisted node carries both. It is served by the static provider, never written to storage — and get @User confirms it from the other side, returning Not found: User. So the row is the self-typed declaration (UserNodeType.CreateMeshNode() → new MeshNode("User") { NodeType = "User" }, i.e. Id == Path == NodeType == "User"), not an account.

That settles the caveat raised above: filtering this id cannot stop serving a real user, because there is no real user behind it — and an account with that id is impossible anyway, since User is a reserved partition name.

Nothing was lost. path:User/_Memex/AiSettings → 0 results; every phantom write aborted on the 42P01 and none persisted. Meanwhile the same repair pass reconciled all 7 real accounts correctly (iser.steinmetz, kalpesh.patel, lucas.mebold, rbuergi, roland.buergi, rsalzmann, thomager12), each AiSettings last modified 12:55Z today. The failed writes carried only machine-generated package-source rows for an account that does not exist.

The enumeration fix is on main (#1074); #1157 adds the regression pin.

_{Read-only verification: search / get via MCP. Nothing on the portal was mutated.}

Merge pull request #1157 from Systemorph/fix/1127-aisettings-phantom-…

Reopened — this fault is happening again.

Still occurring: 106 more since the last update (133 total).

Last seen: 2026-08-13 16:23:01Z
Pods: memex-portal-deployment-6956d54bb5-l6mxv, memex-portal-deployment-8476467cb6-nkbhh, memex-portal-deployment-5bbfc97c64-lq7lw, memex-portal-deployment-76666bd6f5-mghv8, memex-portal-deployment-78bb6bfd5-fpvxk, memex-portal-deployment-cc58dc7c6-ckll8, memex-portal-deployment-bdc9f8dc4-fdkgw, memex-portal-deployment-7d4b5684bb-9lrxl, memex-portal-deployment-7574f9474c-zvqzn, memex-portal-deployment-585f6d7b98-z7ktk, memex-portal-deployment-7745468d76-m65r5, memex-portal-deployment-8678ff8df4-v4wvn, memex-portal-deployment-678559d6df-dgfmh, memex-portal-deployment-6b58bb7468-j5z2z, memex-portal-deployment-7cc5d4c5f5-m8dbt, memex-portal-deployment-55474fd686-b5dqp, memex-portal-deployment-568978584d-rqg2v, memex-portal-deployment-677c8d4fd5-wfzm8, memex-portal-deployment-54d5d4cfc7-4xjsd, memex-portal-deployment-7867998575-tsc42, memex-portal-deployment-844cd49b6-b29lb, memex-portal-deployment-59b7548d9-ctp9r, memex-portal-deployment-55b746c9fc-9n7jg, memex-portal-deployment-7756b5d5b9-6nwjr, memex-portal-deployment-7b87466cf4-zvj4h, memex-portal-deployment-66dc8c687b-6s9hz, memex-portal-deployment-68f7487c67-gznnl, memex-portal-deployment-6964d7f5c-tdx62, memex-portal-deployment-67b44d896d-4gzk9, memex-portal-deployment-54b8768c8f-6flz5, memex-portal-deployment-69b48f968c-5g7lg, memex-portal-deployment-d6f5dc4f6-9j249, memex-portal-deployment-7b67dcb585-vmgf6, memex-portal-deployment-78b498648b-zz524, memex-portal-deployment-7476f9847c-j486p, memex-portal-deployment-754bdc4d55-4cwcm, memex-portal-deployment-75dbd57bbb-n6nhf, memex-portal-deployment-577859654c-gbrbj, memex-portal-deployment-566596dd6-mn5fb, memex-portal-deployment-665c845455-pmbgr, memex-portal-deployment-84c8cdc75-5g78r, memex-portal-deployment-65d9d689bb-jhzx8

Recent log lines

2026-08-13 15:04:32Z memex-portal-deployment-665c845455-pmbgr fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at MeshWeaver/src/MeshWeaver.Markdown.Export/Html/DocumentAreaResolution
      Npgsql.PostgresException (0x80004005): 22P05: unsupported Unicode escape sequence
      
      DETAIL: Detail redacted as it may contain sensitive data. Specify 'Include Error Detail' in the connection string to include this information.
         at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteNonQuery(Boolean async, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 630
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions o…[truncated]
2026-08-13 15:30:00Z memex-portal-deployment-84c8cdc75-5g78r fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at MeshWeaver/src/MeshWeaver.Markdown.Export/Html/DocumentAreaResolution
      Npgsql.PostgresException (0x80004005): 22P05: unsupported Unicode escape sequence
      
      DETAIL: Detail redacted as it may contain sensitive data. Specify 'Include Error Detail' in the connection string to include this information.
         at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteNonQuery(Boolean async, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 630
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions o…[truncated]
2026-08-13 16:23:01Z memex-portal-deployment-65d9d689bb-jhzx8 fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at MeshWeaver/src/MeshWeaver.Markdown.Export/Html/DocumentAreaResolution
      Npgsql.PostgresException (0x80004005): 22P05: unsupported Unicode escape sequence
      
      DETAIL: Detail redacted as it may contain sensitive data. Specify 'Include Error Detail' in the connection string to include this information.
         at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteNonQuery(Boolean async, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 630
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.WriteAsyncCore(MeshNode node, JsonSerializerOptions o…[truncated]

Closing again — the root cause is gone from origin/main, and gone at a deeper level than the fix that closed this originally. Silent for 35 days.

Quiet. The reopen of 2026-09-17T07:52:18Z cites Last seen: 2026-08-13 16:23:01Z — 35.6 days before today (2026-09-18). Ingest is live (the incident store was updating at 2026-09-18T05:42Z), so the silence is the record. Still occurring: 106 more compares accumulated totals against the last update.

Code check. The defect was never a missing migration: the self-typed User NodeType declaration carried nodeType: User, so the pathless nodeType:User enumeration in AiSourcesInstallHook.Users() returned it alongside real accounts, AiSettingsNodeType.PathFor("User") produced User/_Memex/AiSettings, and the router resolved a partition literally named user — which nothing provisions, hence 42P01: relation "user.mesh_nodes" does not exist.

Today that cannot happen, because the declaration no longer claims to be a user. origin/main at 229930bf2f, src/MeshWeaver.Graph/Configuration/UserNodeType.cs:

$ git show origin/main:src/MeshWeaver.Graph/Configuration/UserNodeType.cs | sed -n '96,112p'
public static MeshNode CreateMeshNode() => new(NodeType)
{
    Name = "User",
    // 🚨 A NodeType DECLARATION declares itself a NodeType — never the type it declares.
    // This node used to carry `NodeType = "User"` … The same collision made AI-source installs
    // resolve a partition literally named "User" (`42P01: relation "user.mesh_nodes" does not
    // exist`) and, for the VUser twin, left a stray `vuser` partition behind.
    NodeType = MeshNode.NodeTypePath,

The comment names this failure verbatim. It is enforced, not just written:

$ git ls-tree -r origin/main --name-only | grep -i SelfTyping
src/MeshWeaver.Graph/Security/NodeTypeDeclarationSelfTypingValidator.cs
src/MeshWeaver.Graph/Security/SelfTypedDeclarationDurableRepair.cs
test/MeshWeaver.Graph.Test/NodeTypeDeclarationSelfTypingValidatorTest.cs

And the consumer (now in MeshWeaver.Plugins with the AI engine, #2276) has removed the hand-rolled id != "User" filter that PR #1074 added here, because the cause it guarded against is gone — MeshWeaver.Plugins@origin/main src/MeshWeaver.AI/AiSourcesInstallHook.cs:82-90:

// The hand-rolled 'id != "User"' filter that used to sit here is GONE with its cause … Declarations now declare themselves (NodeType = MeshNode.NodeTypePath, UserNodeType.CreateMeshNode), so this query returns accounts only — and NodeTypeDeclarationSelfTypingTest keeps it that way for every built-in type, not just the two this filter happened to name.

The regression pin from #1157 is live there too: src/MeshWeaver.AI.Test/AiSourcesInstallHookTest.cs asserts, citing this issue by number, that nothing is served at User/_Memex/AiSettings — read through GetMeshNodeStream rather than a query, precisely so the assertion cannot pass vacuously.

Nothing was ever lost: every phantom write aborted on the 42P01 and none persisted, while the same repair pass reconciled all real accounts correctly (verified read-only on the live portal 2026-08-10, on this thread).

If this fault recurs, the recurrence bot reopens this issue with fresh evidence — that is the designed behaviour, and it is why closing a dormant incident is safe.

Navigation Menu

Repository navigation

PostgreSQL schema 'user.mesh_nodes' missing — node creation fails on all pods #1127

What is failing

Probable cause — high confidence

Impact

Where to look

Existing reports

Evidence

The chain

Where `"User"` comes from

What I could not confirm

Suggested fix, once confirmed

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

PostgreSQL schema 'user.mesh_nodes' missing — node creation fails on all pods #1127

Description

What is failing

Probable cause — high confidence

Impact

Where to look

Existing reports

Evidence

Activity

rbuergi commented on Aug 10, 2026

rbuergi commented on Aug 10, 2026

The chain

Where "User" comes from

What I could not confirm

Suggested fix, once confirmed

rbuergi commented on Aug 10, 2026

systemorph-com commented on Sep 17, 2026

rbuergi commented on Sep 18, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Where `"User"` comes from