Node creation fails when PostgreSQL connector open times out in PostgreSqlStorageAdapter #3050

What is failing

Node creation (MeshWeaver.Mesh.CreateNode) fails because the PostgreSQL storage layer cannot open a connection to the database host: PostgreSqlStorageAdapter.ExistsAsyncCore() times out while Npgsql tries to establish a new physical connection (NpgsqlConnector.ConnectAsync → TimeoutException), and the CreateNode handler surfaces this as an unexpected node-creation error. The node that failed to be created was a sample-graph data node under the built-in MeshWeaver/samples/Graph content.

Probable cause

Infrastructure-level PostgreSQL connectivity/availability issue — high confidence that this is the mechanism, moderate confidence on the precise trigger. The stack shows the timeout occurred while opening a brand-new connector (not while executing a query and not while waiting on an existing pool slot), which means the database host was unreachable or accepting connections too slowly at that moment. This is corroborated by sibling incidents in the same failure window and on the same pod: incident a56cefad0e4d4b92 shows the identical connection-open timeout from PostgreSqlStorageAdapter.ReadAsyncCore() seconds before and after this event, and its draft already links this PostgreSQL connectivity problem to existing issue Systemorph/MeshWeaver#1310. A database host restart, resource exhaustion, or brief network partition in the cluster would produce exactly this pattern across all storage-adapter call sites simultaneously.

This incident is likely a duplicate symptom of the same root cause as #1310 / a56cefad0e4d4b92 — filed separately only because the top frame and exception text differ. Consider closing as a duplicate once the underlying PostgreSQL availability issue is tracked.

Impact

Low — isolated and transient. A single node creation failed on one pod, with no further occurrences after the brief failure window. The failed node was sample content under MeshWeaver/samples/Graph, so no user data was lost; a retry would succeed once the database is reachable. However, the same connectivity blip took down multiple storage operations at once (see sibling incident), so if this window recurs at higher volume it would block node creation platform-wide — worth confirming the database host's health around that time.

Where to look

• PostgreSqlStorageAdapter.ExistsAsyncCore() — per the stack trace, plugins-repo/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs around line 1066 (the framework source is not present in this mesh, so no mesh link).
• The Npgsql data-source/pool configuration (connection timeout, pool size, OpenNewConnector behavior) used by MeshWeaver.Hosting.PostgreSql.
• Database host availability in the cluster during the failure window; cross-check sibling incident a56cefad0e4d4b92 and issue Activity tracking fails when PostgreSQL connection pool is temporarily exhausted #1310, which share the identical connection-open timeout.

Also note that CreateNode reports this as a generic "Unexpected error" — consider distinguishing storage-connectivity failures (retryable) from genuine creation errors in the log category, so operators can tell an infra blip from a defect.

Evidence

2026-09-02 03:08:52Z memex-portal-deployment-85859db964-7sqbk fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at MeshWeaver/samples/Graph/Data/PensionFund/BalanceSheetEntry/2024-RealEstate: NpgsqlException: The operation has timed out
      Npgsql.NpgsqlException (0x80004005): The operation has timed out
       ---> System.TimeoutException: The operation has timed out.
         at Npgsql.ThrowHelper.ThrowNpgsqlExceptionWithInnerTimeoutException(String message)
         at Npgsql.Util.NpgsqlTimeout.Check()
         at Npgsql.Util.NpgsqlTimeout.CheckAndGetTimeLeft()
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.ExistsAsyncCore(String path, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 1066
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorage…[truncated]

_{Opened automatically from Admin/_LogIncident/a06ed666b8b0198b. Recurrences are folded into this issue rather than opening new ones.}