Node creation fails when PostgreSQL is unreachable — connection timeouts opening new connectors in PostgreSqlStorageAdapter

What is failing

Node creation fails whenever the PostgreSQL backing store cannot be reached: the CreateNode pipeline's storage read (MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.ReadAsyncCore) times out while opening a new Npgsql connector to the database host, and the whole create operation aborts with an "Unexpected error during node creation".

Probable cause

High confidence that this is database-host unavailability or saturation, not an application defect. Every sample is a raw NpgsqlException wrapping System.TimeoutException, thrown from NpgsqlConnector.ConnectAsync during PoolingDataSource.OpenNewConnector — i.e. the failure happens while establishing the connection, before any SQL runs. The pool trying to open fresh connectors under load and timing out is the classic signature of the DB host being unreachable or accepting connections too slowly.

Supporting evidence: in the same window the PostgreSQL LISTEN/notify channel also failed to connect to the same host with the identical timeout (incident Admin/_LogIncident/fc4f22eb5cffabd0, same pod and same host) — the entire PostgreSQL backend was unreachable, not one call site. The earlier triage of this same fingerprint linked it to GitHub issue #1310 (activity-tracking failures on the same host and port, pool exhaustion variant); I could not re-verify #1310's current state from the mesh, so treat that link as reported rather than confirmed.

The fingerprint has now recurred on a newer deployment generation, so whatever makes the PostgreSQL host slow/unreachable to accept connections is still happening intermittently — worth investigating on the infrastructure side (DB resource saturation, pod-to-DB networking, failover events).

Impact

Bursty and self-recovering, not a sustained outage: node creation fails only while the database is unreachable and recovers without intervention. Affected paths include user thread nodes and sample-data creation, not just internal bookkeeping — during an outage window, any node-create attempt would fail. Given the bursts recur across deployments, this is an intermittent availability risk to all mesh writes until the host-side issue is addressed.

Where to look

MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.ReadAsyncCore — where the timeout surfaces at the ExecuteReader connection open (build paths in the stacks: line 381 in the src copy, line 399 in the plugins-repo copy).
Health of the PostgreSQL host 10.42.18.4:5432 during each burst — connection saturation, resource limits, or network reachability.
Related: incident fc4f22eb5cffabd0 (same window, LISTEN reconnect failure on the same host) and GitHub issue Activity tracking fails when PostgreSQL connection pool is temporarily exhausted #1310 per the prior triage — likely the same root cause; consider consolidating with Activity tracking fails when PostgreSQL connection pool is temporarily exhausted #1310 if it is still open.

Evidence


Fingerprint	`a56cefad0e4d4b92`
Category	`MeshWeaver.Mesh.CreateNode`
Severity	Error
Top frame	`MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.ReadAsyncCore(String path, JsonSerializerOptions options, CancellationToken ct)`
Namespace	`memex`
Pods	`memex-portal-deployment-688b496b7-2sdt8`, `memex-portal-deployment-688b496b7-brp4j`, `memex-portal-deployment-85859db964-7sqbk`
Occurrences	9
First seen	2026-08-23 19:02:11Z
Last seen	2026-09-02 03:09:03Z

Recent log lines

2026-08-23 19:02:11Z memex-portal-deployment-688b496b7-2sdt8 fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at Admin/_LogIncident/0d7c9b186265f004/_Thread/triage-the-log-incident-at-admin-loginci-39de/_Notification/irPfj9lNE0unStJaroV88w
      Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.42.18.4:5432
       ---> System.TimeoutException: Timeout during connection attempt
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.ReadAsyncCore(String path, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 381
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.ReadAsyncCore(String path, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/…[truncated]
2026-08-23 19:02:11Z memex-portal-deployment-688b496b7-2sdt8 fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at Admin/_LogIncident/0d7c9b186265f004/_Thread/triage-the-log-incident-at-admin-loginci-39de/_Usage/DeepSeek_V4_Flash
      Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.42.18.4:5432
       ---> System.TimeoutException: Timeout during connection attempt
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.ReadAsyncCore(String path, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 381
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.ReadAsyncCore(String path, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlSt…[truncated]
2026-08-23 19:02:21Z memex-portal-deployment-688b496b7-brp4j fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at Admin/_LogIncident/0d7c9b186265f004/_Thread/triage-the-log-incident-at-admin-loginci-39de/_Usage/DeepSeek_V4_Flash
      Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.42.18.4:5432
       ---> System.TimeoutException: Timeout during connection attempt
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.ReadAsyncCore(String path, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 381
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.ReadAsyncCore(String path, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlSt…[truncated]
2026-08-23 19:02:21Z memex-portal-deployment-688b496b7-brp4j fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at Admin/_LogIncident/0d7c9b186265f004/_Thread/triage-the-log-incident-at-admin-loginci-39de/_Notification/DHcqgac9SUe4l4Cw6Fzcdw
      Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.42.18.4:5432
       ---> System.TimeoutException: Timeout during connection attempt
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.ReadAsyncCore(String path, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 381
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.ReadAsyncCore(String path, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/…[truncated]
2026-08-23 19:26:08Z memex-portal-deployment-688b496b7-brp4j fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at rsalzmann/_Thread/ThreadComposer
      Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.42.18.4:5432
       ---> System.TimeoutException: Timeout during connection attempt
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.ReadAsyncCore(String path, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 381
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.ReadAsyncCore(String path, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 385
         at MeshWeaver.Mesh.Threading.IoPool.<>c__Display…[truncated]
2026-08-23 19:30:33Z memex-portal-deployment-688b496b7-brp4j fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at rsalzmann/_Thread/ThreadComposer
      Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.42.18.4:5432
       ---> System.TimeoutException: Timeout during connection attempt
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.ReadAsyncCore(String path, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 381
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.ReadAsyncCore(String path, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 385
         at MeshWeaver.Mesh.Threading.IoPool.<>c__Display…[truncated]
2026-08-23 19:30:33Z memex-portal-deployment-688b496b7-brp4j fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at rsalzmann/_Thread/ThreadComposer
      Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.42.18.4:5432
       ---> System.TimeoutException: Timeout during connection attempt
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.ReadAsyncCore(String path, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 381
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.ReadAsyncCore(String path, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 385
         at MeshWeaver.Mesh.Threading.IoPool.<>c__Display…[truncated]
2026-09-02 03:08:34Z memex-portal-deployment-85859db964-7sqbk fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at MeshWeaver/samples/Graph/Data/PensionFund/BalanceSheetEntry/2025-ActiveMembersCapital: NpgsqlException: The operation has timed out
      Npgsql.NpgsqlException (0x80004005): The operation has timed out
       ---> System.TimeoutException: The operation has timed out.
         at Npgsql.ThrowHelper.ThrowNpgsqlExceptionWithInnerTimeoutException(String message)
         at Npgsql.Util.NpgsqlTimeout.Check()
         at Npgsql.Util.NpgsqlTimeout.CheckAndGetTimeLeft()
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter…[truncated]
2026-09-02 03:09:03Z memex-portal-deployment-85859db964-7sqbk fail: MeshWeaver.Mesh.CreateNode[0]
      Unexpected error during node creation at MeshWeaver/samples/Graph/Data/PensionFund/BalanceSheetEntry/2024-NonTechnicalProvisions: NpgsqlException: The operation has timed out
      Npgsql.NpgsqlException (0x80004005): The operation has timed out
       ---> System.TimeoutException: The operation has timed out.
         at Npgsql.ThrowHelper.ThrowNpgsqlExceptionWithInnerTimeoutException(String message)
         at Npgsql.Util.NpgsqlTimeout.Check()
         at Npgsql.Util.NpgsqlTimeout.CheckAndGetTimeLeft()
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.ReadAsyncCore(String path, JsonSerializerOptions options, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 399
         at MeshWea…[truncated]

_{Opened automatically from Admin/_LogIncident/a56cefad0e4d4b92. Recurrences are folded into this issue rather than opening new ones.}

Two findings: the platform-side defect is fixed — and the stated cause is wrong for these samples

#3050 is closed as a duplicate of this issue (same log line, same handler, same fault class; the two fingerprints differ only in which storage call happened to try opening a connector first — the reasoning is on that issue).

1. The platform defect, and the judgement behind the fix

The question is what a create should DO when the store is briefly unreachable, and there were three candidate answers:

Candidate	Verdict
Retry here	Rejected. The bounded retry already ran — `TransientStorageFaults.RetryTransientConnect` (#2521): 250 → 500 → 1000 ms, then the last error surfaces. A fault reaching the handler's terminal arm is one whose budget is honestly spent, so a retry here would aim a second, unbounded-in-aggregate retry at the resource that is already the bottleneck. That is the same non-choice #3031 states for the render path
Fail, as now	Rejected. Honest about the failure, dishonest about its kind. `Unknown` under "Unexpected error during node creation" is indistinguishable from a verdict
Report "unavailable — not refused"	Chosen

The distinction has teeth in exactly one direction: a create that reads as refused tells a caller to stop, and a caller that stops using the id it was retrying and mints a fresh one on its next attempt writes a duplicate — #2229's shape arriving through the reporting layer rather than through a stale query.

Both ends of that wire already existed and were simply not connected. NodeCreationRejectionReason.Unavailable was added for a different unreachable dependency (#1446) and its documented meaning is exactly right — "The create was NOT evaluated … an availability failure, not a verdict." StorageFaults.IsTransientConnectFault is the one classification rule, extracted by #3031 into the assembly all consumers can see. The fix is the branch between them, on both create verbs (CreateNodeRequest and the bulk CreateNodesRequest every installer and static-repo import travels — a guard on one and not the other is a guard on neither).

What the caller now gets:

Node creation at 'Acme/Reports/Monthly' could not be attempted: the data store was unreachable. Nothing was written — this is an availability failure, not a refusal, so retrying the same request (with the same node id) is meaningful.

…and the log line names the store rather than the create, staying at Error where an operator sees it.

Falsification, deliberately included: a real query error (42P01) travels the identical path and must still answer Unknown. Excusing a defect as an outage is the mirror image of this bug, and it is the one that hides — an operator told "temporarily unavailable" about a missing table waits for it to come back. The test asserts both directions, plus a positive control that a create on a healthy path still lands. On the pre-fix code 3 of its 5 cases fail and the other 2 pass, so the change is attributable rather than merely correlated.

2. 🚨 The stated cause does not survive measurement — PostgreSQL was probably not the problem

This issue and #3050 both conclude, with "high confidence", that this is "database-host unavailability or saturation, not an application defect", and send the reader to check the health of 10.42.18.4:5432.

That is not supported by the evidence, at least for the 2026-09-02 samples. A connect timeout is wall-clock and does not pause for GC, so a stalled client produces the byte-identical NpgsqlConnector.ConnectAsync → TimeoutException. Over the affected pod's full retained log (3 h 05 min, memex-portal-deployment-85859db964-7sqbk):


`Npgsql.NpgsqlException` lines	329
`.NET Runtime Platform stalled for …` reports	2 378
Npgsql exceptions emitted within 100 ms after a stall report	294 / 329 = 89.4 %
Share of the window those 100 ms shadows cover	1.77 %
Enrichment over chance	≈ 50×

The coverage figure is the control, and it is what makes this more than "both lines flush when the process resumes": had the database genuinely been unreachable, its timeouts would be spread across the other 98 % of the window. They are not — only 11 of 329 (3.3 %) fall outside even a 5-second shadow of a stall. Sample from the burst:

03:07:01.918  .NET Runtime Platform stalled for 00:00:06.41. GC Pause 00:00:06.33. 11798MB
03:07:01.920  Npgsql.NpgsqlException: The operation has timed out      <- 2 ms later
   … 8 more, all within 35 ms …
03:07:06.898  .NET Runtime Platform stalled for 00:00:04.98
03:07:06.914  Npgsql.NpgsqlException: Failed to connect to 10.42.18.5:5432   <- 16 ms later

That pod was at 96.4 % of its managed-heap hard limit (11 850 MB of an implicit 12 GiB — no DOTNET_GCHeapHardLimit-family variable is set, so .NET takes 75 % of the 16 GiB cgroup limit), threw 43 managed OutOfMemoryExceptions between 02:53:29 and 03:12:45, self-reported 38-second thread-pool delays, and was evicted from the Orleans cluster at 03:12:52 for exactly that reason (#3052 / #3053 — the same host condition wearing different incident numbers). The heap grew ~6.6 GB in four hours.

So the fix stands and the investigation pointer does not. "This process could not reach the store, so the create was not attempted" is true whichever side stalled, and it is all the handler can honestly assert. But the ticket to open is not against PostgreSQL — the memory-growth cluster from this same window is #3044, #3045, #3046, #3049, and what grew the heap between 23:17 and 03:12 is still open there. The 14.6 GiB core dump retained at /data/dumps/coredump.1.1788318781 on that pod is what would answer it.

What this measurement does NOT cover: the eight earlier occurrences on this fingerprint (2026-08-23, pods 688b496b7-2sdt8 / -brp4j). Those pods' logs are gone, so whether they share the client-stall signature is now unmeasurable — not "ruled out". If this fingerprint recurs, the discriminator is cheap and is written down in the doc page below: check the calling pod's own stall lines in the same window, and whether other clients of the same database timed out at the same moment.

Conserved

Doc/Architecture/StoreUnreachableIsNotARefusal — the one classification and its three consumers, the retry-once-then-answer ladder, the falsification boundary, this measurement and the reading rule it produces, and the one known edge left open (the in-process exception mapping still collapses Unavailable and Unknown into InvalidOperationException, and the delete/move/upsert verbs have not been swept for this condition at all).
Doc/Architecture/ReadingASiloEviction — the host-stall side of the same night.
A What's New entry (Category: Fix).

fix(#3051): do not claim "nothing was written" when the write had sta…

fix(mesh): a create whose store is unreachable answers Unavailable, n…

Reopened — this fault is happening again.

Still occurring: 8 more since the last update (17 total).

Last seen: 2026-09-02 11:17:43Z
Pods: memex-portal-deployment-688b496b7-2sdt8, memex-portal-deployment-688b496b7-brp4j, memex-portal-deployment-85859db964-7sqbk, memex-portal-deployment-c7f5ff965-vk9zq

Recent log lines

2026-09-02 11:17:17Z memex-portal-deployment-c7f5ff965-vk9zq fail: MeshWeaver.Mesh.CreateNode[0]
      [CreateNode] STORE UNREACHABLE at Crm/Contact/Release/20260902111531-z15nEpQP: NpgsqlException: The operation has timed out — the create was NOT attempted and nothing was written. The fault is the data store, not this create; the caller is answered Unavailable, so a retry with the same node id is meaningful.
      Npgsql.NpgsqlException (0x80004005): The operation has timed out
       ---> System.TimeoutException: The operation has timed out.
         at Npgsql.ThrowHelper.ThrowNpgsqlExceptionWithInnerTimeoutException(String message)
         at Npgsql.Util.NpgsqlTimeout.Check()
         at Npgsql.Util.NpgsqlTimeout.CheckAndGetTimeLeft()
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean…[truncated]
2026-09-02 11:17:43Z memex-portal-deployment-c7f5ff965-vk9zq fail: MeshWeaver.Mesh.CreateNode[0]
      [CreateNode] STORE UNREACHABLE at Crm/Board/_Activity/compile-202609021115517764d343e47ff6947978c22b99f83355d60: NpgsqlException: The operation has timed out — the create was NOT attempted and nothing was written. The fault is the data store, not this create; the caller is answered Unavailable, so a retry with the same node id is meaningful.
      Npgsql.NpgsqlException (0x80004005): The operation has timed out
       ---> System.TimeoutException: The operation has timed out.
         at Npgsql.ThrowHelper.ThrowNpgsqlExceptionWithInnerTimeoutException(String message)
         at Npgsql.Util.NpgsqlTimeout.Check()
         at Npgsql.Util.NpgsqlTimeout.CheckAndGetTimeLeft()
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataS…[truncated]
2026-09-02 11:17:43Z memex-portal-deployment-c7f5ff965-vk9zq fail: MeshWeaver.Mesh.CreateNode[0]
      [CreateNode] STORE UNREACHABLE at Crm/Contact/Release/20260902111542-z15nEpQP: NpgsqlException: The operation has timed out — the create was NOT attempted and nothing was written. The fault is the data store, not this create; the caller is answered Unavailable, so a retry with the same node id is meaningful.
      Npgsql.NpgsqlException (0x80004005): The operation has timed out
       ---> System.TimeoutException: The operation has timed out.
         at Npgsql.ThrowHelper.ThrowNpgsqlExceptionWithInnerTimeoutException(String message)
         at Npgsql.Util.NpgsqlTimeout.Check()
         at Npgsql.Util.NpgsqlTimeout.CheckAndGetTimeLeft()
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean…[truncated]

Closing — the platform fix merged, and the fault has been quiet 16 days

Dormancy. Newest occurrence Last seen: 2026-09-02 11:17:43Z (the bot's reopen comment above). Measured 2026-09-18T05:52Z that is 16 days quiet, with live log ingest (incidents updated 2026-09-18T05:42Z) and no portal roll in between.

This issue was already closed on its fix (2026-09-02) and the 2026-09-17 reopen was part of the stale-evidence sweep (#4577 / #4610) — its cited occurrence is 3 hours after the close, i.e. the old replicas still emitting between merge and roll. The fix is in origin/main at 229930bf2f29845323e0f350e73829b1bdb4928e:

$ git -C <core> grep -n 'NodeCreationRejectionReason.Unavailable' origin/main -- 'src/**/*.cs'
src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:1324, 1360, 1613, 2198   <- both create verbs
src/MeshWeaver.Mesh.Contract/CreateNodeRequest.cs:561, 1003             <- the documented contract

A create whose store could not be reached now answers Unavailable — "the create was NOT evaluated … an availability failure, not a verdict" — on both CreateNodeRequest and the bulk CreateNodesRequest, instead of falling into the catch-all as Unknown under "Unexpected error during node creation". That distinction is the one with teeth: a caller that reads "refused", stops retrying the same id and mints a fresh one on its next attempt writes a duplicate (#2229's shape). No retry was added — the bounded one (TransientStorageFaults.RetryTransientConnect, #2521) already runs upstream.

Pinned, with the falsification case included, by test/MeshWeaver.Graph.Test/CreateWhenTheStoreIsUnreachableTest.cs:

/// 🚨 <b>A CREATE WHOSE STORE IS UNREACHABLE WAS NOT REFUSED — IT WAS NEVER ATTEMPTED</b>
/// (MeshWeaver#3050 / MeshWeaver#3051).
/// <b>What is deliberately NOT tested here, because it must not exist:</b> a retry.

— and a real query error (42P01) travelling the same path must still answer Unknown, so the mirror-image bug (excusing a defect as an outage) is guarded too. Conserved as Doc/Architecture/StoreUnreachableIsNotARefusal.

The investigation pointer in this ticket's body does not stand, and that is also recorded above: the 2026-09-02 samples measured 89.4 % of Npgsql timeouts inside a 100 ms shadow of a GC stall covering 1.77 % of the window (≈50× enrichment) — a stalled client, not an unreachable PostgreSQL host. That host-side cluster (#3044, #3045, #3046, #3049, #3052, #3053) is all closed. So there is nothing left here to keep open on either reading.

Closing as redundant. If the fault recurs, the recurrence bot reopens this issue with fresh evidence — the designed behaviour, and why closing a dormant incident is safe.

Navigation Menu

Repository navigation

Node creation fails when PostgreSQL is unreachable — connection timeouts opening new connectors in PostgreSqlStorageAdapter #3051

What is failing

Probable cause

Impact

Where to look

Evidence

Two findings: the platform-side defect is fixed — and the stated cause is wrong for these samples

1. The platform defect, and the judgement behind the fix

2. 🚨 The stated cause does not survive measurement — PostgreSQL was probably not the problem

Conserved

Closing — the platform fix merged, and the fault has been quiet 16 days

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Node creation fails when PostgreSQL is unreachable — connection timeouts opening new connectors in PostgreSqlStorageAdapter #3051

Description

What is failing

Probable cause

Impact

Where to look

Evidence

Activity

rbuergi commented on Sep 2, 2026

Two findings: the platform-side defect is fixed — and the stated cause is wrong for these samples

1. The platform defect, and the judgement behind the fix

2. 🚨 The stated cause does not survive measurement — PostgreSQL was probably not the problem

Conserved

systemorph-com commented on Sep 17, 2026

rbuergi commented on Sep 18, 2026

Closing — the platform fix merged, and the fault has been quiet 16 days

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions