Log-incident control plane permanently stops ticketing red logs after a Postgres connection timeout — requires restart to recover

What is failing

The log-incident control plane (MeshWeaver.Observability.LogIncidentControlPlane) cannot execute its periodic log-incident query: every attempt to read incidents from Postgres times out while opening a new connection (NpgsqlException wrapping System.TimeoutException from NpgsqlConnector.ConnectAsync / RawOpen). By the component's own message, once this happens red logs stop being ticketed until the pod is restarted — the triage/ticketing pipeline silently goes dark instead of retrying.

Probable cause

Two layered problems, both supported by the stack traces:

Infra (trigger): Postgres at 10.42.18.4:5432 was unreachable or too slow to connect from the portal pods — both the 2026-08-23 burst (Failed to connect ... Timeout during connection attempt) and the 2026-09-02 recurrence (The operation has timed out) fail in Npgsql.PoolingDataSource.OpenNewConnector → NpgsqlConnector.ConnectAsync, i.e. never get to run a query. Confidence: high for the connection-timeout fact itself; the underlying reason (Postgres down, overloaded, pool exhaustion, or network) is not determinable from the logs alone.
App defect (amplifier): a single failed query appears to put the control plane into a latched fault state that it never retries out of — the log message itself says "no longer be ticketed until restart". A transient DB outage should not permanently disable error ticketing in every affected pod. Confidence: medium — inferred from the message and the fact that pods keep the state until restart; the source for LogIncidentControlPlane is not present in this mesh to confirm.

Impact

18 occurrences over ~10 days (2026-08-23 → 2026-09-02) across 4 pods — low volume, but this is a monitoring-pipeline outage, not a user-facing one: while latched, any production error in the mesh goes un-ticketed and un-tracked. It recurs (a fresh occurrence landed on pod b964b4c4c-wmrhr at 2026-09-02 17:32 UTC, minutes before this triage), so whatever ticketing should have been filed during those windows was lost. The Aug 23 burst on two pods of replica set 7b5c857c97 suggests both pods were latched simultaneously — a shared DB hiccup can blind the whole ticketing layer.

Where to look

MeshWeaver.Observability.LogIncidentControlPlane — the latching/no-recovery behaviour after a failed query; add retry/backoff or a circuit breaker that re-arms, instead of requiring a restart.
MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync — src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:1405 (from the stack frame) is where the query opens its connection.
Ops side: Postgres availability/connect latency at 10.42.18.4:5432 around 2026-08-23 20:04–20:21 UTC and 2026-09-02 17:32 UTC.

No existing incident or issue covering this component/symptom was found in the mesh, so this does not look like a duplicate.

Evidence


Fingerprint	`45523b88fb120821`
Category	`MeshWeaver.Observability.LogIncidentControlPlane`
Severity	Error
Top frame	`MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, String activityUserId, IReadOnlyCollection`1 excludedNodeTypes, CancellationToken ct)+MoveNext()`
Namespace	`memex`
Pods	`memex-portal-deployment-64b5fbc96c-df8kr`, `memex-portal-deployment-7b5c857c97-cgdpw`, `memex-portal-deployment-7b5c857c97-zlc2b`, `memex-portal-deployment-b964b4c4c-wmrhr`
Occurrences	18
First seen	2026-08-23 19:49:40Z
Last seen	2026-09-02 17:32:05Z

Recent log lines

2026-08-23 20:04:38Z memex-portal-deployment-7b5c857c97-cgdpw fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Log-incident query failed — red logs will no longer be ticketed until restart
      Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.42.18.4:5432
       ---> System.TimeoutException: Timeout during connection attempt
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, String activityUserId, IReadOnlyCollection`1 excludedNodeTypes, CancellationToken ct)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 1405
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, S…[truncated]
2026-08-23 20:06:18Z memex-portal-deployment-7b5c857c97-cgdpw fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Log-incident query failed — red logs will no longer be ticketed until restart
      Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.42.18.4:5432
       ---> System.TimeoutException: Timeout during connection attempt
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, String activityUserId, IReadOnlyCollection`1 excludedNodeTypes, CancellationToken ct)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 1405
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, S…[truncated]
2026-08-23 20:08:29Z memex-portal-deployment-7b5c857c97-zlc2b fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Log-incident query failed — red logs will no longer be ticketed until restart
      Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.42.18.4:5432
       ---> System.TimeoutException: Timeout during connection attempt
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, String activityUserId, IReadOnlyCollection`1 excludedNodeTypes, CancellationToken ct)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 1405
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, S…[truncated]
2026-08-23 20:10:08Z memex-portal-deployment-7b5c857c97-zlc2b fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Log-incident query failed — red logs will no longer be ticketed until restart
      Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.42.18.4:5432
       ---> System.TimeoutException: Timeout during connection attempt
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, String activityUserId, IReadOnlyCollection`1 excludedNodeTypes, CancellationToken ct)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 1405
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, S…[truncated]
2026-08-23 20:11:58Z memex-portal-deployment-7b5c857c97-zlc2b fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Log-incident query failed — red logs will no longer be ticketed until restart
      Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.42.18.4:5432
       ---> System.TimeoutException: Timeout during connection attempt
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, String activityUserId, IReadOnlyCollection`1 excludedNodeTypes, CancellationToken ct)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 1405
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, S…[truncated]
2026-08-23 20:13:39Z memex-portal-deployment-7b5c857c97-zlc2b fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Log-incident query failed — red logs will no longer be ticketed until restart
      Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.42.18.4:5432
       ---> System.TimeoutException: Timeout during connection attempt
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, String activityUserId, IReadOnlyCollection`1 excludedNodeTypes, CancellationToken ct)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 1405
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, S…[truncated]
2026-08-23 20:16:05Z memex-portal-deployment-7b5c857c97-zlc2b fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Log-incident query failed — red logs will no longer be ticketed until restart
      Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.42.18.4:5432
       ---> System.TimeoutException: Timeout during connection attempt
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, String activityUserId, IReadOnlyCollection`1 excludedNodeTypes, CancellationToken ct)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 1405
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, S…[truncated]
2026-08-23 20:19:03Z memex-portal-deployment-7b5c857c97-zlc2b fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Log-incident query failed — red logs will no longer be ticketed until restart
      Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.42.18.4:5432
       ---> System.TimeoutException: Timeout during connection attempt
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, String activityUserId, IReadOnlyCollection`1 excludedNodeTypes, CancellationToken ct)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 1405
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, S…[truncated]
2026-08-23 20:20:43Z memex-portal-deployment-7b5c857c97-zlc2b fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Log-incident query failed — red logs will no longer be ticketed until restart
      Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.42.18.4:5432
       ---> System.TimeoutException: Timeout during connection attempt
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, String activityUserId, IReadOnlyCollection`1 excludedNodeTypes, CancellationToken ct)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 1405
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, S…[truncated]
2026-09-02 17:32:05Z memex-portal-deployment-b964b4c4c-wmrhr fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Log-incident query failed — red logs will no longer be ticketed until restart
      Npgsql.NpgsqlException (0x80004005): The operation has timed out
       ---> System.TimeoutException: The operation has timed out.
         at Npgsql.ThrowHelper.ThrowNpgsqlExceptionWithInnerTimeoutException(String message)
         at Npgsql.Util.NpgsqlTimeout.Check()
         at Npgsql.Util.NpgsqlTimeout.CheckAndGetTimeLeft()
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, S…[truncated]

_{Opened automatically from Admin/_LogIncident/45523b88fb120821. Recurrences are folded into this issue rather than opening new ones.}

Reopened — this fault is happening again.

Still occurring: 1 more since the last update (19 total).

Last seen: 2026-09-19 17:26:28Z
Pods: memex-portal-deployment-6cd5d8f887-2kcwk, memex-portal-deployment-64b5fbc96c-df8kr, memex-portal-deployment-7b5c857c97-cgdpw, memex-portal-deployment-7b5c857c97-zlc2b, memex-portal-deployment-b964b4c4c-wmrhr

Recent log lines

2026-08-23 20:20:43Z memex-portal-deployment-7b5c857c97-zlc2b fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Log-incident query failed — red logs will no longer be ticketed until restart
      Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.42.18.4:5432
       ---> System.TimeoutException: Timeout during connection attempt
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, String activityUserId, IReadOnlyCollection`1 excludedNodeTypes, CancellationToken ct)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs:line 1405
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, String userId, String basePath, S…[truncated]
2026-09-02 17:32:05Z memex-portal-deployment-b964b4c4c-wmrhr fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Log-incident query failed — red logs will no longer be ticketed until restart
      Npgsql.NpgsqlException (0x80004005): The operation has timed out
       ---> System.TimeoutException: The operation has timed out.
         at Npgsql.ThrowHelper.ThrowNpgsqlExceptionWithInnerTimeoutException(String message)
         at Npgsql.Util.NpgsqlTimeout.Check()
         at Npgsql.Util.NpgsqlTimeout.CheckAndGetTimeLeft()
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.PostgreSql.PostgreSqlStorageAdapter.QueryNodesInnerAsync(ParsedQuery query, JsonSerializerOptions options, S…[truncated]
2026-09-19 17:26:28Z memex-portal-deployment-6cd5d8f887-2kcwk fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Log-incident query failed (attempt 1) — nothing is ticketed while it is down; re-opening the watch in 00:00:01. Incidents keep being RECORDED throughout, so nothing is lost, only delayed.
      Npgsql.NpgsqlException (0x80004005): Exception while reading from stream
       ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
       ---> System.Net.Sockets.SocketException (104): Connection reset by peer
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow)
         at System.Net.Sockets.NetworkStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
         at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize)
         at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
         at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlReadBuffer.<Ensure>g__EnsureLong|54_0(NpgsqlReadBuffer buffer, Int32 count, Boolean async, Boolean readingNotifications)
         at Npgsql.Internal.NpgsqlReadBuffer.<Ensure>g__EnsureLong|54_0(NpgsqlReadBuffer buffer, Int32 count, Boolean async, Boolean readingNotifications)
         at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at Npgsql.NpgsqlDataReader.Read(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataReader.Read(Boolean async, CancellationToken cancellationToken)
         a…[truncated]

_{Re-addressed by the current identity function: this incident inherited 45523b88fb120821 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Closed — the filed defect is fixed, and production carries the proof

What was filed: the log-incident control plane permanently stopped ticketing red logs after one
failed Postgres query and needed a pod restart to recover. The component's own log line said so:
Log-incident query failed — red logs will no longer be ticketed until restart.

The fix

Systemorph/MeshWeaver.Plugins@2e1d498338d38d7962f30bf82f3c4a84e871c516 —
"fix(observability): a failed incident query re-opens the watch (MeshWeaver#3138)", 2026-09-02,
ancestor of origin/main (git merge-base --is-ancestor → 0). LogIncidentControlPlane.OpenIncidentWatch
replaced the terminal .Catch(… ⇒ Observable.Empty) — which completed the sequence — with a
Defer + RetryWhen that re-opens the query, unbounded in count and bounded in rate. The Defer
is load-bearing: it makes each attempt re-ask rather than re-attach to the faulted chain, which is
what lets GetQuery's faulted-chain eviction (#1316) hand back a new chain instead of the corpse.
onGiveUp remains the last line of defence rather than the policy.

Control: src/MeshWeaver.Observability.Test/IncidentWatchSurvivesAFailedQueryTest.cs, on a
TestScheduler with no clock and no mesh — a failed query re-opens the watch (pre-fix opens == 1
and the sequence completed having delivered nothing), a watch is never given up on however long the
store is down, and the backoff doubles to a one-minute ceiling.

Why the 2026-09-19 17:26:28Z recurrence is the fix WORKING, not the defect

This is the part that matters, because that sighting is what reopened the issue. Its log line is not
the filed message — it is the retry arm:

2026-09-19 17:26:28Z memex-portal-deployment-6cd5d8f887-2kcwk fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Log-incident query failed (attempt 1) — nothing is ticketed while it is down; re-opening the
      watch in 00:00:01. Incidents keep being RECORDED throughout, so nothing is lost, only delayed.

2e1d49833 is the commit that both deleted the filed template from production code and
introduced that string (git log -S on each, over LogIncidentControlPlane.cs, returns that one
commit). The old template survives in the tree only inside the test's XML docs, quoted as history.
So the presence of this exact line in a production log is proof that the fixed code is running and
that the recovery path was taken — the watch re-opened one second later. No inference about pod
restarts is needed for that reading.

The two older samples still retained on this incident (2026-08-23, 2026-09-02) carry the filed
template. The incident is therefore folding sightings across a code change, which is the whole
explanation for the reopen: the fingerprint keys on the category, the exception and the top frame —
PostgreSqlStorageAdapter.QueryNodesInnerAsync, an Npgsql connect timeout — and those are unchanged
by design, because the trigger is unchanged. Only the consequence was the defect, and the
consequence is what was fixed.

The residue, and where it is being handled

MeshWeaver.LogWatcher ingests fail:/crit: lines only (src/MeshWeaver.LogWatcher/Program.cs;
fail: is the console prefix for LogError). The retry arm reports at LogError, so every transient
database blip now mints an incident about the ticketing pipeline itself and reopens this issue —
a self-referential noise loop. A watch that recovers in one second having lost nothing is not an
incident; a watch that has not completed a query for many minutes is. That levelling is being changed
in MeshWeaver.Plugins (retry arm → Warning, plus one duration-based Error escalation per
outage so a genuinely long ticketing outage is still ticketed exactly once, and onGiveUp unchanged).
It is a severity/cost-model change with its own control, not a verbosity change.

If this issue is reopened again before that lands, the reopen is the retry notice, not the defect —
check the sample's message text first: (attempt N) … re-opening the watch in is the healthy arm.

Bearing on the 2026-09-19 10:34Z "the instrument is dark" reading

Related, and worth recording here because this issue is that instrument: the 10:34Z roll-gated
review across twelve bot-filed incidents could not establish "no recurrence since the roll" because
nothing anywhere had folded a sighting since 07:28:51Z. Ingestion has since resumed and drained its
backlog. Sightings on the current image (replicas up 08:34:53 / 08:39:58 / 08:39:59Z) have been
folded and reported for six fingerprints in that population — #3045 at 08:54:16Z, #3593 at 08:58:09Z,
#1246 at 08:59:06Z, #2480 at 08:59:41Z, #2307 at 09:03:41Z, and this one at 17:26:28Z, reported in
comments between 13:11Z and 17:27Z. The positive control that review asked for therefore now exists,
and a silent fingerprint is once again capable of meaning something — subject to its own trigger
regime having occurred (a teardown fault stays silent between rolls whatever its state).

_{Closed 2026-09-19 after verifying the fix commit, its control test, and the production log text.

Read-only on the mesh; every fact above is checkable from this repo, the Plugins tree, and this

issue's own comments.}

Addendum — the three messages, and which one would mean the defect is back

Re-verified against MeshWeaver.Plugins@origin/main = f6a8bd380 (my earlier reads were against a
checkout that turned out to be stale, so every fact in the close above was taken again):

git merge-base --is-ancestor 2e1d498338d3 origin/main → 0, still on main.
The retry-arm string re-opening the watch in → present, LogIncidentControlPlane.cs.
The filed template, Log-incident query failed — red logs will no longer be ticketed until restart
→ present in exactly one place on main, the XML docs of IncidentWatchSurvivesAFailedQueryTest,
quoted as history. No production path emits it.

One thing worth pinning, because it would read as a regression and is not one. The component now has
three messages, and two of them share the phrase the issue title is built from. A git grep for
that phrase alone will not tell them apart — and it will also miss the third, because the literal is
split across two C# strings ("… ticketed " + "until restart"), so the contiguous phrase does not appear
in the file at all. Match on the opening clause instead:

opening clause	arm	level	what it means
`Log-incident query failed (attempt N) —`	retry	`Error`	healthy. The watch re-opens after the stated delay; incidents keep being recorded throughout. This is what the 2026-09-19T17:26:28Z sighting was, and why this issue was closed.
`Log-incident watch could not be re-opened —`	give-up	`Error`	the real thing. `Catch` past the retry — the last line of defence, not the policy. If this appears, ticketing genuinely has stopped until restart and this issue's defect is back in substance, through a different door.
`Log-incident control plane stopped — the watch chain faulted.`	outer subscription	`Error`	the whole chain died rather than one query. Also genuine, and not what was filed.

So: a reopen carrying the first clause is the retry notice and should be read against the levelling
change described above. A reopen carrying the second or third is a real ticketing outage and
deserves its own investigation rather than being read as this one recurring — neither existed when this
issue was filed.

🚨 And a note on the instrument for anyone re-checking the above: git grep <pattern> origin/main -- 'src/**/*.cs' silently matches nothing (git does not glob ** in a pathspec without :(glob)), which reads exactly like "the string is gone". Use -- src, and run a positive control — a string you know is there — in the same command. My first attempt at the third bullet was a false negative from precisely that.

_{2026-09-19. Read-only; no state change (this issue stays closed).}

Navigation Menu

Repository navigation

Log-incident control plane permanently stops ticketing red logs after a Postgres connection timeout — requires restart to recover #3138

What is failing

Probable cause

Impact

Where to look

Evidence

Closed — the filed defect is fixed, and production carries the proof

The fix

Why the 2026-09-19 17:26:28Z recurrence is the fix WORKING, not the defect

The residue, and where it is being handled

Bearing on the 2026-09-19 10:34Z "the instrument is dark" reading

Addendum — the three messages, and which one would mean the defect is back

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Log-incident control plane permanently stops ticketing red logs after a Postgres connection timeout — requires restart to recover #3138

Description

What is failing

Probable cause

Impact

Where to look

Evidence

Activity

systemorph-com commented on Sep 19, 2026

rbuergi commented on Sep 19, 2026

Closed — the filed defect is fixed, and production carries the proof

The fix

Why the 2026-09-19 17:26:28Z recurrence is the fix WORKING, not the defect

The residue, and where it is being handled

Bearing on the 2026-09-19 10:34Z "the instrument is dark" reading

rbuergi commented on Sep 19, 2026

Addendum — the three messages, and which one would mean the defect is back

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions