Skip to content

Postgres health check Unhealthy: TLS handshake to PostgreSQL times out on connector open (duplicate of #5314) #5347

Closed as duplicate

Description

@systemorph-com

What is failing

The PostgreSql health check on the memex-cloud portal reported Unhealthy: opening a new Npgsql connector stalled in TLS negotiation (TaskCanceledException in SslStream.ForceAuthenticationAsync → NpgsqlConnector.NegotiateEncryption) and the check's cancellation token fired at ~6.5s.

Probable cause

Medium confidence — a transient network/TLS handshake delay to the PostgreSQL host exceeded the health check's timeout budget on the first pooled connector open of that window. The stack runs entirely through Npgsql and the HealthChecks.NpgSql library; no application frame is implicated.

Already known — likely duplicate

This is the same defect as MeshWeaver#5314 (incident Admin/_LogIncident/b2cc907bf65c0478): same log site, exception type and top frame, same pod memex-portal-deployment-5c444645f8-8pdzb, and this occurrence (14:44:41) falls two minutes after that incident's last sample (14:42:39). This ticket can likely be closed as a duplicate, but a human should confirm the burst at 13:42–14:44 UTC is fully covered by the earlier issue.

Impact

Single occurrence on a single portal pod; at most a momentary readiness blip, no user-facing failure evidenced.

Where to look

  • HealthChecks.NpgSql.NpgSqlHealthCheck.CheckHealthAsync (AspNetCore.Diagnostics.HealthChecks) — health check entry point.
  • Npgsql connector open path: Npgsql.Internal.NpgsqlConnector.Open / NegotiateEncryption.
  • Pod→Postgres network/TLS latency around 2026-09-22 14:44 UTC; consider health check timeout vs. NpgsqlTimeout.

Evidence

Fingerprint c9defbf7ce85d8f9
Category Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService
Severity Error
Exception System.Threading.Tasks.TaskCanceledException
Top frame HealthChecks.NpgSql.NpgSqlHealthCheck.CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken)
Namespace memex-cloud
Pods memex-portal-deployment-5c444645f8-8pdzb
Occurrences 1
First seen 2026-09-22 14:44:41Z
Last seen 2026-09-22 14:44:41Z
Routing not determined — no configured route matches the category Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines
2026-09-22 14:44:41Z memex-portal-deployment-5c444645f8-8pdzb fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check PostgreSql with status Unhealthy completed after 6455.52ms with message 'A task was canceled.'
      System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize)
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
         at System.Net.Security.SslStream.ReceiveHandshakeFrameAsync[TIOAdapter](CancellationToken cancellationToken)
         at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.NegotiateEncryption(SslMode sslMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.SetupEncryption(SslMode sslMode, GssEncryptionMode gssEncryptionMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         a…[truncated]

Opened automatically from Admin/_LogIncident/c9defbf7ce85d8f9. Recurrences are folded into this issue rather than opening new ones.

Activity

  1. added
    bugSomething isn't working
    sev:LLow - cosmetic, rare edge case, or developer-only
    on Sep 22, 2026
  2. rbuergi commented on Sep 23, 2026

    @rbuergi
    Contributor

    Duplicate of #5314. Confirmed from the code site and discriminator, not from the title alone.

    • Same log site: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103], Health check PostgreSql with status Unhealthy … 'A task was canceled.'
    • Same code path and discriminator: TaskCanceledException raised inside the TLS handshake of a new pooled connector: NpgSqlHealthCheck.CheckHealthAsync → NpgsqlConnection.Open → PoolingDataSource.OpenNewConnector → NpgsqlConnector.NegotiateEncryption → SslStream.ForceAuthenticationAsync. Every frame from ForceAuthenticationAsync outward is identical to Postgres health check fails intermittently: TLS handshake to PostgreSQL times out on new connector open #5314's.
    • Why the fingerprint differs: the incident fingerprint includes the innermost frame. Here the cancellation landed two frames deeper in the same handshake (SslStream.ReceiveHandshakeFrameAsync → EnsureFullTlsFrameAsync), which gives c9defbf7ce85d8f9 instead of b2cc907bf65c0478. That is a different moment in one handshake, not a different defect.
    • Same pod and burst: memex-portal-deployment-5c444645f8-8pdzb at 14:44:41Z. Postgres health check fails intermittently: TLS handshake to PostgreSQL times out on new connector open #5314's third sample is on the same pod at 14:42:39Z.

    The burst at 13:42–14:44Z on 2026-09-22 is fully covered by #5314. It has samples at 13:42:03 and 13:47:08 on 7cc85f47c-nm2qp and at 14:42:39 on 5c444645f8-8pdzb, and this one at 14:44:41 is the fourth. Analysis continues on #5314.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingsev:LLow - cosmetic, rare edge case, or developer-only

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions