Fix nightly Stress Tests (Chaos TCP): transparent reconnect + robust chaos assertions

[marcschier]

Description

Fixes the nightly Stress Tests / Chaos TCP job (e.g. run 29068846476), which had failed on every run since it was introduced — the workflow runs on a schedule and never gated PRs, so the failures went unnoticed.

Root cause

The Chaos TCP tests point a ConfiguredEndpoint at a local TcpChaosProxy and set UpdateBeforeConnect = false so the session connects through the proxy. Two problems prevented this from working:

1. ManagedSession.HandleConnectAsync ignored ConfiguredEndpoint.UpdateBeforeConnect and hard-coded a pre-connect re-discovery for all non-OpenAPI endpoints. Re-discovery adopted the server's advertised EndpointUrl, so the session connected directly to the server, bypassing the proxy — no chaos ever reached the connection. Two tests "passed" vacuously.
2. Once the proxy was genuinely engaged, the transparent-reconnect feature had a gap: a request that hit the just-dropped channel (before the manager detected the drop) failed with BadConnectionClosed instead of transparently recovering.

Changes

Product

• ManagedSession now honours ConfiguredEndpoint.UpdateBeforeConnect (default true). Setting it false opens the channel against exactly the supplied URL (proxy / gateway / NAT / pinned endpoint) without re-discovering.
• ManagedTransportChannelLease.SendRequestAsync transparently resends idempotent requests (Read / Browse / TranslateBrowsePaths / GetEndpoints / FindServers) across a channel-manager reconnect. On a transient transport-drop error it forces a coalesced reconnect and resends once the shared channel recovers.
  • A new ChannelEntry.ReconnectGeneration counter guards this so a reconnect is only forced for a genuinely undetected drop (same generation, still Ready), never for a stale in-flight failure that arrives after recovery — which previously spawned spurious extra reconnect cycles that tore down healthy channels.
  • Non-idempotent requests (Create*, Write, Call, Publish, …) surface the error so higher-level recovery (e.g. the subscription engine's own re-create loop) stays in control; requests are never double-applied.

Tests / harness

• TcpChaosProxy no longer rethrows the expected OperationAborted raised by an in-flight upstream ConnectAsync during DropAllConnections (it was faulting the awaited connection task and failing the test).
• Chaos reconnect assertions (L3-A1/A2/A5) are made robust to the inherent non-determinism of reconnect counts under concurrent chaos (coalescing merges drops landing in one recovery window; a stale keep-alive Bad can add one cycle). Survival is asserted via FailureRate, ReconnectFailed == 0, and the coalescing fan-out relationship rather than exact ReconnectStarted/Completed == dropCount.
• Subscription post-drop recovery window widened (2s → 8s) to allow re-creating all sessions' subscriptions and monitored items on a loaded CI agent.
• New unit tests in ClientChannelManagerManagedTests cover the new product paths (fast-PR / codecov, since the nightly ChaosTCP job does not run in the fast pipeline): idempotent transient-drop retry, non-idempotent no-retry, non-transient no-retry, and ChannelEntry.ReconnectGeneration increment.

Validation (local, net10.0)

• Chaos TCP: 6/6 pass across many random seeds (incl. post-merge re-runs of 338475986 and 1999999999).
• Opc.Ua.Client.Tests ChannelManager/ManagedSession: 256/256 (incl. the 4 new tests).
• Opc.Ua.Sessions.Tests ChannelManager/Reconnect: 139/139.
• Core / Client / Stress build with 0 warnings.

Notes

• This branch has been merged up to date with master (redundancy PR OPC UA Part 4 6.6 Redundancy (server + client) with opt-in distributed high-availability #3918).
• The two failing Opc.Ua.PubSub.Tests in CI — UdpLoopbackActionResponderAnswersRequesterAsync and UdpLoopbackDiscoveryPublisherAnswersSubscriberRequests — are unrelated to this PR (UDP-loopback request/response tests in the PubSub library, associated with OPC UA Part 4 6.6 Redundancy (server + client) with opt-in distributed high-availability #3918) and are timing/environment-flaky. They are out of scope for this change.

Related Issues

• Fixes the recurring nightly Chaos TCP failure (workflow: .github/workflows/stress-test.yml).

Checklist

• I have signed the CLA and read the CONTRIBUTING doc.
• I have added tests that prove my fix is effective or that my feature works and increased code coverage.
• I have added all necessary documentation.
• I have verified that my changes do not introduce (new) build or analyzer warnings.
• I ran all tests locally using the UA.slnx solution against at least .net framework and .net 10, and all passed.
• I fixed all failing and flaky tests in the CI pipelines and all CodeQL warnings.
• I have addressed all PR feedback received.

[codecov]

Codecov Report

❌ Patch coverage is 85.18519% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.36%. Comparing base (7f40965) to head (ec25121).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
.../Channels/Internal/ManagedTransportChannelLease.cs	83.33%	4 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3986      +/-   ##
==========================================
- Coverage   77.70%   77.36%   -0.34%     
==========================================
  Files        1385     1385              
  Lines      183248   183293      +45     
  Branches    31926    31932       +6     
==========================================
- Hits       142385   141799     -586     
- Misses      29318    30000     +682     
+ Partials    11545    11494      -51     

Files with missing lines	Coverage Δ
Libraries/Opc.Ua.Client/Session/ManagedSession.cs	63.04% <100.00%> (+0.04%)	⬆️
...ore/Stack/Client/Channels/Internal/ChannelEntry.cs	77.34% <100.00%> (+0.11%)	⬆️
.../Channels/Internal/ManagedTransportChannelLease.cs	69.17% <83.33%> (+5.36%)	⬆️

... and 46 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Fixes the nightly Stress Tests / Chaos TCP suite by ensuring sessions truly connect through the chaos proxy (so chaos is actually applied), and by hardening the transparent reconnect logic + test assertions against non-deterministic reconnect counts under concurrent drops.

Changes:

• Honor ConfiguredEndpoint.UpdateBeforeConnect in ManagedSession so pinned/proxied URLs aren’t overwritten by discovery.
• Add idempotent-request retry across transient channel drops in ManagedTransportChannelLease.SendRequestAsync, guarded by a new ChannelEntry.ReconnectGeneration.
• Update Chaos TCP stress tests and harness to avoid false failures: swallow expected proxy teardown exceptions, relax reconnect-count assertions, and widen recovery polling window.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
Tests/Opc.Ua.Stress.Tests/Channels/Fakes/TcpChaosProxy.cs	Swallows expected connection teardown exceptions so chaos-induced aborts don’t fault the test harness.
Tests/Opc.Ua.Stress.Tests/Channels/Chaos/TransparentReconnectChaosTests.cs	Makes reconnect assertions robust to coalescing/non-deterministic reconnect counts; improves failure diagnostics.
Tests/Opc.Ua.Stress.Tests/Channels/Chaos/SubscriptionSurvivalChaosTests.cs	Extends post-drop recovery wait to reduce CI flakiness under load.
Tests/Opc.Ua.Stress.Tests/Channels/Chaos/BlockAcceptChaosTests.cs	Adjusts reconnect assertions to assert exercised+no-failures rather than exact counts.
Stack/Opc.Ua.Core/Stack/Client/Channels/Internal/ManagedTransportChannelLease.cs	Adds bounded retry for idempotent requests across transient drops and coalesced reconnects.
Stack/Opc.Ua.Core/Stack/Client/Channels/Internal/ChannelEntry.cs	Introduces ReconnectGeneration counter to detect stale in-flight failures vs. undetected drops.
Libraries/Opc.Ua.Client/Session/ManagedSession.cs	Computes updateBeforeConnect based on ConfiguredEndpoint.UpdateBeforeConnect (and OpenAPI exceptions).