OPC UA Part 4 6.6 Redundancy (server + client) with opt-in distributed high-availability - #3918
Merged
Merged
Conversation
…ng with hardened shared store Adds an opt-in provider model (Opc.Ua.Server.Distributed) to replicate address-space topology/values and session state across server replicas for active/passive and active/active HA, exposed via documented OPC UA redundancy (ServiceLevel/RedundantServerArray). The single-instance in-memory path stays zero-overhead (default NullRecordProtector). Building blocks: ISharedKeyValueStore (+in-memory), INodeStateStore, address-space synchronizer, leader election (static + shared-store lease CAS), service-level providers, distributed value cache, shared session store. Wired via IServerStartupTask hosting seam + fluent DI (UseDistributedAddressSpace / AddServerServiceLevel / redundancy options). CustomNodeManager2 opts in via ILocalAddressSpaceSource. Security hardening (plan 30, SDL + IEC 62443): IRecordProtector + AesCbcHmacRecordProtector (AES-256-CBC Encrypt-then-MAC, verify-before-decrypt, fail-closed) on every shared record; KeyRingRecordProtector (staged key rotation); SharedSingleUseNonceRegistry (cross-replica single-use server nonce via store CAS); key zeroization. Docs/HighAvailabilitySecurity.md captures the STRIDE threat model and operator guidance. Docs: HighAvailability.md, HighAvailabilitySecurity.md, Docs/README link, HighAvailabilityServer sample. Plans 28/29/30. 85 Distributed unit tests (net10.0 + net48).
…reconnect (S5) Implements the opt-in mirrored fast-reconnect from plans 30/31. After a failover a client reconnects to a standby by re-running ActivateSession; the AuthenticationToken is only a lookup key and the standby still performs the full client-certificate signature validation against a mirrored, single-use serverNonce (token-only hijack and nonce replay are both closed). - SharedSessionEntry: extended with the full reconstruction state (serverNonce, clientNonce, client cert blob, SecurityPolicy/Mode, endpoint, timeout, client description); encrypted at rest via the record protector. - SessionManager: additive, backward-compatible RestoreSessionAsync + SupportsSessionRestore seam in the ActivateSession miss-path (default returns null => unchanged behaviour; m_sessions stays private). - DistributedSessionManager: mirrors encrypted session state on create/activate, removes on close; on restore enforces REQ-UA-7 (same SecurityPolicy/Mode), consumes the serverNonce single-use across the replica set (replay defence), reconstructs the session, and logs provenance with a one-way token digest. - ISessionManagerFactory seam on StandardServer (wired from DI by the hosted service, supplying a server-certificate provider) + DistributedSessionManagerFactory + UseDistributedSessions(...) fluent API. Safe default EnableFastReconnect=false (re-auth on failover). - Docs (HighAvailability.md, HighAvailabilitySecurity.md) updated. Tests: 97 Distributed (net10+net48); no regression (61 Session + 57 client SessionTests integration pass). Remaining: two-server network e2e (S6).
…roring (S6) DistributedSessionMirrorIntegrationTests stands up a fully-started server whose ISessionManagerFactory builds a DistributedSessionManager, and verifies end-to-end that a session created/activated through the real service handlers is mirrored encrypted to the shared store (a wrong-key reader fails closed) and removed on close. This closes the factory -> StandardServer.CreateSessionManager -> mirror runtime-wiring gap. The full secured two-server token-reuse reconnect happy-path remains a documented follow-up (the stack client does re-auth-on-failover; the direct-service helper only drives unsecured sessions). The restore decision logic (REQ-UA-7 + single-use nonce/replay) is unit-tested and the base ActivateSession signature path is integration-tested. 98 Distributed tests pass (net10 + net48).
…t (F6/F9) Workstream A from plans/32: - F6: SharedKeyValueSessionStore keys entries by the SHA-256 digest of the AuthenticationToken (SharedKeyValueSessionStore.KeyFor) instead of the raw token, so a backend's key enumeration/monitoring/dumps never expose the token. Test asserts the keyspace contains no raw token. - F9: a successful cross-replica restore emits a distinct AuditSessionEventState (Session/RestoredFromSharedStore) via the new IAuditEventServer.ReportAuditSessionRestoredEvent, in addition to the standard AuditActivateSession, with a one-way token digest for provenance. - F7: analyzed — the decrypted serverNonce becomes the session's retained working Nonce (no extra plaintext copy in the manager); Nonce.Data zeroization on dispose is a pre-existing server-wide Core concern tracked separately. Docs updated. 99 Distributed tests pass (net10 + net48).
…ng in sample (FG) Workstream FG from plans/32: - New Docs/KubernetesDeployment.md: worked replicaset deployment (StatefulSet + headless Service, leader election, readiness tied to ServiceLevel, KEK + shared ApplicationInstanceCertificate provisioning via Secrets, security checklist). Linked from Docs/README.md and HighAvailability.md. - HighAvailabilityServer sample: wires UseDistributedSessions (opt-in HA_FAST_RECONNECT) and an optional AesCbcHmacRecordProtector from a base64 HA_RECORD_KEY (encrypted shared store), demonstrating the production-hygiene pattern. - Transparent redundancy remains a documented deployment pattern (single virtual endpoint + shared session store + subscription transfer; no new transport).
…se failover) design
…A-13, B) On failover to a redundant server, when EnableTokenReuseFailover is set, the client re-activates the existing session by reusing the current AuthenticationToken (signing over the new channel + last serverNonce) instead of CreateSession, falling back to re-authentication if the standby rejects the token. - Session.cs: extracted ReactivateExistingSessionAsync (the token-reuse activation core, shared with UpdateSessionAsync); RecreateInPlaceCoreAsync tries it first (adopting the failover server's cert) before the existing clear + fresh-CreateSession fallback; new EnableTokenReuseFailover property (copied across recreate clones). OpenAsync is untouched. - ManagedSession: EnableTokenReuseFailover option threaded through CreateAsync + ctor and applied to the inner session; ManagedSessionOptions.EnableTokenReuseFailover; ManagedSessionBuilder.WithTokenReuseFailover(). Default off (re-auth on failover). No regression: 57 client SessionTests + 135 reconnect/failover integration tests pass (net10). Updated the ctor-reflection test for the new parameter.
DistributedSessionFailoverIntegrationTests: two secured servers share one store via DistributedSessionManager; a ManagedSession client with WithTokenReuseFailover fails over from the active to the standby. The standby restores the mirrored session and re-activates it with the reused AuthenticationToken, so the client's SessionId is preserved (a fresh re-authentication would change it). Passes on net10 + net48. Docs (HighAvailability.md, HighAvailabilitySecurity.md) updated: client WithTokenReuseFailover() usage + the e2e validation; plans/32 marks B and C done.
marcschier
commented
Jun 26, 2026
Split all distributed/HA implementation into a new OPCFoundation.NetStandard.Opc.Ua.Server.Distributed project that references Opc.Ua.Server; core now keeps only the seams. - Move ILocalAddressSpace/ILocalAddressSpaceSource to the Opc.Ua.Server namespace (NodeManager/); extract a shared internal PredefinedNodesAddressSpace reused by both CustomNodeManager2 and AsyncCustomNodeManager (both now implement ILocalAddressSpaceSource). Apply path is async (no sync-over-async). - Remove the node-state-store-registry hook from ServerInternalData; the distributed startup task owns its own registry. - Add CryptoUtils.ZeroMemory/FixedTimeEquals polyfills and reuse them in the record protector, EncryptedSecret, and KeyCredentialBridgeAuthenticator. - Fix CA2213 in AddressSpaceSynchronizer (dispose enumerator in DisposeAsync). - Rebase HighAvailabilityServer sample onto AsyncCustomNodeManager; expand the sample README with shared-store wiring and an active/active setup. - Strip internal tracking tags (REQ-UA/Finding/IEC/SDL) from code and XML docs. - Merge HighAvailabilitySecurity.md into HighAvailability.md and fix links. - Wire the new project into UA.slnx, the test project, and the sample.
Add a separate net8.0+ package providing true active/active (multi-writer) replication for the distributed server, built on the Crdt + Crdt.Transport NuGet packages. Opt-in; the base distributed library and its active/passive path are unchanged. - New Libraries/Opc.Ua.Server.Distributed.Crdt project (net8.0;net9.0;net10.0, no-op shell on legacy CI legs via RestrictForLegacyTfm); references the base distributed project + Crdt + Crdt.Transport. - Address space A/A: CrdtAddressSpaceSynchronizer models topology and values as last-writer-wins maps and gossips state over Crdt.Transport (in-memory / TCP / UDP, optional TLS); every replica writes, received state is merged and applied. A topology merge preserves the locally-known value so it never regresses a concurrently-updated value (values are versioned by their own entries). - Sessions A/A: CrdtSharedKeyValueStore replicates encrypted session entries by gossip and reuses DistributedSessionManager; the single-use server nonce stays on a strongly-consistent ISingleUseNonceRegistry (CRDTs cannot enforce exactly-once), and the CRDT store rejects compare-and-swap. - Fluent opt-in: UseCrdtAddressSpace(...) / UseCrdtSessions(...) with a shared CrdtGossipOptions base (ReplicaId, UseTcpGossip/UseUdpGossip/AddPeer, limits). - Tests: new Opc.Ua.Server.Distributed.Crdt.Tests (net8/net10) — convergence, multi-writer, concurrent-value LWW, serializer round-trip, CAS/Watch boundary; plus a CRDT exercise in the AOT test project. - Docs: HighAvailability.md 'Active/active with CRDTs' section incl. the single-use-nonce security boundary; Crdt/Crdt.Transport added to CPM. Crdt/Crdt.Transport 1.0.0 (MIT, NativeAOT-ready) restore from nuget.org.
marcschier
commented
Jun 26, 2026
marcschier
commented
Jun 26, 2026
marcschier
commented
Jun 26, 2026
…mple, clarify HA docs - Rename the user-facing fluent surface from Crdt* to Replicated*: UseReplicatedAddressSpace / UseReplicatedSessions, ReplicatedAddressSpaceOptions / ReplicatedSessionOptions / ReplicatedGossipOptions, ReplicatedServerBuilderExtensions (implementation classes and the Crdt package/namespace keep their names). - Integrate the CRDT package into the HighAvailabilityServer sample with an HA_MODE switch (ap = active/passive leader-write, aa = active/active CRDT gossip via HA_GOSSIP_PORT / HA_GOSSIP_PEERS); document running two AA replicas in the sample README. - HighAvailability.md: reference the implemented CRDT library (not 'deferred'), document that a write to a non-leader replica is discarded in active/passive, and rewrite the value-participation note to describe how participation works. Addresses review comments on Docs/HighAvailability.md and the sample README.
…Ua.Server.Distributed.Tests Organize the source files of both Opc.Ua.Server.Distributed and Opc.Ua.Server.Distributed.Crdt into logical subfolders (AddressSpace, KeyValueStore, Redundancy, Values, Sessions, Security for the base library; AddressSpace, Sessions for the CRDT adapter), with the top-level DI entry points kept at the project root. Namespaces are unchanged. Mirror the organization in the test projects: - Extract the base distributed tests from Opc.Ua.Server.Tests/Distributed into a dedicated Opc.Ua.Server.Distributed.Tests project (mirroring the src project), with the same subfolders; add it to UA.slnx and InternalsVisibleTo. Test namespaces are unchanged. - Organize Opc.Ua.Server.Distributed.Crdt.Tests into AddressSpace/Sessions subfolders mirroring the CRDT src. Addresses review comments asking to mirror the test projects to the src side and to organize the files logically without changing namespaces.
marcschier
changed the title
Add unit tests for the previously-uncovered CRDT adapter surface flagged by codecov (patch coverage 67.65%): - ReplicatedGossipOptions/ReplicatedSessionOptions: defaults, UseTcp/UseUdp transport factory wiring, AddPeer, CreateReaderOptions/CreateTransport, null-arg guards. - CrdtAddressSpaceStartupTask: attaches a synchronizer to opted-in (ILocalAddressSpaceSource) node managers and skips the rest; null-arg guards. - CrdtSessionManagerFactory + UseReplicatedSessions/UseReplicatedAddressSpace registration; null-arg guards. - ByteStringCrdtSerializer JSON round-trip (null/empty/data); CrdtSharedKeyValueStore.ScanAsync prefix filtering. CRDT package line coverage 67% -> 93.7%; 25/25 tests pass on net8.0 and net10.0.
Collaborator
Author
|
Addressed the codecov patch-coverage feedback in 4f4d7ed. The 80% patch gap was concentrated in the new
CRDT package line coverage rises from ~67% to 93.7%; 25/25 tests pass on net8.0 and net10.0. Codecov will re-evaluate the patch on this commit. |
…t + non-transparent Server model: per-mode Server.ServerRedundancy (RedundancySupport, RedundantServerArray, non-transparent ServerUriArray, transparent CurrentServerId); sub-range ServiceLevel providers (Table 105) + load balancing; RequestServerStateChange/Maintenance/EstimatedReturnTime (6.6.5, admin-gated); NTRS capability; FindServers returns the RedundantServerSet (IRedundantServerSetProvider). New shared Opc.Ua.ServiceLevels/ServiceLevelSubrange in core. Client: RedundantManagedClient realizing all Table 107 modes (Cold/Warm/Hot(a)/Hot(b)/HotAndMirrored); RedundancySupport wording; ServiceLevel sub-range failover rules; FindServers ServerUri->endpoint resolution (no security downgrade); client redundancy via TransferSubscriptions; non-transparent network redundancy. State mirroring (opt-in provider seams; single-instance unchanged): session takeover, subscription-definition mirror, async notification sequence/Republish mirror, best-effort continuation-point envelope, deterministic EventId provider, RegisterNodes replica-consistent. Extensions: Opc.Ua.Server.Distributed.Crdt (active/active) and Opc.Ua.Server.Distributed.Kubernetes (Lease election, peer discovery, ServiceLevel->readiness). Samples (HighAvailabilityServer, RedundantClient), docs mapped to 6.6, conformance tests. Security: AES-CBC+HMAC record protection, fail-closed RecordProtectionGuard for external stores (base + CRDT session paths), single-use server-nonce replay protection, K8s TLS hostname validation. Validated on net10 and net48.
… review findings CRDT active/active extension (Opc.Ua.Server.Distributed.Crdt): - Bump Crdt/Crdt.Transport 1.0.0 -> 1.0.2 (now ship netstandard2.0/2.1 assets) - Widen TFMs from net8.0+ to $(LibTargetFrameworks) (net472/net48/netstandard2.1/net8/9/10); remove RestrictForLegacyTfm; gate IsAotCompatible/binding-gen to net8+ - Fail closed for unauthenticated networked gossip (TCP without mutual TLS, UDP); add AllowUnauthenticatedGossip opt-out for dev/test; gossip-port NetworkPolicy guidance Server review findings: - AddServerRedundancy warns when non-transparent mode has no IServiceLevelProvider - Rename AddManualFailover -> AddRequestServerStateChange ([Obsolete] shim retained) - Consolidate ServerRedundancyOptions peer inputs (RedundantPeers canonical) - DeterministicEventId: exclude per-replica ReceiveTime, compute after distinguishing fields are populated (replica-stable per 6.6.2.2) - Subscription retransmission mirror: delta path, namespace/server tables once per subscription, bounded-parallel drain Client review findings: - Maintenance(0) fails over to a healthy peer (Table 105); EstimatedReturnTime backoff - Warm backups use MonitoringMode.Disabled until failover (Table 107) - Hot(b) dedup uses exact value identity (no hash-collision drops); value-struct key - Subscription-template ownership: client disposes retained templates - De-duplicate WithNetworkRedundancy registration Validated: 0 warnings; tests green on net10 + net48 (CRDT 30, Distributed 178, Client redundancy 75, InformationModel redundancy 7).
…eview feedback Rename (full identity: folders, csproj, AssemblyName, PackageId, RootNamespace, C# namespaces, references, UA.slnx, InternalsVisibleTo): - Opc.Ua.Server.Distributed -> Opc.Ua.Server.Redundancy - Opc.Ua.Server.Distributed.Crdt -> Opc.Ua.Server.Redundancy.Crdt - Opc.Ua.Server.Distributed.Kubernetes -> Opc.Ua.Server.Redundancy.K8s (identity only; the word "Kubernetes" and k8s client types are preserved in prose/code) - Tests mirror the rename (.Tests/.Crdt.Tests/.K8s.Tests/.Integration.Tests) - Application HighAvailabilityServer -> RedundantServer - Documentation updated to the new names (present tense, merged-to-master state) Review feedback (verified previously-resolved PR comments are satisfied; fixed gaps): - async node-manager base, A/A sample README, REQ-UA tags, CA2213, central CryptoUtils polyfills, ILocalAddressSpace namespace - all confirmed in current code Roadmap findings implemented: - OPCFoundation#28 ServerUriArray-only peers are now failover candidates (connect + read live level) - #8 FetchRedundancyInfo follow-up reads merged into one ReadValuesAsync - OPCFoundation#29 ContinuationPoint mirroring documented as envelope-only (partial-SHALL boundary) - OPCFoundation#30 Server.ServerRedundancy typed to NonTransparent/Transparent subtype by mode - OPCFoundation#20 Add* (standard) vs Use* (extension) convention documented + ServiceLevel cross-ref - OPCFoundation#24 transparent-mode shared application-key blast-radius/rotation guidance expanded CI green (fixes pre-existing all-TFM build break, unrelated to the rename): - Opc.Ua.Sessions.Tests redundancy test doubles: implement IServerRedundancyHandler. ShouldFailover and use RedundancySupport (RedundancyMode was removed) - Integration tests: add RestrictForLegacyTfm + CustomTestTarget fallback so legacy TFM CI passes no-op instead of failing with empty TargetFrameworks - Integration Maintenance assertion updated to spec-aligned behavior (Maintenance with an available peer warrants failover, Part 4 Table 105) Validated: full UA.slnx builds on net10 + net48; Redundancy 178, Crdt 30, K8s 27, Integration 3, Client redundancy 76, Sessions failover 4 - all pass.
The private ManagedSession constructor gained enableTokenReuseFailover (bool) and networkRedundancy (NetworkRedundancyOptions?) parameters during the network-redundancy work, but two reflection-based test helpers still used the old signature, so GetConstructor returned null and failed the fixture SetUp (52 cascading failures = the test-ubuntu-latest-Client / Fast PR test CI failures). - ManagedSessionComplianceTests.CreateManagedSessionWithInner: add the 4th bool + NetworkRedundancyOptions to the ctor type list and invoke args - ManagedSessionTests: add NetworkRedundancyOptions to the ctor type list and invoke args Validated: full Opc.Ua.Client.Tests now 1530 passed / 0 failed on net10.
The legacy-TFM no-op shell strips all references, so on netstandard2.0/2.1 there is no
assembly providing System.Object and the empty compile failed with CS8021 ("No value
for RuntimeMetadataVersion"). net4x builds got System.Object from the implicit targeting
pack so they worked. Supply an explicit RuntimeMetadataVersion for the no-op shell so the
empty assembly compiles on every legacy TFM.
This was the netstandard2.0 leg of the build-*-all-tfm failure; it affected every
RestrictForLegacyTfm project (Core.Diagnostics, Network.Fuzz*, Redundancy.K8s*,
Redundancy.Integration.Tests, RedundantServer, RedundantClient, Mcp, Minimal* samples).
Validated: full UA.slnx builds on net10, net48, and netstandard2.0; the no-op shell
compiles cleanly on net472/net48/netstandard2.0/netstandard2.1.
marcschier
changed the title
… from codecov The new redundancy libraries are ~80% unit-covered, but codecov/patch was dragged down by genuinely-untestable Kubernetes IO (real HTTP to the API server, the readiness HTTP listener, and cluster-bound startup tasks). - Add KubernetesServerBuilderExtensionsTests: a DI-registration test that exercises UseKubernetes / UseKubernetesLeaderElection / UseKubernetesPeerDiscovery / UseKubernetesReadiness (the previously 0%-covered builder wiring) plus null-builder guards, resolving the registered services out-of-cluster with no IO. - codecov.yml: ignore the four integration-only K8s IO files (KubernetesHttpApiClient, KubernetesReadinessServer, KubernetesReadinessStartupTask, KubernetesPeerDiscoveryStartupTask) - mirrors the existing Applications/** exemption; the Kubernetes logic (models, factory, lease election, peer parsing, readiness mapping, builder wiring) stays measured. Measured redundancy-library line coverage (codecov view) is now ~88%, above the 80% patch target. K8s tests: 29 passed.
…Lock ambiguity) The net10-only RedundantServer sample used RestrictForLegacyTfm, which only no-ops legacy TFMs (net4x/ns2.x). Under the Linux all-TFM CI leg (CustomTestTarget=net8.0) the app still built net10 while referencing the net8-built Opc.Ua.Types, so its System.Threading.Lock polyfill collided with net10's BCL Lock (CS0433) in HaSampleNodeManager. Follow CustomTestTarget like the libraries so the app's framework references match the TFM being built (net8 app + net8 types => only the polyfill Lock). Validated: RedundantServer builds on net8.0 and net10.0.
Core.Diagnostics.Tests builds net8/net9/net10 and had an unconditional build-ordering ProjectReference to the net10-only McpServer (so its assembly exists for the reflective McpServer tests). On the Linux all-TFM CI legs (CustomTestTarget=net8.0/net9.0) the solution build forced McpServer to the leg TFM, which it cannot target (its own deps build net8/net9), failing the UA.slnx build. - Reference McpServer only when building net10 (CustomTestTarget '' or net10.0). - Make both reflective LoadMcpAssembly helpers Assert.Ignore (skip) when the net10 Opc.Ua.Mcp assembly is absent, instead of asserting it exists - so the net8/net9 test legs skip the MCP reflective tests cleanly while net10 still runs them. Pre-existing infra issue (unrelated to the redundancy feature) surfaced once the RedundantServer net8 Lock fix let the Linux all-TFM build progress past net8. Validated: full UA.slnx builds on net8.0/net9.0/net10.0; Core.Diagnostics McpServer tests pass on net10 (13) and skip/pass without failure on net8.
marcschier
commented
Jun 28, 2026
- IDE0300 collection expression in SessionCoverageTests - IDE0062 static local function + RCS0021 brace placement in TcpListenerChannelTests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ix RCS1197 StringBuilder.AppendJoin was added in netstandard2.1 / .NET Core 2.1 and is absent on net472/net48/netstandard2.0. Add AppendJoin extension overloads (string/char separator; IEnumerable<T>, params object[], params string[]) to the System.Polyfills class so the RCS1197 fix compiles uniformly on all TFMs. Apply the RCS1197 fix at ClientChannelManagerManagedTests (Append(string.Join) -> AppendJoin) which now resolves to the polyfill on net48 and the BCL instance method on modern TFMs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
These sites were previously assumed net48-incompatible, but investigation shows RCS1249 fired on all TFMs (net472..net10): the ! is redundant everywhere. On net48/net472 nullable warnings are suppressed (common.props NoWarn=nullable) so removal is oblivious-safe; on net8/9/10 the analyzer confirms the operand is provably non-null. The fixer precisely removed only the redundant argument- position ! (keeping needed receiver ! in SubscriptionTests). Verified building all four projects on net10 (strict) and net48. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The RCS1249 null-forgiving-operator cleanup wrapped the '?? throw' expression in a (string?) cast, which reintroduced nullability and failed the build with CS8603 (possible null reference return) on the non-nullable Task<string> method. Remove the redundant cast; the '?? throw' already guarantees a non-null result.
…ndation#3981) Expose Server.ServerRedundancy as the correct generated subtype (TransparentRedundancyState / NonTransparentRedundancyState / ServerRedundancyState) for the configured RedundancySupport mode, with mode-specific children (CurrentServerId / ServerUriArray) coming from the generated model instead of hand-written PropertyState construction. No hardcoded numeric redundancy type NodeIds remain. - Add generic core primitive AsyncCustomNodeManager.ReplacePredefinedInstanceSubtypeAsync (+ IDiagnosticsNodeManager) that swaps a predefined instance node for a differently-typed instance, preserving identity, well-known child NodeIds and values, and emitting a ModelChange. - Add ServerRedundancyController (IServerRedundancyController) with runtime ChangeModeAsync + RedundancySupport->ObjectTypeIds mapping; ServerRedundancyStartupTask now delegates to it. - Simplify DiagnosticsNodeManager: drop hand-written CurrentServerId/ServerUriArray helpers and dynamic TypeDefinitionId mutation. - Tests: subtype/browse-model assertions incl. runtime mode change; core primitive unit tests.
The idiom-polish analyzer sweep removed two semantically required else branches from DtlsRecordProtection. The AEAD seal path consequently fell through into the integrity-only path, overwriting ciphertext and its AEAD tag with plaintext plus HMAC. The AEAD open path likewise decrypted the record and then incorrectly revalidated the AEAD tag as an HMAC. Restore the mutually exclusive AEAD and integrity-only paths. This fixes the aot-ubuntu-latest ProtectsDtlsRecord_AotSafe failure and the related PubSub.Udp DTLS round-trip/replay failures.
Guard the two control-flow sites changed by the formatter sweep in MinMaxAggregateCalculator. Cover Minimum/Maximum and Minimum2/Maximum2 in both ActualTime mode (selected raw-sample timestamp) and plain mode (processing-slice timestamp), asserting exact source/server timestamps and rejecting the alternative timestamp source.
Merged
7 tasks
SksKeyResponse stamped unpacked keys with DateTime.UtcNow, ignoring the TimeProvider injected into pull/push providers. Under FakeTimeProvider, new keys were immediately due again. Once the refresh lead time was reached, ComputeNextDelay returned zero and the background task entered a synchronous refresh loop inside FakeTimeProvider.Advance, hanging the entire PubSub test process. Unpack keys with the provider's clock and throttle already-due refreshes with a positive delay. Add a direct issue-timestamp regression test.
…PubSub samples to ghcr (OPCFoundation#3984) - Add Dockerfiles for ConsoleLdsServer, MinimalBoilerServer, MinimalCalcServer, PumpDeviceIntegrationServer, McpServer (aspnet base for HTTP/SSE), and ConsoleReferencePubSubClient (root-context build, net10.0, PublishAot=false). - Rewrite .github/workflows/docker-image.yml into a matrix that builds and pushes all 10 sample images (refserver, ldsserver, boilerserver, calcserver, pumpserver, mcpserver, redundantserver, redundantclient, redundantpubsub, pubsubclient) to ghcr on the existing ubuntu-latest runner, with per-image gha build cache scopes and fail-fast disabled. - Document the additional images in Docs/ContainerReferenceServer.md.
…ment (OPCFoundation#3981) - Move ReplacePredefinedInstanceSubtypeAsync into its own capability interface IPredefinedNodeSubtypeReplacer implemented by AsyncCustomNodeManager; remove it from IDiagnosticsNodeManager. ServerRedundancyController now resolves the capability via 'server.DiagnosticsNodeManager as IPredefinedNodeSubtypeReplacer'. - Document the capability (usage example, when/when-not to use, reuse candidates, and the fluent-API rationale) in Docs/CoreNodeManagerVsCustomNodeManager2.md. - Trim the user-facing ServerRedundancy subtype paragraph and reword the runtime paragraph to focus on IServerRedundancyController.ChangeModeAsync in Docs/HighAvailability.md.
…ype-instances Model Server.ServerRedundancy via generated subtype instances (OPCFoundation#3981)
This was
linked to
issues
Jul 10, 2026
|
|
ADO restores/builds with CustomTestTarget=net48, then invokes dotnet test -f net48 without that property. The previous placeholder mapped legacy builds to net8.0, so the later test failed with NETSDK1005 and emitted no TRX. Expose an explicit net48 placeholder target with test infrastructure while keeping real sample process tests and app references net10-only.
marcschier
commented
Jul 10, 2026
Address PR review feedback by removing the pumpserver image entry from the Docker Sample Images CI workflow.
The explicit net48 placeholder now produced a TRX, but the ADO evaluator rejects runs with total=0. Compile one net48-only placeholder test so the legacy job records a successful execution while all real process-level sample tests and app references remain net10-only.
7 tasks
marcschier
added a commit
that referenced
this pull request
Jul 12, 2026
…chaos assertions (#3986) # Description Fixes the nightly **Stress Tests / Chaos TCP** job (e.g. run [29068846476](https://github.com/OPCFoundation/UA-.NETStandard/actions/runs/29068846476)), which had failed on **every run since it was introduced** — the workflow runs on a schedule and never gated PRs, so the failures went unnoticed. ## Root cause The Chaos TCP tests point a `ConfiguredEndpoint` at a local `TcpChaosProxy` and set `UpdateBeforeConnect = false` so the session connects *through* the proxy. Two problems prevented this from working: 1. `ManagedSession.HandleConnectAsync` **ignored** `ConfiguredEndpoint.UpdateBeforeConnect` and hard-coded a pre-connect re-discovery for all non-OpenAPI endpoints. Re-discovery adopted the server's advertised `EndpointUrl`, so the session connected **directly to the server, bypassing the proxy** — no chaos ever reached the connection. Two tests "passed" vacuously. 2. Once the proxy was genuinely engaged, the transparent-reconnect feature had a gap: a request that hit the just-dropped channel (before the manager detected the drop) failed with `BadConnectionClosed` instead of transparently recovering. ## Changes **Product** - **`ManagedSession`** now honours `ConfiguredEndpoint.UpdateBeforeConnect` (default `true`). Setting it `false` opens the channel against exactly the supplied URL (proxy / gateway / NAT / pinned endpoint) without re-discovering. - **`ManagedTransportChannelLease.SendRequestAsync`** transparently resends **idempotent** requests (`Read` / `Browse` / `TranslateBrowsePaths` / `GetEndpoints` / `FindServers`) across a channel-manager reconnect. On a transient transport-drop error it forces a *coalesced* reconnect and resends once the shared channel recovers. - A new **`ChannelEntry.ReconnectGeneration`** counter guards this so a reconnect is only forced for a genuinely undetected drop (same generation, still `Ready`), never for a stale in-flight failure that arrives after recovery — which previously spawned spurious extra reconnect cycles that tore down healthy channels. - **Non-idempotent** requests (`Create*`, `Write`, `Call`, `Publish`, …) surface the error so higher-level recovery (e.g. the subscription engine's own re-create loop) stays in control; requests are never double-applied. **Tests / harness** - **`TcpChaosProxy`** no longer rethrows the expected `OperationAborted` raised by an in-flight upstream `ConnectAsync` during `DropAllConnections` (it was faulting the awaited connection task and failing the test). - Chaos reconnect assertions (L3-A1/A2/A5) are made robust to the inherent non-determinism of reconnect **counts** under concurrent chaos (coalescing merges drops landing in one recovery window; a stale keep-alive `Bad` can add one cycle). Survival is asserted via `FailureRate`, `ReconnectFailed == 0`, and the coalescing fan-out relationship rather than exact `ReconnectStarted/Completed == dropCount`. - Subscription post-drop recovery window widened (`2s → 8s`) to allow re-creating all sessions' subscriptions and monitored items on a loaded CI agent. - **New unit tests** in `ClientChannelManagerManagedTests` cover the new product paths (fast-PR / codecov, since the nightly ChaosTCP job does not run in the fast pipeline): idempotent transient-drop retry, non-idempotent no-retry, non-transient no-retry, and `ChannelEntry.ReconnectGeneration` increment. ## Validation (local, net10.0) - **Chaos TCP**: 6/6 pass across many random seeds (incl. post-merge re-runs of `338475986` and `1999999999`). - `Opc.Ua.Client.Tests` ChannelManager/ManagedSession: **256/256** (incl. the 4 new tests). - `Opc.Ua.Sessions.Tests` ChannelManager/Reconnect: **139/139**. - Core / Client / Stress build with **0 warnings**. ## Notes - This branch has been merged up to date with `master` (redundancy PR #3918). - The two failing `Opc.Ua.PubSub.Tests` in CI — `UdpLoopbackActionResponderAnswersRequesterAsync` and `UdpLoopbackDiscoveryPublisherAnswersSubscriberRequests` — are **unrelated** to this PR (UDP-loopback request/response tests in the PubSub library, associated with #3918) and are timing/environment-flaky. They are out of scope for this change. ## Related Issues - Fixes the recurring nightly Chaos TCP failure (workflow: `.github/workflows/stress-test.yml`). ## Checklist - [ ] I have signed the [CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf) and read the [CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md) doc. - [x] I have added tests that prove my fix is effective or that my feature works and increased code coverage. - [ ] I have added all necessary documentation. - [x] I have verified that my changes do not introduce (new) build or analyzer warnings. - [ ] I ran **all** tests locally using the **UA.slnx** solution against at least .net **framework** and .net **10**, and all passed. - [ ] I fixed **all** failing and flaky tests in the CI pipelines and **all** CodeQL warnings. - [ ] I have addressed **all** PR feedback received.
marcschier
added a commit
to marcschier/UA-.NETStandard
that referenced
this pull request
Jul 23, 2026
Large merge bringing in ~60 upstream commits, most notably the repository layout consolidation (OPCFoundation#4010) that moves the whole tree under src/, samples/, tests/, tools/, docs/, and fuzzing/. Also includes the repo-wide source-generated logging migration (OPCFoundation#3908/OPCFoundation#4003), Part 4 6.6 Redundancy (OPCFoundation#3918), runtime NodeSet-backed node managers (OPCFoundation#3990), MinimalClient sample (OPCFoundation#4006), and many CTT/compliance and security fixes. Conflict resolution: * UA.slnx - took upstream's new-layout solution structure and added our branch-only UaLens under the /samples/ folder (samples/Opc.Ua.Lens/Opc.Ua.Lens.csproj). Dropped the stale Applications/McpServer entry (McpServer already lives under master's /tools/ section as tools/Opc.Ua.Mcp). * .azurepipelines/preview.yml, signlist{Debug,Release}.txt - took upstream's versions wholesale (pure old->new layout path rewrites for upstream-managed files; UaLens was never sign-listed). Relocations for our branch-only files that the consolidation could not auto-move (they exist only on our side, so upstream's rename didn't touch them): * Applications/Opc.Ua.Lens/ -> samples/Opc.Ua.Lens/ (alongside the other sample apps). Updated its csproj ProjectReferences from the old Stack/ + Libraries/ paths to the new src/ layout, and dropped the obsolete <None Remove ...Docs\NugetREADME.md> line (common.props no longer ships a shared root readme; each package owns its local NugetREADME.md, which UaLens already has). * Libraries/Opc.Ua.Client/UserManagement/{IUserManagementClient, UserManagementClient,UserManagementUser}.cs -> src/Opc.Ua.Client/UserManagement/ so they compile back into the (moved) client project. Suppressed CA1873 in the UaLens csproj (with comment + TODO): upstream's source-generated-logging migration enabled this analyzer repo-wide and it flags UaLens's 146 direct ILogger.Log* call sites. Migrating UaLens logging to [LoggerMessage] partials is tracked as follow-up. UaLens build clean (0 warnings / 0 errors). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch implements OPC 10000-4 §6.6 Redundancy end to end — server and client, transparent and non-transparent, across all failover modes (
None,Cold,Warm,Hot,HotAndMirrored,Transparent) — and layers an opt-in distributed high-availability (HA) capability on top so replicas can share address-space, session, and subscription state and expose redundancy to clients through the documented OPC UA mechanisms (for example, running a server replica set across nodes in Kubernetes).The single-instance, in-memory path stays zero-overhead: every distributed feature is opt-in through dependency injection, with a direct-construction fallback, and replaced APIs are kept and marked
[Obsolete].New packages
OPCFoundation.NetStandard.Opc.Ua.Server.RedundancyOPCFoundation.NetStandard.Opc.Ua.Server.Redundancy.CrdtOPCFoundation.NetStandard.Opc.Ua.Server.Redundancy.K8sAlso adds the worked samples
Applications/RedundantServerandApplications/RedundantClient, and the guidesDocs/HighAvailability.mdandDocs/HighAvailabilityKubernetes.md.Server side (§6.6)
AddServerRedundancy(...)populates the liveServer.ServerRedundancynodes (RedundancySupportplusRedundantServerArray/ServerUriArray/CurrentServerId) and drivesServer.ServiceLevelfrom anIServiceLevelProvider(leader high, standby low).Server.ServerRedundancyis typed to the NonTransparent or Transparent subtype according to the configured mode.AddRequestServerStateChange(...)implements the §6.6.5 manual-failover method, withEstimatedReturnTimeand the Maintenance / NoDataServiceLevelsub-ranges.FindServers(ConfiguredRedundantServerSetProvider) and advertise theNTRSdiscovery capability for GDS/NTRS registration.EventIdsynchronization for Transparent and HotAndMirrored sets (§6.6.2.2), excluding per-replica fields so de-duplicating clients do not double-process events.Client side (§6.6)
DefaultServerRedundancyHandlerreadsServer.ServerRedundancyandServer.ServiceLeveland fails over to the highest-ServiceLevel peer. It honors Maintenance (Part 4 Table 105 — disconnect to an available peer), keeps Warm backups disabled until failover (Table 107), and treats peers known only throughServerUriArrayas failover candidates.RedundantManagedClientimplements the Cold, Warm, Hot (a), Hot (b), and HotAndMirrored client patterns: one active session, optional lightweight ServiceLevel status-check sessions to backups, and — for Hot (b) — merging of concurrent reporting streams with exact value-identity de-duplication. HotAndMirrored failover re-activates the mirrored session with the existingAuthenticationTokeninstead of recreating it.Distributed state (opt-in; an extension beyond §6.6)
ISharedKeyValueStore(in-memory default) andINodeStateStore, with anAddressSpaceSynchronizerthat bridges aCustomNodeManager2's predefined nodes to the shared store, so node and reference additions/removals and value changes propagate to other replicas.IDistributedValueCachelets read/write callbacks cache the last value with a freshness bound; monitored items use the normal read pipeline and therefore participate only when the read path does.DistributedSessionManager(wired through a new additiveISessionManagerFactoryseam onStandardServer) mirrors encrypted session state across replicas. On a failover reconnect the standby restores the session and performs the fullActivateSessionclient-certificate signature validation against a single-useserverNonceconsumed via compare-and-swap across the replica set, enforcing the same SecurityPolicy and SecurityMode. TheAuthenticationTokenis a lookup key only, never an authenticator: entries are keyed by the SHA-256 digest of the token, and a cross-replica restore emits a distinct audit event.IRecordProtector; secret-bearing mirrors require a registered protector or an explicit opt-out.Active/active (CRDT)
Opc.Ua.Server.Redundancy.Crdtadds leaderless multi-writer address-space replication with CRDTs and gossip (UseReplicatedAddressSpace) plus CRDT-backed session metadata (UseReplicatedSessions). Networked gossip is fail-closed without authenticated transport (mutual TLS for TCP; an explicit opt-out for isolated dev/test). CRDT state is eventually consistent, so exactly-once decisions (such as the single-use nonce registry) stay on a strongly consistent store. The package is available on all stack target frameworks.Kubernetes
Opc.Ua.Server.Redundancy.K8sprovides Kubernetes Lease leader election, EndpointSlice-based peer discovery, and ServiceLevel-driven readiness for running an OPC UA server replica set.Docs/HighAvailabilityKubernetes.mdcovers StatefulSet/Deployment and Service manifests, RBAC, probes, time synchronization, secrets, gossip-port NetworkPolicy, and GDS/NTRS registration.Compatibility and validation
ISessionManagerFactoryandIServerStartupTask); replaced APIs marked[Obsolete]for migration.net472,net48,netstandard2.0,netstandard2.1,net8.0,net9.0,net10.0); NativeAOT-compatible where the runtime supports it.See
Docs/HighAvailability.mdfor the full OPC 10000-4 §6.6 mapping, theAdd*(standard nodes) versusUse*(extension) builder convention, and the security/rotation guidance.