RavenDb - Ensure only one durability agent is instantiated

[Bishbulb]

Overview

RavenDbMessageStore.StartScheduledJobs eagerly built and started a RavenDbDurabilityAgent at boot. However, the NodeAgentController also built another agent for wolverinedb://ravendb/durability via the IAgentFamily / MessageStoreCollection path. The result is that we had two durability agents registered under different URIs (internal://scheduledjobs vs wolverinedb://ravendb/durability), so the controller didn't dedupe them, and they polled concurrently in the same process.

Both agents share the singleton RavenDbMessageStore, which led to this behavior:

• Agent 1 follows the code path to do an initial acquire and sets _scheduledLock and _lastScheduledLockIndex.
• Agent 2 sees _scheduledLock != null on the shared instance and takes the code path to do a renewal, which results in a PUT with the current index (this succeeds against the server because that index is still valid)
• Both agents now believe they hold the lock, and both fetch the same envelopes at the same change vector, and race to mark them Incoming. Loser surfaces ConcurrencyException, Wolverine requeues the failed message, and the timeout fires twice.

I observed this when testing a real workload, and added instrumentation which showed two RavenDbDurabilityAgent instance ids both LOCK-ACQUIRED in the same window, fetching the same IncomingMessage doc at the same change vector, which resulted in downstream concurrency exceptions.

Fix

Drop the eager StartTimers() call. The cluster-managed agent built via MessageStoreCollection.BuildAgentAsync is the single owner of the scheduled-jobs poller and recovery loop; NodeAgentController calls StartAsync on it. The agent returned from
StartScheduledJobs is held by WolverineRuntime.DurableScheduledJobs purely for its disposal-time StopAsync, which is null-safe on the unstarted task fields.

Note

Wolverine.CosmosDb.Internals.CosmosDbMessageStore.StartScheduledJobs mirrors this pattern (agent.As<CosmosDbDurabilityAgent>().StartTimers()) and very likely has the same bug. Flagging in case you want to look at that separately.

[jeremydmiller]

Closing in favor of #2629, which carries the same production fix with a reflection-free regression test (adds public IsPolling / InnerAgents accessors instead of poking at private fields). Thanks @Bishbulb — kept you as co-author on the new commit. The CosmosDb note you flagged is preserved in #2629's description for follow-up.