Monotonic write guard is per-process, and the PG upsert has no version condition — a fresh replica's first write can roll a node back #971

Summary

MonotonicWriteGuardStorageAdapter enforces "the store never moves backward" using a per-process in-memory high-water mark. PR #968 fixes a real window in it (the mark was claimed after the row was mutated, so a writer on the IStorageAdapter.Changes feed could overwrite a newer row with a stale snapshot — acked writes dying silently).

That fix is correct and worth landing on its own, but it leaves a structural gap that #968 deliberately does not close, and the gap matters in production rather than in tests.

The gap

Two parts, and they compound:

1. The mark is per-process. A second replica starts with an empty mark table, so its first write for a given path has nothing to compare against.
2. PostgreSqlStorageAdapter's upsert carries no version condition. Nothing at the store level refuses a write whose version is below the durable row's.

So on a fresh replica's first write to a path, monotonicity is enforced by nothing at all — not by the in-process guard (no mark yet) and not by the database (no condition on the UPDATE). Any stale snapshot that replica happens to hold will be written straight over newer durable state.

This is invisible in single-process tests, which is why it has not shown up: the guard looks sound because in one process the mark is always populated by the time it matters.

Why it deserves a real fix rather than a wider in-memory guard

The invariant is a property of the row, not of a process's memory. Any in-process structure re-derives it and will be wrong again the next time the topology changes (a rollout briefly running two pods, a KEDA scale-out, a restarted replica). The durable fix is a store-level compare-and-set: make the write conditional on the version it believes it is replacing, and refuse (or re-read and retry the merge) when the condition fails.

Note the same reasoning applies to SnowflakePartitionStorageProvider, which has not been checked.

Impact

Silent data loss, and the shape is nasty: the stale write is acked, so nothing surfaces. The next write is minted one above the rolled-back row and is accepted for a perfectly good reason, so the lineage continues from the wrong place. What an operator sees is a node that quietly reverted — the Version=12 → Version=2 shape the guard exists to prevent — with no error anywhere.

Provenance

Split out of #826 / PR #968, which root-caused and fixed the in-process window. Recorded there as "residual, not fixed — needs a store-level compare-and-set; deserves its own issue."