Skip to content

Settings-backed stores lose concurrent updates #586

Closed

Description

@pratyush618

Every dashboard feature store keeps a whole JSON document in one settings key and mutates it read-then-write: auth:users, webhooks:subscriptions, webhooks:deliveries:<id>, and the overrides:* rows. Storage offers set_setting and nothing conditional, so two writers that read the same version both write, and the later write wins wholesale.

Concurrent writers are not hypothetical: the dashboard is meant to run as more than one replica against one backend, and #584 adds a standalone server that can run alongside an SDK dashboard on the same rows.

Raised in review on #584 (two findings, same root cause).

What it costs today

The affected writes are admin-frequency — creating a user, editing a webhook, setting an override — so the window is small and the blast radius is one lost edit rather than corruption. That is why #584 shipped without a fix rather than growing a locking scheme inside one crate.

What would fix it

A compare-and-set at the storage layer, e.g. set_setting_if(key, expected_version_or_value, new), with each backend implementing it natively:

  • SQLite / Postgres: UPDATE … WHERE value = $expected, or a version column
  • Redis: WATCH/MULTI, or a Lua script

The stores would then retry on conflict. Storage::acquire_lock exists and could serialise these writes instead, but it makes every settings write take a lock round-trip and leaves the lock's own TTL to reason about — worth comparing before choosing.

This is the "settings CAS" item already noted as core debt in the cross-SDK parity work; the SDK dashboards have the same pattern and would benefit from the same primitive.

Activity

  1. pratyush618 commented on Aug 2, 2026

    @pratyush618
    ContributorAuthor

    Fixed in #605 (`eb633048`).

    Storage::set_setting_if already existed on all three backends and taskito-server already used it — no SDK binding exposed it, so every SDK dashboard store still read-modify-wrote. Now exposed in the PyO3, napi and JNI bindings, with the same retry helper in each SDK: decode, mutate in place, compare the decoded document, write conditionally, retry.

    Stores routed through it: users/OAuth users, webhook subscriptions, the delivery log, task/queue overrides, and the middleware disable list, in all three SDKs.

    Two deliberate behaviour changes, both matching what taskito-server had already decided: an emptied disable list leaves a [] row rather than deleting the key (the delete sat outside the compare-and-set), and authenticate no longer resurrects a user deleted mid-login.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions