POC: Dedicated feature flags read store #54145

Summary

Build a minimal, end-to-end prototype of the dedicated read store for the feature flags service. The prototype exercises the full data path (Kafka consumption, version-ordered upserts, dedicated PG storage, and shadow reads) from the flags service on a small, fixed set of teams.

The POC is not a partial production rollout. It is a correctness and operability test on canary traffic.

Goals

1. Correctness under real data. Shadow reads against the dedicated store return the same person row (or absence-of-row) as the shared Persons DB for the chosen canary teams.
2. CDC consumer keeps up. The standalone consumer maintains a short lag on both clickhouse_person and clickhouse_person_distinct_id for the target teams under steady state, and recovers after a forced restart without data loss or duplicates.
3. Version resolution is correct end-to-end. Out-of-order messages, person deletions (+100 version bump), and distinct_id reassignment during person merges all converge to the correct final state.
4. Bootstrap + CDC handoff is seamless. A backfilled team matches the live stream without gaps, duplicates, or stale overwrites.
5. Read path performance is acceptable. The GIN-indexed single-table lookup returns at a latency comparable to or better than the current two-table JOIN for p50 and p99.
6. Write path holds up under realistic load. Worth calling out separately from "consumer keeps up": flags is a read-heavy service, but the persons state it depends on is write-heavy, and this is the first time we'd be running a write-heavy workload against a GIN-indexed array column. Write traffic is biased toward identified persons (2+ distinct_ids), not the 95%-single-id row-count distribution, so the index update cost under real traffic is the thing to watch. The POC should surface sustained write-path p99 and tail behavior on the dedicated store, a consumer can stay caught up by buffering while the DB is quietly struggling, so lag alone isn't enough signal.

Non-goals

• Groups, cohorts, and hash key overrides. The POC builds flags_person_lookup only. Groups/cohorts are simpler versions of the same pattern; hash key overrides are a distinct and more complex problem (dual-write, state-transition semantics) and are deferred until a separate design pass.
• Production traffic at scale. The POC is scoped to a few canary teams.
• Final schema / operator lock-in. We may need to iterate on column choices, index layout, and version semantics.
• Final cost / instance sizing. POC runs on a small dev Aurora instance. Production sizing is a follow-up once working-set measurements.
• Decommissioning of the shared Persons DB read path. The shared DB remains the authoritative read source; the POC only shadow-reads against the dedicated store.

Architecture

See the diagram. For the POC, the components marked NEW in green are in scope, and only the flags_person_lookup table from the dedicated store subgraph:

• CDC Consumer (new) — standalone Rust binary consuming clickhouse_person + clickhouse_person_distinct_id, filtered to the POC team whitelist.
• flags_person_lookup (new) — single table on a small dev Aurora PG instance, with the denormalized schema and GIN index described below.
• Shadow read path in the flags service — reads both the shared Persons DB (authoritative) and the dedicated store (comparison), logs the diff, returns the shared DB result.
• Reconciliation job (new) — hourly, rolling through the POC teams, closes the application-level CDC dual-write gap.

Groups, cohorts, and hash key overrides paths from the diagram are present for context but remain untouched by the POC.

Components to build

1. Dedicated store schema and infrastructure

A small Aurora PG instance with the flags_person_lookup table and a heartbeat table for lag monitoring. One row per person, keyed on (team_id, person_uuid), with the denormalized distinct_ids array and independent version columns for the person and distinct_id streams. Partitioned by team_id from day one. Pin the Aurora PG version (e.g. 15.13) to match the production ceiling, Aurora caps individual tables at 32 TB and total instance storage at 128 TB on 15.12 / 256 TB on 15.13.

2. CDC consumer (standalone Rust binary)

A new standalone Rust service that consumes clickhouse_person and clickhouse_person_distinct_id, filters messages down to the canary team whitelist, and applies version-guarded idempotent upserts into the dedicated store. Must correctly handle the tricky paths: person deletions (the +100 version bump), and distinct_id reassignment during person merges (the "new mapping only" semantic).

3. Shadow read path in the flags service

A second, parallel query path against flags_person_lookup that runs alongside the existing shared-DB person lookup for canary teams only. The shared DB remains authoritative — the dedicated store result is compared and logged, never returned to the caller. Pre-filtered at the request level so non-canary traffic never touches the dedicated store. Errors in the shadow path must be invisible to the handler. This is strictly a read-comparison mechanism, not a production fallback architecture, the POC does not rehearse "fall back to shared DB on dedicated-store failure," because a permanent fallback to the shared Persons DB would defeat the isolation goal of the whole project.

4. Bootstrap (snapshot-then-CDC)

A one-shot backfill job that populates the dedicated store with historical person data for canary teams. Follows the DBLog pattern: start the CDC consumer first from the live Kafka tail, then paginate the shared Persons DB using the same version-guarded upserts so live messages win over stale snapshot rows. Restartable on failure.

5. Reconciliation job

A periodic job that closes the application-level dual-write gap (Plugin Server commits to Postgres, then fire-and-forget produces to Kafka). Walks each canary team, diffs source and dedicated store versions, and re-applies any rows where the source is ahead. Same upsert semantics as the CDC consumer.

6. Observability

Grafana dashboard and alerts covering consumer lag, upsert rate, shadow-read match/mismatch rate, read-path latency comparison, and reconciliation drift.