Repository navigation
Prove Rust flags hypercache parity via shadow comparison #88339
Description
Goal
Prove that the Rust flags-cache-builder produces the same flags.json entries as the Python (Celery) builder, at full fleet volume, with zero serve exposure. The builder builds every team's entry from a shadow: true invalidation, diffs it against the live Python entry, and never writes.
Exit criterion: a sustained stretch at 100% shadow volume with confirmed mismatches (flags_cache_shadow_mismatch_total) at or returning to zero. When that holds, the serve ramp carries operational risk only.
Why shadow instead of a slow serve ramp
Serve routing covers 3 canary projects today. A serve ramp exposes users to any parity bug while it gathers evidence. Shadow comparison gathers the same evidence across the whole fleet with no user exposure, and soaks the builder at the volume it must absorb anyway.
Tasks
- Label verifier fixes by writer, as the production backstop (feat(flags): attribute hypercache verifier fixes to the primary writer #87049)
- Split the verifier-fixes panel by writer (PostHog/grafana-dashboards#415)
- Merge and deploy shadow compare mode in the builder (feat(flags): add shadow compare mode to the flags cache builder #87891). Must reach production before the publisher.
- Merge the builder dashboard (PostHog/grafana-dashboards#416)
- Shadow publisher:
_enqueue_invalidationalso producesshadow: trueinvalidations for Celery-owned teams, behind a new percentage gate (not the serve-routing flag) - Ramp shadow volume 1% -> 10% -> 100%; watch confirmed mismatches, real-build e2e latency, and read-replica load
- Alert (charts): warn on confirmed mismatches over 6h
- Triage each confirmed mismatch as a Rust parity bug until the series holds at zero
Out of scope
- Cohort-triggered shadow coverage: cohort edits bypass the Kafka topic; the writer-labeled verifier counter is the backstop
- The serve ramp itself: starts after the exit criterion holds
- Python builder deletion and the end-state Rust sweep
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
JamesPatrickGill commentedon Aug 26, 2026
Publisher status and two ramp prerequisites
Publisher. #88585 is approved and CI is green. It is not merged yet.
Correction to the task list. The publisher does not hook
_enqueue_invalidation. It publishes from the tail ofupdate_team_service_flags_cache, gated on a successful cache write. Publishing at invalidation time raced the rebuild it measures: the builder read fresh Postgres against a pre-rebuild Redis entry and reported the gap as drift. Two invalidations inside one stale window hash the same, so the tracker promoted Celery lag to a confirmed parity defect.Consumer. #87891 is merged, and its commit is contained in the current prod-us and prod-eu deploy.
Gate flag, done.
flags-cache-shadow-comparenow exists at 0% rollout, aggregated on the project group type. It is a separate flag fromflags-cache-kafka-dual-writeand never changes which writer serves a team. Do not delete it while the producer code is live. Local evaluation cannot resolve a flag that does not exist, so shadow volume would drop to zero with no log and no counter.Alert, still open. This one blocks the ramp.
FlagsCacheInvalidationDeliveryFailuresin PostHog/charts fires at a threshold of 0 on any delivery failure for theflags_cache_invalidationtopic. Shadow messages and real invalidations share that topic, andposthog_kafka_producer_messages_totalcarries no label that separates them.Today a firing means a Kafka-routed team serves a stale
/flagspayload. After the ramp it mostly means one telemetry message did not land. The runbook still tells the responder to look for stale flag serving, and its mitigation step says to turn offflags-cache-kafka-dual-write. That is the serve-routing flag. So a lost shadow message can lead someone to disable the Rust serve path.Real invalidations come from the 3 canary projects. Shadow messages will come from every Celery-owned team. At 100% most firings of this alert would be false positives.
Suggested additions to the task list, above the ramp step:
flags-cache-shadow-comparegate flag at 0%FlagsCacheInvalidationDeliveryFailuresand its runbook in PostHog/charts, before the ramp