Skip to content

Prod-ready realtime behavioral cohorts #80291

Closed

Description

@matheus-vb

Where we are

The streaming pipeline (shuffler, processor, seeder) is deployed on current master and healthy, ingesting events for allowlisted teams and producing membership transitions to the shadow topic.
The backfill machinery is fully built and merged: seed topic, run and chunk tables, seeder scans, fenced and paced seed apply, reconcile snapshots with a dedicated marker topic, kind aware completion, and the readiness finalizer.
Backfill runs work end to end but are operator driven from a toolbox pod, and they park short of completion: behavioral runs stop at reconciling because the Django finalizer is disabled, person property runs stop at seeding because person reconcile dispatch is off in the seeder.
The save triggered dispatch path is fully built but dark: the Django fleet has none of the cohort env vars set (realtime allowlist, trigger allowlist, attestations).
The processor PVC has been resized to 150GiB.

Next steps

  1. Ops hardening. Arm the processor's storage guards: COHORT_PERSON_RECORD_TTL_DAYS so the person record store stops growing unboundedly, and COHORT_SEED_DISK_PAUSE_PCT so seeding pauses itself before disk pressure. Add a first set of alert rules covering stalled runs, failed chunks, disk utilization, produce errors, and trigger outcomes, since with automatic triggering nobody will be watching runs by hand. Also explain the recent person seed skips by reason and recreate any run that lost seeds.

  2. One time run cleanup. Produce an inventory of every backfill run in an active status across all teams, terminalize the stale ones, and write down the short list of verified runs the finalizer is allowed to stamp. This must land before the finalizer is enabled, because the finalizer stamps readiness for anything that qualifies the moment it turns on and stamps are one way.

  3. Django enablement. Ship the standing env on web and workers: realtime team allowlist, the merge gate and durability attestations, a person topic bytes budget, a longer default person scan horizon, and the finalizer flag. Success looks like the parked behavioral runs reaching completed and stamping readiness within minutes of rollout. Then run resave_cohorts for the allowlisted teams and verify every realtime cohort has a non null condition_type, since stale classification silently breaks behavioral evaluation.

  4. Person run completion. Make person property runs travel the same path as behavioral ones. Flip the flags membership stamp policy in every region first (inert while flags reads are off, but the ordering matters), then enable person reconcile dispatch in the seeder so parked person runs get reconciled and observed, then enable the person readiness gate so the finalizer completes and stamps them.

  5. Automatic triggering. Enable the trigger allowlist for a single pilot team so that saving a behavioral cohort creates, seeds, reconciles, and completes a backfill run with no human involvement. Prove it with a hands off cohort edit end to end, then soak for a week on the alert set before widening the allowlist to more teams.

Out of scope for this milestone: pointing membership output at the topic CDP consumes, and the flags read path.
Small code fixes recommended alongside: a catalog refresh staleness metric (a silently stale filter catalog during a run is invisible today), failing a run when a chunk exhausts its retries (today it parks in seeding forever and blocks future runs for that cohort), and splitting the trigger refusal counter so budget refusals and blocked run slots are distinguishable.

Activity

  1. moved this from Todo to In Progress in Feature Flagson Aug 12, 2026
  2. gustavohstrassburger commented on Sep 11, 2026

    @gustavohstrassburger
    Contributor

    hey @matheus-vb, can we move this to done?

  3. removed their assignment
    on Sep 21, 2026
  4. matheus-vb commented on Sep 28, 2026

    @matheus-vb
    MemberAuthor

    Done. Code fixes in #82769. Rollout in PostHog/charts#14352, PostHog/charts#14666, PostHog/charts#15312, PostHog/charts#15395 and PostHog/charts#15502.

  5. moved this from In Progress to Done in Feature Flagson Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions