Skip to content

Move the CDP cohort membership consumer to the processor's output topic safely #88010

Closed

Description

@matheus-vb

The CDP cohort membership consumer (CdpCohortMembershipConsumer, plugin server mode cdp-cohort-membership) upserts membership changes into the cohort_membership Postgres table. Its brokers still point at the cluster that hosts the legacy cohort_membership_changed topic, which no longer receives messages.

The move follows the topic creation in #88009. The consumer hardcodes the cohort_membership_changed topic name, so once the new topic exists on the cluster the cohort stream processor produces to, the move is a broker configuration change on the deployment alone. No consumer code changes are needed and #87866 is not needed. The consumer can move before or after the processor flips its writes, since reading an empty topic is harmless, but both must happen before anything depends on the Postgres table.

Three things need care during the move.

Keep the consumer group id unchanged. Autoscaling on consumer lag matches the literal group id, so changing it silently degrades scaling. The group has no committed offsets on the new cluster either way, so it starts fresh regardless.

Do not expect the table to converge on its own. The new topic starts empty, so there is no replay burst and no history to catch up from. The cohort_membership table converges through reconcile runs dispatched per cohort after the consumer is live, each of which replays that cohort's full current membership.

Add consumer lag alerting for this group. Topic settings are chosen deliberately at creation in #88009, but the consumer group needs lag alerting before the Postgres table depends on the feed, since lag past retention is silent data loss.

Activity

  1. changed the title [-]Repoint the CDP cohort membership consumer to the cluster the cohorts pipeline produces on[/-] [+]Move the CDP cohort membership consumer to the processor's output topic safely[/+] on Aug 24, 2026
  2. matheus-vb commented on Sep 28, 2026

    @matheus-vb
    MemberAuthor

    Done in PostHog/charts#14659 (consumer moved to the new topic's cluster, same group id) and PostHog/charts#14800 (lag alerting). EU follows in #107988.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions