You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Create the cohort_membership_changed topic on the Kafka cluster the cohorts pipeline uses #88009
The realtime cohorts pipeline keeps all of its topics on one Kafka cluster. The legacy cohort_membership_changed topic lives on a different cluster, and it stopped receiving data when the legacy cohort calculation producer was deleted.
Create a new cohort_membership_changed topic on the cluster the pipeline uses. This lets the cohort stream processor produce membership changes and the CDP membership consumer read them from the same cluster.
Choose the partition count deliberately instead of copying the legacy topic. The legacy topic was sized for the old calculation fan out, and the pipeline's own topics use smaller counts. The partition count caps consumer parallelism and is painful to change later, so pick a value we are willing to live with. Choose retention and durability deliberately as well, since this topic feeds the cohort_membership Postgres table rather than a diagnostic harness.
Reopened after the consumer move settled in #82624 was reconsidered. Nothing consumes the shadow topic yet, so flipping the processor's COHORT_MEMBERSHIP_CHANGED_TOPIC setting to the new topic is an env change plus one restart in a window with no active backfill runs, and that stops being cheap the moment a consumer depends on the output. The shadow topic's history also carried reconcile completion markers before #76111, which fail the CDP consumer's schema validation and crash the pod, so a fresh topic is clean by construction rather than by retention arithmetic. The CDP membership consumer already hardcodes the cohort_membership_changed topic name, so once this topic exists it follows with a broker configuration change alone and #87866 is not needed.
Closing in favor of the plan settled in #82624: instead of creating a new topic and moving the producer, the consumer moves to the processor's existing output topic (#87866 plus a deployment change). #88010 tracks doing that move safely.
Reopening per the thread decision: the consumer move to the shadow topic is reversed in favor of creating this topic and flipping the processor's writes while nothing consumes the output. Body updated with the rationale, and #88017 plus its sub issues updated to match.
Activity
matheus-vb commentedon Aug 24, 2026
Closing in favor of the plan settled in #82624: instead of creating a new topic and moving the producer, the consumer moves to the processor's existing output topic (#87866 plus a deployment change). #88010 tracks doing that move safely.
matheus-vb commentedon Aug 24, 2026
Reopening per the thread decision: the consumer move to the shadow topic is reversed in favor of creating this topic and flipping the processor's writes while nothing consumes the output. Body updated with the rationale, and #88017 plus its sub issues updated to match.