Repository navigation
chore(cohorts): make the cohort membership consumer's source configurable - #87866
Closed
gustavohstrassburger wants to merge 1 commit into
Closed
gustavohstrassburger wants to merge 1 commit into
gustavohstrassburger wants to merge 1 commit into
Conversation
…able The cdp-cohort-membership consumer is the only writer to the Postgres cohort_membership table, which the feature-flags realtime cohort path reads. It stays, but its Kafka source has to move off the calculated-events cluster that is being decommissioned. Add CDP_COHORT_MEMBERSHIP_CONSUMER_TOPIC and _GROUP_ID, following the existing CDP_LEGACY_EVENT_CONSUMER_* precedent, so a deployment can point the consumer at the cohort-stream-processor's output topic without a code change. Defaults keep today's behaviour. Also drop the now-dead cohort_membership_changed entry from the Django Kafka routing map: no Python producer remains, and it pointed at the cluster going away. Generated-By: PostHog Desktop Task-Id: dcb5016e-2bdc-425b-9b29-c455bbee721e
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
1 task
github-actions
Bot
added
feature/cohorts
Contributor
🤖 CI report
|
Contributor
There was a problem hiding this comment.
Pull request overview
Makes the cohort membership consumer’s Kafka topic and group configurable while removing obsolete routing.
Changes:
- Adds deployment-configurable topic and group defaults.
- Adds the shadow topic constant.
- Removes stale Django Kafka routing.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Summary |
|---|---|
posthog/settings/kafka.py |
Updates routing configuration examples. |
posthog/kafka_client/routing.py |
Removes obsolete cohort topic routing. |
nodejs/src/common/config/kafka-topics.ts |
Adds the shadow topic constant. |
nodejs/src/cdp/consumers/cdp-cohort-membership.consumer.ts |
Uses configurable Kafka source settings; custom-value coverage is still needed. |
nodejs/src/cdp/config.ts |
Defines configurable consumer settings and defaults. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+34
to
+35
| groupId: config.CDP_COHORT_MEMBERSHIP_CONSUMER_GROUP_ID, | ||
| topic: config.CDP_COHORT_MEMBERSHIP_CONSUMER_TOPIC, |
6 of 11 tasks
This was referenced Aug 24, 2026
Contributor
Author
|
Instead of renaming, we'll create a fresh topic with the same hardcoded name. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
cdp-cohort-membershipconsumer is the only writer to the Postgrescohort_membershiptable, and that table is what the feature-flags realtime cohort read path queries (rust/feature-flags/src/cohorts/membership/realtime_provider.rs). The new Rustcohort-stream-processoronly produces membership changes to Kafka; nothing else persists them.#79808 proposed deleting the consumer as part of decommissioning the
warpstream-calculated-eventscluster. That would remove the sink the realtime behavioral cohorts path needs, so it is closed in favour of this PR: retire the cluster, keep the consumer.The consumer's topic was hardcoded, so it could not follow its producer to a different cluster and topic without a code change.
Refs #82624
Changes
CDP_COHORT_MEMBERSHIP_CONSUMER_TOPICandCDP_COHORT_MEMBERSHIP_CONSUMER_GROUP_ID, following the existingCDP_LEGACY_EVENT_CONSUMER_*pattern. Defaults match today's behaviour, so nothing changes until a deployment sets them.cohort_membership_changed_shadowtopic constant, and dropped thecohort_membership_changedentry from the Django Kafka routing map — no Python producer remains and it pointed at the cluster being destroyed.How did you test this code?
npx tsc --noEmitinnodejs/: no errors in any file this PR touches. The remaining errors are pre-existing and come from unbuilt workspace packages (@posthog/hogvm,@posthog/replay-anonymizer).ruff checkandruff format --checkon the two changed Python files: clean.cdp-cohort-membership.consumer.test.tsalready builds the consumer from config and produces to the resulting topic, so a hardcoded or mis-wired topic fails it.Automatic notifications
Docs update
None.
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Authored with Claude Code (PostHog Desktop). No repo skills were invoked.
The starting point was a review of #82624's plan rather than a code change: tracing every reader and writer of the Postgres
cohort_membershiptable showed the consumer is load-bearing for the realtime path, not dead weight. The alternative of leaving the topic hardcoded and repointing only the brokers was rejected — the producer writes to a different topic name, so the consumer needs both.Created with PostHog Desktop