[gustavohstrassburger]

Problem

The warpstream-calculated-events cluster is being decommissioned and its cohort_membership_changed topic is dead — its only producer was removed when the legacy precalculated realtime-cohort path was deleted. What is left are consumers, and they have to go before the cluster can be destroyed.

The replacement is the Rust cohort-stream-processor, which writes to cohort_membership_changed_shadow on the warpstream-ingestion cluster. Nothing in that path is touched here.

Changes

• ClickHouse migration 0293 drops cohort_membership_ws_mv then kafka_cohort_membership_ws on NodeRole.INGESTION_MEDIUM, cloud-only for US/EU/DEV — the same guard and node role migration 0245 used to create them.
• 0245 becomes a no-op with a NOTE, following the pattern in 0004_kafka.py, because the SQL builders it imported are removed.
• Removes the _ws Kafka table and MV definitions from posthog/models/cohortmembership/sql.py plus the CONSUMER_GROUP_COHORT_MEMBERSHIP_WS constant.
• Deletes the cdp-cohort-membership Node consumer, its capability, its server.ts wiring, and its tests.
• Removes KAFKA_COHORT_MEMBERSHIP_CHANGED from the Python and Node topic constants and its CALCULATED_EVENTS routing entry.

This removes the last writer to the Postgres cohort_membership table. rust/feature-flags still reads that table, and its realtime read path is gated off by default, so no flag behavior changes here.

The cdp-cohort-membership PluginServerMode member is kept as a no-capability stub with a TODO, matching how cdp-precalculated-filters was handled. An unknown PLUGIN_SERVER_MODE throws at boot, so removing the member before charts stops setting it would crash-loop the pod.

Ordering

1. This PR and its paired charts PR (which removes the deployment) merge and deploy.
2. Migration 0293 runs.
3. Only then can posthog-cloud-infra destroy the warpstream-calculated-events cluster.

Not removed

KafkaClusterProfile.CALCULATED_EVENTS and CLICKHOUSE_KAFKA_WARPSTREAM_CALCULATED_EVENTS_NAMED_COLLECTION stay. The named collection is still used by kafka_precalculated_events_ws and kafka_precalculated_person_properties_ws, which also sit on this cluster. Those need their own cleanup before the cluster is actually free.

The MSK-cluster kafka_cohort_membership + cohort_membership_mv pair also stays. It reads the same dead topic, but it is on the default cluster, migration 0248 already drops it in cloud, and it is the local/CI definition of the table — so it does not block the decommission.

How did you test this code?

No new tests; this is a removal. What was run:

• ruff check and ruff format --check on every changed Python file — clean.
• tsc --noEmit -p . in nodejs/ — the only errors are pre-existing unbuilt workspace packages (@posthog/hogvm, @posthog/cyclotron, @posthog/replay-anonymizer). None are in the files this PR touches.
• Repo-wide grep for COHORT_MEMBERSHIP_CHANGED, cohort_membership_changed, CALCULATED_EVENTS, and cdp-cohort-membership; every remaining hit is the shadow topic, cohort-stream-processor, cohort-seeder, feature-flags, or the precalculated-events tables listed above.

Not run: mypy (not installed here), the Python and Jest suites, and the migration itself against a real ClickHouse. No ClickHouse snapshot changed — the _ws tables were never in schema.py.

Automatic notifications

• Publish to changelog?

Docs update

Updated the WarpStream virtual-cluster table in .agents/skills/monitoring-ingestion-pipeline/SKILL.md and a stale routing-override example in posthog/settings/kafka.py.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Written by Claude via PostHog Code. No repo skills were available to invoke in this environment; posthog/clickhouse/migrations/AGENTS.md was read directly instead and its conventions followed for the drop migration.

Two decisions worth a reviewer's attention, both deviations from the original plan:

• The plan expected CALCULATED_EVENTS and the named-collection setting to be fully removable. They are not — precalculated_events and precalculated_person_properties still bind _ws Kafka tables to this cluster, so the cluster is not free after this PR alone.
• The plan asked for the cdp-cohort-membership mode enum member to be deleted. It is stubbed instead, for the crash-loop reason above.

Also note posthog/clickhouse/migrations/AGENTS.md asks for migration-only PRs. This one mixes a migration with consumer removal because the two are one decommission step; splitting it into a stack is reasonable if a reviewer prefers.

---

Created with PostHog Code

[trunk-io]

Merging to master in this repository is managed by Trunk.

• To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

[gustavohstrassburger]

@greptile review

[greptile-apps]

_{Reviews (1): Last reviewed commit: "chore(cohorts): remove warpstream-calcul..." | Re-trigger Greptile}

[Copilot]

Pull request overview

Removes the legacy WarpStream calculated-events cohort-membership ingestion path (dead cohort_membership_changed topic + remaining consumers) so the WarpStream cluster can be decommissioned, while leaving the Rust cohort-stream-processor shadow path untouched.

Changes:

• Adds ClickHouse migration 0293 to drop the WarpStream cohort-membership Kafka table + MV on NodeRole.INGESTION_MEDIUM (cloud-only US/EU/DEV), and turns migration 0245 into a no-op.
• Removes the WarpStream cohort-membership ClickHouse DDL helpers + consumer group constant, and deletes the Node cdp-cohort-membership consumer (capability + server wiring + tests).
• Removes the cohort_membership_changed topic constant and its CALCULATED_EVENTS routing entry, plus updates related docs/config comments.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
posthog/settings/kafka.py	Updates routing-override example comment away from the removed topic.
posthog/models/cohortmembership/sql.py	Removes WarpStream _ws Kafka table/MV SQL definitions and related imports.
posthog/kafka_client/topics.py	Drops KAFKA_COHORT_MEMBERSHIP_CHANGED constant.
posthog/kafka_client/routing.py	Removes routing entry tying cohort-membership topic to CALCULATED_EVENTS.
posthog/clickhouse/migrations/max_migration.txt	Bumps max migration to 0293.
posthog/clickhouse/migrations/0293_drop_warpstream_calculated_events_cohort_membership.py	New migration dropping cohort_membership_ws_mv and kafka_cohort_membership_ws (cloud-only).
posthog/clickhouse/migrations/0245_warpstream_cohort_membership_kafka_engine.py	Converts prior create-migration into a no-op with explanatory note.
posthog/clickhouse/kafka_engine.py	Removes CONSUMER_GROUP_COHORT_MEMBERSHIP_WS.
nodejs/tests/helpers/kafka.ts	Removes removed topic from test topic list/imports.
nodejs/src/types.ts	Removes cdpCohortMembership capability type member.
nodejs/src/server.ts	Removes CdpCohortMembershipConsumer import and startup wiring.
nodejs/src/common/config/kafka-topics.ts	Removes KAFKA_COHORT_MEMBERSHIP_CHANGED constant.
nodejs/src/common/config.ts	Keeps PluginServerMode.cdp-cohort-membership as a stub enum member with TODO note.
nodejs/src/cdp/consumers/cdp-cohort-membership.consumer.ts	Deletes the consumer implementation.
nodejs/src/cdp/consumers/cdp-cohort-membership.consumer.test.ts	Deletes the consumer test suite.
nodejs/src/cdp/_tests/fixtures.ts	Removes cohort-membership-specific test fixtures/types.
nodejs/src/capabilities.ts	Removes realtime cohort capability group and returns {} for cdp_cohort_membership mode.
.agents/skills/monitoring-ingestion-pipeline/SKILL.md	Updates WarpStream virtual-cluster topic mapping documentation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[gustavohstrassburger]

Superseded by a split, per posthog/clickhouse/migrations/AGENTS.md requiring migration PRs to be migration-only:

• chore(clickhouse): drop precalculated warpstream kafka engine tables #79807 — the ClickHouse drop migration
• chore(cohorts): remove the cdp cohort membership consumer #79808 — the consumer removal

The split also widened the migration. This PR only dropped the cohort_membership Kafka engine pair, but the precalculated_events and precalculated_person_properties _ws pairs are still bound to the same warpstream_calculated_events named collection on master, so the cluster would not have been free to decommission. #79807 drops all three.

Dropping this PR's other change too: it neutralized the already-applied migration 0245 to operations = [] because the SQL builders it imports were being deleted. Leaving those builders in place avoids rewriting an applied migration, and is consistent with the precalculated pairs, whose builders also stay.

[github-actions]

🤖 CI report

ℹ️ ClickHouse migration SQL — 2 migration(s)

ClickHouse migration SQL per cloud environment

• unset
  • 0245_warpstream_cohort_membership_kafka_engine — operations list is empty under this environment
  • all

    DROP TABLE IF EXISTS precalculated_events_ws_mv

    DROP TABLE IF EXISTS kafka_precalculated_events_ws

    DROP TABLE IF EXISTS precalculated_person_properties_ws_mv

    DROP TABLE IF EXISTS kafka_precalculated_person_properties_ws

• US, EU, DEV
  • 0245_warpstream_cohort_membership_kafka_engine — operations list is empty under this environment
  • medium

    DROP TABLE IF EXISTS precalculated_events_ws_mv

    DROP TABLE IF EXISTS kafka_precalculated_events_ws

    DROP TABLE IF EXISTS precalculated_person_properties_ws_mv

    DROP TABLE IF EXISTS kafka_precalculated_person_properties_ws

    DROP TABLE IF EXISTS cohort_membership_ws_mv

    DROP TABLE IF EXISTS kafka_cohort_membership_ws

[trunk-io]

   

View Full Report ↗︎ ⋅ Docs