chore(clickhouse): drop precalculated warpstream kafka engine tables by gustavohstrassburger · Pull Request #79807 · PostHog/posthog

gustavohstrassburger · 2026-08-07T18:12:46Z

Problem

Nobody can decommission the warpstream-calculated-events cluster while ClickHouse still holds Kafka engine tables against it. Those tables serve no one: both precalculated topics lost their producers, so the four objects reading them consume nothing.

They cannot just be left behind. A Kafka engine table whose named collection has been removed from the ClickHouse config error-loops on every poll, so the tables have to go before the collection does.

Changes

Drops the Kafka engine table and its materialized view for each of the two precalculated topics.

Topic	Kafka engine table
`clickhouse_prefiltered_events`	`kafka_precalculated_events_ws`
`clickhouse_precalculated_person_properties`	`kafka_precalculated_person_properties_ws`

Within each pair the materialized view is dropped first, so it stops feeding its writable table before the Kafka table disappears.

Both drops are unconditional, because 0229 created both pairs unconditionally. 0243 later dropped the events pair everywhere and recreated it cloud-only, since off-cloud warpstream_calculated_events resolves to the same brokers as msk_cluster and the pair double-writes. That migration left kafka_precalculated_person_properties_ws behind, which this drop finally removes. Every statement is DROP TABLE IF EXISTS, so the migration is idempotent either way.

kafka_cohort_membership_ws reads from the same collection but stays. It is dropped separately, so its removal lands with the cohort work that owns it rather than inside a migration about precalculated tables.

Data tables are untouched. precalculated_events and precalculated_person_properties both stay, as do the MSK-fed Kafka tables, which resolve a different named collection and are unaffected by this cluster going away.

How did you test this code?

hogli ci:preflight --strict on the push: ruff lint, ruff format, and the migration conflict check all pass.
Confirmed all four imported DROP_*_SQL helpers exist on master, so the migration needs no model change.

Not done: the migration was never run against a real ClickHouse, and no automated test exercises it.

Automatic notifications

Publish to changelog?

Docs update

None.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Authored with Claude Code (PostHog Code). Skills invoked: /clickhouse-migrations for the original migration, /writing-pr-descriptions for this body. Also read posthog/clickhouse/migrations/AGENTS.md for the node-role and guard conventions.

The migration started as cohort-membership-only, widened to all three topics on the same named collection, then narrowed to the two precalculated pairs so the cohort drop can ship on its own schedule.

Kept migration-only per AGENTS.md. The matching consumer removal is a separate PR, and the now-unused SQL builders stay in place so no already-applied migration needs rewriting.

Part of a cross-repo decommission, with companion PRs in PostHog/charts and posthog-cloud-infra. This one is safe to land first and independently.

trunk-io · 2026-08-07T18:12:49Z

😎 Merged successfully - details.

gustavohstrassburger · 2026-08-07T18:13:12Z

@greptile review

greptile-apps · 2026-08-07T18:15:55Z

_{Reviews (1): Last reviewed commit: "chore(cohorts): drop warpstream-calculat..." | Re-trigger Greptile}

github-actions · 2026-08-07T18:18:23Z

🤖 CI report

⚠️

Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

⚠️

Playwright — 1 failed

🎭 Playwright report · View test results →

❌ 1 failed test:

Hover chart to see tooltip with data point values (chromium)

These issues are not necessarily caused by your changes.
Annoyed by this section? Help fix flakies and failures and it will go green!

⚠️

Backend snapshots — 1 updated (1 modified, 0 added, 0 deleted)

Query snapshots: Backend query snapshots updated

Changes: 1 snapshots (1 modified, 0 added, 0 deleted)

What this means:

Query snapshots have been automatically updated to match current output
These changes reflect modifications to database queries or schema

Next steps:

Review the query changes to ensure they're intentional
If unexpected, investigate what caused the query to change

Review snapshot changes →

ℹ️ ClickHouse migration SQL — 1 migration(s)

ClickHouse migration SQL per cloud environment

unset

all

DROP TABLE IF EXISTS precalculated_events_ws_mv

DROP TABLE IF EXISTS kafka_precalculated_events_ws

DROP TABLE IF EXISTS precalculated_person_properties_ws_mv

DROP TABLE IF EXISTS kafka_precalculated_person_properties_ws

US, EU, DEV

medium

DROP TABLE IF EXISTS precalculated_events_ws_mv

DROP TABLE IF EXISTS kafka_precalculated_events_ws

DROP TABLE IF EXISTS precalculated_person_properties_ws_mv

DROP TABLE IF EXISTS kafka_precalculated_person_properties_ws

trunk-io · 2026-08-07T18:34:54Z

_{View Full Report ↗︎ ⋅ Docs}

greptile-apps · 2026-08-11T19:49:46Z

_{Reviews (2): Last reviewed commit: "chore(cohorts): drop warpstream-calculat..." | Re-trigger Greptile}

Copilot

Pull request overview

Drops ClickHouse Kafka engine tables and their materialized views that point at the warpstream_calculated_events named collection, so the warpstream-calculated-events WarpStream cluster can be decommissioned without Kafka-engine poll error loops.

Changes:

Add ClickHouse migration 0296 to drop the two precalculated _ws Kafka/MV pairs unconditionally, and the cohort-membership _ws pair behind the same cloud-only guard used when creating it.
Update max_migration.txt to point to the new migration.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
posthog/clickhouse/migrations/max_migration.txt	Bumps ClickHouse migration pointer to `0296_*`.
posthog/clickhouse/migrations/0296_drop_warpstream_calculated_events_kafka_engines.py	Drops WarpStream-calculated-events Kafka engine tables and MVs (MV-first ordering; cloud guard preserved for cohort membership).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

gustavohstrassburger · 2026-08-11T20:13:52Z

+    else [
+        run_sql_with_exceptions(
+            "DROP TABLE IF EXISTS cohort_membership_ws_mv",
+            node_roles=[NodeRole.INGESTION_MEDIUM],
+        ),
+        run_sql_with_exceptions(
+            "DROP TABLE IF EXISTS kafka_cohort_membership_ws",
+            node_roles=[NodeRole.INGESTION_MEDIUM],
+        ),


Good catch — fixed in 361df20. Switched both to DROP_COHORT_MEMBERSHIP_WS_MV_SQL() / DROP_COHORT_MEMBERSHIP_WS_KAFKA_TABLE_SQL().

The emitted SQL is byte-identical, so no behaviour change, and it now mirrors migration 0245 which created these objects using the same module.

greptile-apps · 2026-08-11T20:11:51Z

_{Reviews (3): Last reviewed commit: "chore(cohorts): drop warpstream-calculat..." | Re-trigger Greptile}

gustavohstrassburger · 2026-08-11T20:13:53Z

@greptile review

greptile-apps · 2026-08-11T20:17:15Z

_{Reviews (4): Last reviewed commit: "chore(cohorts): use shared cohort member..." | Re-trigger Greptile}

haacked

Looks good. The drop set is complete and nothing on master reads these objects. One non-blocking suggestion inline.

One thing for the companion infra PR rather than this one: 0229 creates its WS Kafka tables unconditionally and hard-depends on the warpstream_calculated_events named collection, so once that collection leaves an environment's config, a fresh bootstrap replaying migrations from zero fails at 0229. That's pre-existing on master, but worth timing the collection removal around.

gustavohstrassburger · 2026-08-13T16:47:13Z

hey @PostHog/clickhouse, can I get a stamp here please?

gustavohstrassburger · 2026-08-18T16:32:41Z

hey @rorylshanks, since you were involved in this, do you mind giving this a look?

Generated-By: PostHog Code Task-Id: 08dc6a1f-fda7-4722-af9e-a292f90c1f09

The migration drops kafka_precalculated_person_properties_ws and its MV, but the HCL still declared them, so the convergence gate saw the migrated cluster diverge from the golden schema. Removes both from the ingestion_medium local layer and regenerates golden/ and sql/. Only the local goldens move: the events pair is already absent off-cloud, so nothing else in the composition changes. Generated-By: PostHog Desktop Task-Id: 6664f40e-4d5f-4818-89a8-b9ebdcefc718

gustavohstrassburger · 2026-08-27T19:06:05Z

/trunk merge

deployment-status-posthog · 2026-08-27T20:23:36Z

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-08-27 20:23 UTC	Run
prod-us	✅ Deployed	2026-08-27 20:45 UTC	Run
prod-eu	✅ Deployed	2026-08-28 07:22 UTC	Run

pr-assigner-resolver-posthog Bot assigned gustavohstrassburger Aug 7, 2026

github-actions Bot added feature/cohorts Feature Tag: Cohorts team/feature-flags labels Aug 7, 2026

gustavohstrassburger mentioned this pull request Aug 7, 2026

chore(cohorts): remove the cdp cohort membership consumer #79808

Closed

1 task

github-project-automation Bot added this to Feature Flags Aug 7, 2026

gustavohstrassburger mentioned this pull request Aug 7, 2026

chore(cohorts): remove warpstream-calculated-events cohort membership path #79799

Closed

1 task

haacked moved this to Todo in Feature Flags Aug 10, 2026

haacked mentioned this pull request Aug 10, 2026

Sprint August 10 to August 21 #78117

Closed

gustavohstrassburger force-pushed the posthog-code/drop-warpstream-calculated-events-kafka-engines branch 2 times, most recently from a275685 to 485db5f Compare August 11, 2026 17:23

gustavohstrassburger marked this pull request as ready for review August 11, 2026 19:46

gustavohstrassburger requested a review from a team as a code owner August 11, 2026 19:46

Copilot AI lite review requested due to automatic review settings August 11, 2026 19:46

Copilot started reviewing on behalf of gustavohstrassburger August 11, 2026 19:47

View session

gustavohstrassburger marked this pull request as draft August 11, 2026 19:47

Copilot AI reviewed Aug 11, 2026

View reviewed changes

gustavohstrassburger force-pushed the posthog-code/drop-warpstream-calculated-events-kafka-engines branch from 485db5f to 0df5491 Compare August 11, 2026 20:09

gustavohstrassburger marked this pull request as ready for review August 11, 2026 20:09

gustavohstrassburger moved this from Todo to In Review in Feature Flags Aug 11, 2026

haacked approved these changes Aug 12, 2026

View reviewed changes

Comment thread posthog/clickhouse/migrations/0297_drop_warpstream_calculated_events_kafka_engines.py Resolved

gustavohstrassburger force-pushed the posthog-code/drop-warpstream-calculated-events-kafka-engines branch from ab0e113 to da2d825 Compare August 12, 2026 21:03

gustavohstrassburger mentioned this pull request Aug 13, 2026

Decommission warpstream-calculated-events: precalculated tables and cohort membership #82624

Open

6 of 11 tasks

haacked mentioned this pull request Aug 25, 2026

Sprint - August 24 to September 4 #84504

Closed

gustavohstrassburger changed the title ~~chore(cohorts): drop warpstream-calculated-events kafka engine tables~~ chore(clickhouse): drop precalculated warpstream kafka engine tables Aug 26, 2026

rorylshanks approved these changes Aug 27, 2026

View reviewed changes

github-project-automation Bot moved this from In Review to Approved in Feature Flags Aug 27, 2026

gustavohstrassburger added 2 commits August 27, 2026 10:22

chore(cohorts): drop warpstream-calculated-events kafka engine tables

a4989ef

Generated-By: PostHog Code Task-Id: 08dc6a1f-fda7-4722-af9e-a292f90c1f09

chore(clickhouse): scope warpstream drop to precalculated kafka engines

2f7ae8f

gustavohstrassburger force-pushed the posthog-code/drop-warpstream-calculated-events-kafka-engines branch from 7e42a7a to 2f7ae8f Compare August 27, 2026 13:22

tests-posthog Bot and others added 3 commits August 27, 2026 13:40

test(backend): update query snapshots

2e80ef6

test(backend): update query snapshots

e26335a

trunk-io Bot merged commit 06cf6d7 into master Aug 27, 2026
240 checks passed

trunk-io Bot deleted the posthog-code/drop-warpstream-calculated-events-kafka-engines branch August 27, 2026 19:16

github-project-automation Bot moved this from Approved to Done in Feature Flags Aug 27, 2026

haacked mentioned this pull request Sep 9, 2026

Sprint - September 7 to September 18 #92782

Closed

Conversation

gustavohstrassburger commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Changes

How did you test this code?

Automatic notifications

Docs update

🤖 Agent context

Uh oh!

trunk-io Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gustavohstrassburger commented Aug 7, 2026

Uh oh!

greptile-apps Bot commented Aug 7, 2026

Uh oh!

github-actions Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 CI report

Query snapshots: Backend query snapshots updated

ClickHouse migration SQL per cloud environment

Uh oh!

trunk-io Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Aug 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

gustavohstrassburger Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot commented Aug 11, 2026

Uh oh!

gustavohstrassburger commented Aug 11, 2026

Uh oh!

greptile-apps Bot commented Aug 11, 2026

Uh oh!

haacked left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gustavohstrassburger commented Aug 13, 2026

Uh oh!

gustavohstrassburger commented Aug 18, 2026

Uh oh!

gustavohstrassburger commented Aug 27, 2026

Uh oh!

Uh oh!

deployment-status-posthog Bot commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploy status

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gustavohstrassburger commented Aug 7, 2026 •

edited

Loading

trunk-io Bot commented Aug 7, 2026 •

edited

Loading

github-actions Bot commented Aug 7, 2026 •

edited

Loading

trunk-io Bot commented Aug 7, 2026 •

edited

Loading

deployment-status-posthog Bot commented Aug 27, 2026 •

edited

Loading