diff --git a/docs/internal/clickhouse-deletion-coverage.md b/docs/internal/clickhouse-deletion-coverage.md index 7d00b6a7c8fe..4fe059dc33a8 100644 --- a/docs/internal/clickhouse-deletion-coverage.md +++ b/docs/internal/clickhouse-deletion-coverage.md @@ -95,7 +95,7 @@ Leaving one out stays possible, and `PERSON_ID_REWRITE_EXEMPT` is where that dec - `sharded_events` — all sweeps. - `sharded_events_json` — all sweeps, on the events cluster, through patch parts instead of mutations (`uses_patch_parts`). Optional: only present after the native-JSON migration. -- `sharded_flag_evaluations` — person, team, queued-uuid and deferred event removal. Not immediate event removal or property removal (below). Optional. +- `sharded_flag_evaluations` — person, team, queued-uuid, deferred event removal, and property removal of event `properties` (below). Not immediate event removal, and not property removal with a HogQL predicate. Optional. - `sharded_posthog_document_embeddings_` — event and team deletion, through `delete_event_documents`. An embedded document is keyed by the id of the thing it describes (`document_id`), and an Event deletion's key is that same id, so the pending dictionary is joined on `(team_id, Event, document_id)`. Every per-model table listed by the error tracking facade's `document_embedding_tables` is swept and counted. Native property-removal requests fail when the selected rows retain a requested permanent or temporary property, or a matching person `$set`/`$set_once` instruction. @@ -146,59 +146,73 @@ So a run can mark requests verified without proving the rows are gone. What stop removing nothing is upstream of the count: `MutationRunner.reuse_since` keeps a run from adopting a mutation an earlier run enqueued, which is the failure the count was added to notice. -### Property removal does not reach `flag_evaluations` +### Property removal on `flag_evaluations` + +The property-removal job sweeps `sharded_flag_evaluations` with the same per-shard copy, delete, reingest and verify ops it runs on the events tables. +The copy writes each shard's matching rows to S3 with the named keys dropped and each affected extracted column reset, the delete removes the originals, and the reingest inserts the cleaned copy back. +That works because `materialize()` creates columns as `DEFAULT `, so an insert can set the column directly. +`DeletionTarget.accepts_property_rewrite` marks each table the job sweeps, and the gate refuses while a table without it holds rows a request names. `person_properties` and `group0..group4_properties` no longer exist on the table: no Insight or Hog function used either as a breakdown or a filter, so the ClickHouse team dropped them directly on both prod clusters, and `posthog/models/flag_evaluations/sql.py` no longer declares them, so any environment built from the migrations matches. Event `properties` and `person_id` are still sent. -Because the table can no longer hold person properties, only the event-`properties` half of a request can match rows here. +Because the table can no longer hold person properties, only the event-`properties` half of a request applies here. +`DeletionTarget.stores_person_properties` is `False` on `FLAG_EVALUATIONS`, so the job drops the `person_properties` half for this table, and a request that names only person properties does not touch it. +That is accurate for rows written since the producer stopped sending `person_properties` (2026-09-05, #95693), and a deliberate blind spot for whatever a row written before then still carries: those values are out of reach until the row's TTL passes. -The events property-removal path copies each shard's matching rows to S3 with the keys dropped and each affected materialized column reset (`NULL` when nullable, `''` otherwise), then deletes the originals and inserts the cleaned copy back. -The staged files retain every physical row, while copy, reingest and verification progress counts each event UUID once so unmerged `ReplacingMergeTree` duplicates do not block the request. -That works because `materialize()` creates columns as `DEFAULT `, so an insert can set the column directly. +#### Typed columns + +Migration `0301_flag_evaluations_default_columns` recreated the nine typed columns as `DEFAULT `, computed from `properties` when an insert omits them. +Each carries the `column_materializer::` comment the events columns carry, on both the shard and the proxy, so the job discovers them on the `flag_evaluations` read table the same way it discovers materialized columns on `events`. +The copy writes each affected typed column as `''`, which is also what the column's expression computes for an absent key. + +A request naming `$feature_flag` therefore completes: the cleaned row goes back with `flag_key = ''`, and per-flag queries stop counting it. +`flag_key` is in the table's sort key, where ClickHouse refuses any `ALTER TABLE … UPDATE` (`Cannot UPDATE key column 'flag_key'`, measured on 26.6.2), but the copy replaces the value in a `SELECT`, so the sort key is no obstacle. + +The same names on `sharded_events` (`$session_id`, `$window_id`, `$group_0..4`) are true `MATERIALIZED` columns with that comment. +The copy reads the `SELECT *` shape of the table, which leaves `MATERIALIZED` columns out, and the reingest recomputes them from the cleaned `properties`. -All of that machinery (column discovery, staged rewrite, shard walk) is scoped to `events`; none of it reaches `flag_evaluations`. -Until it does, `get_property_removal_shards` refuses to start when the table holds rows matching a request's event `properties`, so such a request cannot complete while data it named survives. -The check costs nothing while the table is empty. +#### Replay duplicates -The person-property half of a request is different, whether or not it also names event properties: `DeletionTarget.stores_person_properties` is `False` on `FLAG_EVALUATIONS`, so the gate does not build a `person_properties` predicate against the table at all, and that half of the request completes regardless of what the column holds. -That is accurate for rows written since the producer stopped sending `person_properties` (2026-09-05, #95693), and a deliberate blind spot for whatever a row written before then still carries: those values are out of the gate's reach until the row's TTL passes. +`sharded_flag_evaluations` is a plain `MergeTree` that keeps Kafka replay duplicates, so one uuid can have several identical rows. +The staged files keep every physical row, so the reingest puts every duplicate back, cleaned. +The copy, the delete's count, the reingest and the shard verify count each uuid once, so duplicates do not block the request. +The same counting covers unmerged `ReplacingMergeTree` duplicates on events. -The schema stopped being a second obstacle with migration `0301_flag_evaluations_default_columns`, which recreated the nine typed columns as `DEFAULT `, the kind `materialize()` mints on events; they were true ClickHouse `MATERIALIZED` before, which is not assignable at all. -Measured against ClickHouse 26.6.2 on the `DEFAULT` shape: +#### Rows past the TTL -- `CREATE TABLE` accepts a `DEFAULT`-from-`properties` column (`flag_key`) in the sort key, and an insert that omits the typed columns computes them from `properties`. -- Assigning to a non-key typed column is accepted: `ALTER TABLE … UPDATE session_id = ''` completes. Under `MATERIALIZED` it was rejected with `Cannot UPDATE materialized column 'session_id'`. -- Updating `properties` is accepted, alone and in the events-path form that resets affected typed columns in the same mutation. - The `MATERIALIZED`-era rejection (`Updated column 'properties' affects MATERIALIZED column 'flag_key', which is a key column`) does not fire for `DEFAULT` dependents. -- An `UPDATE` of `properties` does not recompute the typed columns; rows keep their stored values. - The rewrite must reset each affected column explicitly, exactly as the events path already does. -- `flag_key` itself can never be reset: `ALTER TABLE … UPDATE flag_key = ''` is rejected with `Cannot UPDATE key column 'flag_key'` (`CANNOT_UPDATE_COLUMN`), whatever the column kind. - A request naming `$feature_flag` therefore still cannot be honored by mutation; that one property needs a refusal, or the heavier rewrite: `INSERT … SELECT` the cleaned rows omitting the typed columns so the shard recomputes them, then lightweight-delete the originals. +A request window can reach rows past the 90-day TTL that a part still holds, because `ttl_only_drop_parts = 1` drops a part only once its newest row expires. +The copy stages those rows like any others. +A reingested part that holds only expired rows can then drop before the reingest or the shard verify counts it. +`DeletionTarget.ttl_days` makes both checks compare only rows at least two days short of their TTL, on the table side and the staged side alike. +A part that holds one of those rows cannot drop, so every counted row is still there. +Each expired row that drops took its property with it. -The switch also changed two behaviors, measured on the same shape: +#### HogQL predicates -- `SELECT *` on the shard now includes the nine typed columns, where `MATERIALIZED` hid them. Nothing in the repo depended on the hidden shape. -- An insert that names a typed column stores the given value even when it contradicts `properties`, where `MATERIALIZED` rejected such inserts. - Producers must omit the columns; the Kafka path enforces that because `writable_flag_evaluations` does not declare them. +`compile_hogql_predicate` compiles against the events schema only (see the HogQL section below), so the job cannot apply a request's predicate to this table. +A property removal with a predicate leaves `flag_evaluations` out of the sweep, and `get_property_removal_shards` and `verify_property_removal` refuse the request while the table holds rows that match its other criteria. +Sweeping the table without the predicate would erase rows the predicate excludes. -The remaining fix is pointing the events rewrite machinery at this table, with the `$feature_flag` limitation above built into whatever it does here. +#### Late arrivals -`flag_evaluations` stays deliberately absent from `MATERIALIZATION_VALID_TABLES` until that lands: new `materialize()`-minted columns would only widen what the unfixed path silently leaves behind. +`flag_evaluations_mv` stamps `inserted_at` with `now64()` when it processes a row, as the events MV does. +A row that the consumer reads after the marker therefore falls outside the request, as on events. +The stamp comes before the Distributed forward to the shards, so a row stamped just before the marker can still reach a shard after the copy has read it. +That window is the forward delay that the events tables also have, not the consumer lag. -#### If a request arrives before the fix lands +#### Other readers -Today the refusal costs nothing, because the table is empty. -Once it holds rows, a property removal with `delete_all_events` refuses whenever a single flag-evaluation row carries the named event property, and the operator has no way through: `delete_all_events` and `events` are mutually exclusive on the model, so the request cannot be narrowed to exclude `$feature_flag_called`, and the admin Retry button replays the same failure. -The only exits are waiting out the TTL or shipping the fix above. The table partitions by month with `ttl_only_drop_parts = 1`, so a part drops only once its newest row expires: the real wait is up to about 120 days, not the 90-day TTL. `posthog/models/flag_evaluations/sql.py` says the same thing next to the partition clause. -A person-property-only request is not stuck this way: as above, the gate does not check `flag_evaluations` for one at all. +Cleaned rows carry `inserted_at = marker`, as on events. +A reader that checkpoints `flag_evaluations` on `inserted_at` sees a rewritten row again if the marker is later than its checkpoint. +The copy carries every other column through unchanged. +`flag_evaluations_backfill_job` does not copy a day while a data deletion request run is queued or executing, so it does not re-insert an original the job removed. -Refusing beats silently under-deleting, so the gate is the right default. -If the fix has not landed by the time real traffic hits, the cheaper stopgaps are letting a request exclude event names so an operator can scope around the table, or recording an explicit, audited acknowledgement on the request so an operator can accept the residue rather than being stuck. -Doing nothing means the first affected GDPR request becomes an escalation. +`flag_evaluations` stays absent from `MATERIALIZATION_VALID_TABLES`. +The job finds typed columns through their comments in `system.columns`, not through that set, and adding the table there enrolls it in `materialize()` and the HogQL property planner, which is a separate change. ### Immediate event removal skips `flag_evaluations` `get_event_removal_shards` leaves `flag_evaluations` out of its targets, so an immediate request neither sweeps the table nor checks it for matching rows. -The rows age out with the table's TTL, which in practice is up to about 120 days (see above). +The rows age out with the table's TTL. The table partitions by month with `ttl_only_drop_parts = 1`, so a part drops only once its newest row expires: the real wait is up to about 120 days, not the 90-day TTL. Deferred event removal still queues the table's uuids, and `deletes_job` removes them. The skip exists because of the HogQL gap below: before it, the gate refused every immediate request with a predicate whose team had matching `$feature_flag_called` rows. @@ -242,7 +256,7 @@ When one of those runs starts during a copy, the shard stops, and its error name ## Adding a table -Register it in `PERSONAL_DATA_TARGETS`, with capability flags reflecting what its schema can actually take and what the sweep code actually implements: `accepts_property_rewrite` needs the rewrite machinery to reach the table, not just assignable columns; `stores_person_properties` needs the table's `person_properties` column to actually hold reachable data, not just exist in the schema; see `FLAG_EVALUATIONS` for a table where those diverged. +Register it in `PERSONAL_DATA_TARGETS`, with capability flags reflecting what its schema can actually take and what the sweep code actually implements: `accepts_property_rewrite` makes the property-removal job sweep the table, so it needs a rewrite that cleans every copy the table keeps, not just assignable columns; `stores_person_properties` needs the table's `person_properties` column to actually hold reachable data, not just exist in the schema; see `FLAG_EVALUATIONS` for a table that rewrites event properties without person properties. Set `accepts_person_id_rewrite` unless you mean to leave the table out of the squash, which needs `person_id` outside the table's sorting and partition keys; skipping it is what stranded `flag_evaluations` rows on a merged-away person (#93035). A table you do leave out goes in `PERSON_ID_REWRITE_EXEMPT` with the reason. If it is not going to be swept, add it to `TTL_ONLY_TABLES` with the window you are accepting. diff --git a/posthog/api/test/dashboards/__snapshots__/test_dashboard.ambr b/posthog/api/test/dashboards/__snapshots__/test_dashboard.ambr index 1ca1694203d5..a941bc3f4581 100644 --- a/posthog/api/test/dashboards/__snapshots__/test_dashboard.ambr +++ b/posthog/api/test/dashboards/__snapshots__/test_dashboard.ambr @@ -12757,17 +12757,6 @@ ''' # --- # name: TestDashboard.test_retrieve_dashboard.12 - ''' - SELECT "django_content_type"."id", - "django_content_type"."app_label", - "django_content_type"."model" - FROM "django_content_type" - WHERE ("django_content_type"."app_label" = 'dashboards' - AND "django_content_type"."model" = 'dashboard') - LIMIT 21 - ''' -# --- -# name: TestDashboard.test_retrieve_dashboard.13 ''' SELECT "posthog_taggeditem"."id", "posthog_taggeditem"."tag_id", @@ -12788,7 +12777,7 @@ 5 /* ... */)) ''' # --- -# name: TestDashboard.test_retrieve_dashboard.14 +# name: TestDashboard.test_retrieve_dashboard.13 ''' SELECT "posthog_team"."id", "posthog_team"."uuid", @@ -12887,7 +12876,7 @@ LIMIT 21 ''' # --- -# name: TestDashboard.test_retrieve_dashboard.15 +# name: TestDashboard.test_retrieve_dashboard.14 ''' SELECT "posthog_filesystem"."team_id", "posthog_filesystem"."id", @@ -12912,7 +12901,7 @@ AND "posthog_filesystem"."shortcut" IS NOT NULL)) ''' # --- -# name: TestDashboard.test_retrieve_dashboard.16 +# name: TestDashboard.test_retrieve_dashboard.15 ''' SELECT "posthog_filesystemshortcut"."id", "posthog_filesystemshortcut"."team_id", @@ -12935,7 +12924,7 @@ AND "posthog_filesystemshortcut"."href" IS NOT NULL)) ''' # --- -# name: TestDashboard.test_retrieve_dashboard.17 +# name: TestDashboard.test_retrieve_dashboard.16 ''' SELECT "posthog_insightvariable"."id", "posthog_insightvariable"."name", @@ -12955,6 +12944,368 @@ ORDER BY "posthog_insightvariable"."name" ASC ''' # --- +# name: TestDashboard.test_retrieve_dashboard.17 + ''' + SELECT "posthog_dashboardtile"."id", + "posthog_dashboardtile"."dashboard_id", + "posthog_dashboardtile"."insight_id", + "posthog_dashboardtile"."text_id", + "posthog_dashboardtile"."button_tile_id", + "posthog_dashboardtile"."widget_id", + "posthog_dashboardtile"."team_id", + "posthog_dashboardtile"."layouts", + "posthog_dashboardtile"."color", + "posthog_dashboardtile"."filters_hash", + "posthog_dashboardtile"."last_refresh", + "posthog_dashboardtile"."refreshing", + "posthog_dashboardtile"."refresh_attempt", + "posthog_dashboardtile"."filters_overrides", + "posthog_dashboardtile"."show_description", + "posthog_dashboardtile"."transparent_background", + "posthog_dashboardtile"."deleted", + "posthog_dashboarditem"."id", + "posthog_dashboarditem"."name", + "posthog_dashboarditem"."derived_name", + "posthog_dashboarditem"."description", + "posthog_dashboarditem"."team_id", + "posthog_dashboarditem"."filters", + "posthog_dashboarditem"."filters_hash", + "posthog_dashboarditem"."query", + "posthog_dashboarditem"."query_metadata", + "posthog_dashboarditem"."order", + "posthog_dashboarditem"."deleted", + "posthog_dashboarditem"."saved", + "posthog_dashboarditem"."created_at", + "posthog_dashboarditem"."refreshing", + "posthog_dashboarditem"."created_by_id", + "posthog_dashboarditem"."is_sample", + "posthog_dashboarditem"."short_id", + "posthog_dashboarditem"."favorited", + "posthog_dashboarditem"."refresh_attempt", + "posthog_dashboarditem"."last_modified_at", + "posthog_dashboarditem"."last_modified_by_id", + "posthog_dashboarditem"."dashboard_id", + "posthog_dashboarditem"."last_refresh", + "posthog_dashboarditem"."layouts", + "posthog_dashboarditem"."color", + "posthog_dashboarditem"."dive_dashboard_id", + "posthog_dashboarditem"."updated_at", + "posthog_dashboarditem"."deprecated_tags", + "posthog_dashboarditem"."tags", + "posthog_team"."id", + "posthog_team"."uuid", + "posthog_team"."organization_id", + "posthog_team"."parent_team_id", + "posthog_team"."project_id", + "posthog_team"."api_token", + "posthog_team"."app_urls", + "posthog_team"."name", + "posthog_team"."slack_incoming_webhook", + "posthog_team"."created_at", + "posthog_team"."updated_at", + "posthog_team"."anonymize_ips", + "posthog_team"."completed_snippet_onboarding", + "posthog_team"."has_completed_onboarding_for", + "posthog_team"."onboarding_tasks", + "posthog_team"."ingested_event", + "posthog_team"."ingested_production_event", + "posthog_team"."ingested_production_event_last_checked_at", + "posthog_team"."person_processing_opt_out", + "posthog_team"."secret_api_token", + "posthog_team"."secret_api_token_backup", + "posthog_team"."session_recording_opt_in", + "posthog_team"."session_recording_sample_rate", + "posthog_team"."session_recording_minimum_duration_milliseconds", + "posthog_team"."session_recording_linked_flag", + "posthog_team"."session_recording_network_payload_capture_config", + "posthog_team"."session_recording_masking_config", + "posthog_team"."session_recording_url_trigger_config", + "posthog_team"."session_recording_url_blocklist_config", + "posthog_team"."session_recording_event_trigger_config", + "posthog_team"."session_recording_trigger_match_type_config", + "posthog_team"."session_recording_trigger_groups", + "posthog_team"."session_replay_config", + "posthog_team"."session_recording_retention_period", + "posthog_team"."event_retention_months", + "posthog_team"."conversations_enabled", + "posthog_team"."conversations_settings", + "posthog_team"."proactive_tasks_enabled", + "posthog_team"."survey_config", + "posthog_team"."surveys_opt_in", + "posthog_team"."product_tours_opt_in", + "posthog_team"."capture_console_log_opt_in", + "posthog_team"."capture_performance_opt_in", + "posthog_team"."capture_dead_clicks", + "posthog_team"."autocapture_opt_out", + "posthog_team"."autocapture_web_vitals_opt_in", + "posthog_team"."autocapture_web_vitals_allowed_metrics", + "posthog_team"."autocapture_exceptions_opt_in", + "posthog_team"."autocapture_exceptions_errors_to_ignore", + "posthog_team"."logs_settings", + "posthog_team"."llm_gateway_enabled_at", + "posthog_team"."llm_gateway_revoked_at", + "posthog_team"."llm_gateway_overspend_allowance_usd", + "posthog_team"."heatmaps_opt_in", + "posthog_team"."receive_org_level_activity_logs", + "posthog_team"."web_analytics_pre_aggregated_tables_enabled", + "posthog_team"."web_analytics_pre_aggregated_tables_version", + "posthog_team"."flags_persistence_default", + "posthog_team"."feature_flag_confirmation_enabled", + "posthog_team"."feature_flag_confirmation_message", + "posthog_team"."default_evaluation_environments_enabled", + "posthog_team"."require_evaluation_environment_tags", + "posthog_team"."default_evaluation_contexts_enabled", + "posthog_team"."require_evaluation_contexts", + "posthog_team"."session_recording_version", + "posthog_team"."signup_token", + "posthog_team"."is_demo", + "posthog_team"."access_control", + "posthog_team"."week_start_day", + "posthog_team"."inject_web_apps", + "posthog_team"."test_account_filters", + "posthog_team"."test_account_filters_default_checked", + "posthog_team"."path_cleaning_filters", + "posthog_team"."timezone", + "posthog_team"."data_attributes", + "posthog_team"."person_display_name_properties", + "posthog_team"."live_events_columns", + "posthog_team"."recording_domains", + "posthog_team"."human_friendly_comparison_periods", + "posthog_team"."cookieless_server_hash_mode", + "posthog_team"."primary_dashboard_id", + "posthog_team"."default_data_theme", + "posthog_team"."extra_settings", + "posthog_team"."modifiers", + "posthog_team"."correlation_config", + "posthog_team"."session_recording_retention_period_days", + "posthog_team"."plugins_opt_in", + "posthog_team"."opt_out_capture", + "posthog_team"."event_names", + "posthog_team"."event_names_with_usage", + "posthog_team"."event_properties", + "posthog_team"."event_properties_with_usage", + "posthog_team"."event_properties_numerical", + "posthog_team"."external_data_workspace_id", + "posthog_team"."external_data_workspace_last_synced_at", + "posthog_team"."api_query_rate_limit", + "posthog_team"."revenue_tracking_config", + "posthog_team"."drop_events_older_than", + "posthog_team"."base_currency", + "posthog_team"."business_model", + "posthog_user"."id", + "posthog_user"."password", + "posthog_user"."last_login", + "posthog_user"."first_name", + "posthog_user"."last_name", + "posthog_user"."is_staff", + "posthog_user"."date_joined", + "posthog_user"."uuid", + "posthog_user"."current_organization_id", + "posthog_user"."current_team_id", + "posthog_user"."email", + "posthog_user"."pending_email", + "posthog_user"."distinct_id", + "posthog_user"."is_email_verified", + "posthog_user"."credentials_reviewed_at", + "posthog_user"."requested_password_reset_at", + "posthog_user"."requested_2fa_reset_at", + "posthog_user"."has_seen_product_intro_for", + "posthog_user"."strapi_id", + "posthog_user"."is_active", + "posthog_user"."role_at_organization", + "posthog_user"."theme_mode", + "posthog_user"."partial_notification_settings", + "posthog_user"."anonymize_data", + "posthog_user"."allow_impersonation", + "posthog_user"."toolbar_mode", + "posthog_user"."hedgehog_config", + "posthog_user"."allow_sidebar_suggestions", + "posthog_user"."shortcut_position", + "posthog_user"."passkeys_enabled_for_2fa", + "posthog_user"."hide_mcp_hints", + "posthog_user"."ui_configuration", + "posthog_user"."onboarding_skipped_at", + "posthog_user"."onboarding_skipped_reason", + "posthog_user"."onboarding_skipped_organization_id", + "posthog_user"."onboarding_delegated_to_invite_id", + "posthog_user"."onboarding_delegated_to_organization_id", + "posthog_user"."onboarding_delegation_accepted_at", + "posthog_user"."events_column_config", + "posthog_user"."email_opt_in", + T6."id", + T6."password", + T6."last_login", + T6."first_name", + T6."last_name", + T6."is_staff", + T6."date_joined", + T6."uuid", + T6."current_organization_id", + T6."current_team_id", + T6."email", + T6."pending_email", + T6."distinct_id", + T6."is_email_verified", + T6."credentials_reviewed_at", + T6."requested_password_reset_at", + T6."requested_2fa_reset_at", + T6."has_seen_product_intro_for", + T6."strapi_id", + T6."is_active", + T6."role_at_organization", + T6."theme_mode", + T6."partial_notification_settings", + T6."anonymize_data", + T6."allow_impersonation", + T6."toolbar_mode", + T6."hedgehog_config", + T6."allow_sidebar_suggestions", + T6."shortcut_position", + T6."passkeys_enabled_for_2fa", + T6."hide_mcp_hints", + T6."ui_configuration", + T6."onboarding_skipped_at", + T6."onboarding_skipped_reason", + T6."onboarding_skipped_organization_id", + T6."onboarding_delegated_to_invite_id", + T6."onboarding_delegated_to_organization_id", + T6."onboarding_delegation_accepted_at", + T6."events_column_config", + T6."email_opt_in", + "posthog_text"."id", + "posthog_text"."body", + "posthog_text"."agent_context", + "posthog_text"."created_by_id", + "posthog_text"."last_modified_at", + "posthog_text"."last_modified_by_id", + "posthog_text"."team_id", + "posthog_buttontile"."id", + "posthog_buttontile"."url", + "posthog_buttontile"."text", + "posthog_buttontile"."placement", + "posthog_buttontile"."style", + "posthog_buttontile"."created_by_id", + "posthog_buttontile"."last_modified_at", + "posthog_buttontile"."last_modified_by_id", + "posthog_buttontile"."team_id", + "posthog_dashboardwidget"."id", + "posthog_dashboardwidget"."widget_type", + "posthog_dashboardwidget"."name", + "posthog_dashboardwidget"."description", + "posthog_dashboardwidget"."config", + "posthog_dashboardwidget"."created_by_id", + "posthog_dashboardwidget"."last_modified_at", + "posthog_dashboardwidget"."last_modified_by_id", + "posthog_dashboardwidget"."team_id", + T10."id", + T10."password", + T10."last_login", + T10."first_name", + T10."last_name", + T10."is_staff", + T10."date_joined", + T10."uuid", + T10."current_organization_id", + T10."current_team_id", + T10."email", + T10."pending_email", + T10."distinct_id", + T10."is_email_verified", + T10."credentials_reviewed_at", + T10."requested_password_reset_at", + T10."requested_2fa_reset_at", + T10."has_seen_product_intro_for", + T10."strapi_id", + T10."is_active", + T10."role_at_organization", + T10."theme_mode", + T10."partial_notification_settings", + T10."anonymize_data", + T10."allow_impersonation", + T10."toolbar_mode", + T10."hedgehog_config", + T10."allow_sidebar_suggestions", + T10."shortcut_position", + T10."passkeys_enabled_for_2fa", + T10."hide_mcp_hints", + T10."ui_configuration", + T10."onboarding_skipped_at", + T10."onboarding_skipped_reason", + T10."onboarding_skipped_organization_id", + T10."onboarding_delegated_to_invite_id", + T10."onboarding_delegated_to_organization_id", + T10."onboarding_delegation_accepted_at", + T10."events_column_config", + T10."email_opt_in", + T11."id", + T11."password", + T11."last_login", + T11."first_name", + T11."last_name", + T11."is_staff", + T11."date_joined", + T11."uuid", + T11."current_organization_id", + T11."current_team_id", + T11."email", + T11."pending_email", + T11."distinct_id", + T11."is_email_verified", + T11."credentials_reviewed_at", + T11."requested_password_reset_at", + T11."requested_2fa_reset_at", + T11."has_seen_product_intro_for", + T11."strapi_id", + T11."is_active", + T11."role_at_organization", + T11."theme_mode", + T11."partial_notification_settings", + T11."anonymize_data", + T11."allow_impersonation", + T11."toolbar_mode", + T11."hedgehog_config", + T11."allow_sidebar_suggestions", + T11."shortcut_position", + T11."passkeys_enabled_for_2fa", + T11."hide_mcp_hints", + T11."ui_configuration", + T11."onboarding_skipped_at", + T11."onboarding_skipped_reason", + T11."onboarding_skipped_organization_id", + T11."onboarding_delegated_to_invite_id", + T11."onboarding_delegated_to_organization_id", + T11."onboarding_delegation_accepted_at", + T11."events_column_config", + T11."email_opt_in" + FROM "posthog_dashboardtile" + INNER JOIN "posthog_dashboard" ON ("posthog_dashboardtile"."dashboard_id" = "posthog_dashboard"."id") + LEFT OUTER JOIN "posthog_dashboarditem" ON ("posthog_dashboardtile"."insight_id" = "posthog_dashboarditem"."id") + LEFT OUTER JOIN "posthog_team" ON ("posthog_dashboarditem"."team_id" = "posthog_team"."id") + LEFT OUTER JOIN "posthog_user" ON ("posthog_dashboarditem"."created_by_id" = "posthog_user"."id") + LEFT OUTER JOIN "posthog_user" T6 ON ("posthog_dashboarditem"."last_modified_by_id" = T6."id") + LEFT OUTER JOIN "posthog_text" ON ("posthog_dashboardtile"."text_id" = "posthog_text"."id") + LEFT OUTER JOIN "posthog_buttontile" ON ("posthog_dashboardtile"."button_tile_id" = "posthog_buttontile"."id") + LEFT OUTER JOIN "posthog_dashboardwidget" ON ("posthog_dashboardtile"."widget_id" = "posthog_dashboardwidget"."id") + LEFT OUTER JOIN "posthog_user" T10 ON ("posthog_dashboardwidget"."created_by_id" = T10."id") + LEFT OUTER JOIN "posthog_user" T11 ON ("posthog_dashboardwidget"."last_modified_by_id" = T11."id") + WHERE (NOT ("posthog_dashboardtile"."deleted" + AND "posthog_dashboardtile"."deleted" IS NOT NULL) + AND NOT ("posthog_dashboard"."deleted") + AND NOT ("posthog_dashboard"."deleted" + AND "posthog_dashboardtile"."deleted" + AND "posthog_dashboardtile"."deleted" IS NOT NULL) + AND (NOT "posthog_dashboarditem"."deleted" + OR "posthog_dashboardtile"."insight_id" IS NULL) + AND "posthog_dashboardtile"."dashboard_id" = 99999 + AND "posthog_dashboardtile"."dashboard_id" = 99999 + AND NOT ("posthog_dashboard"."deleted" + AND "posthog_dashboardtile"."deleted" + AND "posthog_dashboardtile"."deleted" IS NOT NULL) + AND (NOT "posthog_dashboarditem"."deleted" + OR "posthog_dashboardtile"."insight_id" IS NULL)) + ORDER BY "posthog_dashboarditem"."order" ASC + ''' +# --- # name: TestDashboard.test_retrieve_dashboard.18 ''' SELECT "posthog_dashboardtile"."id", diff --git a/posthog/dags/data_deletion_requests.py b/posthog/dags/data_deletion_requests.py index 8128ef5352dc..31e5cfda6ead 100644 --- a/posthog/dags/data_deletion_requests.py +++ b/posthog/dags/data_deletion_requests.py @@ -1,5 +1,6 @@ import json import time +import logging from collections.abc import Callable, Collection, Iterator from dataclasses import dataclass, field, replace from datetime import UTC, datetime, timedelta @@ -27,7 +28,7 @@ from posthog.clickhouse.adhoc_events_deletion import ADHOC_EVENTS_DELETION_TABLE from posthog.clickhouse.client import sync_execute -from posthog.clickhouse.client.connection import ClickHouseUser +from posthog.clickhouse.client.connection import ClickHouseUser, NodeRole from posthog.clickhouse.cluster import ( ClickhouseCluster, LightweightDeleteMutationRunner, @@ -56,10 +57,10 @@ ) from posthog.models.deletion_targets import ( COVERAGE_DOC, - EVENTS_JSON, FLAG_EVALUATIONS, PERSONAL_DATA_TARGETS, DeletionTarget, + NotCounted, TargetPlacement, UnsweepableRowsError, UnsweptRowsError, @@ -67,8 +68,9 @@ assert_sweep_complete, placement_for, resolve_placements, + surviving_rows_sql, ) -from posthog.models.event.sql import DISTRIBUTED_EVENTS_JSON_TABLE, json_property_presence_expr +from posthog.models.event.sql import json_property_presence_expr from posthog.models.person.bulk_delete import ( PersonDeletionStep, delete_persons_profile, @@ -324,8 +326,8 @@ def _get_affected_mat_columns( The comment itself is a sufficient identifier — it is PostHog-specific and the ``elements_chain::*`` family is excluded explicitly. - Callers pass ``"events"`` deliberately; see docs/internal/clickhouse-deletion-coverage.md for - why the other tables' typed columns cannot be discovered or reset here. + Callers pass the target's Distributed read table, because that is where events keeps the + comments. flag_evaluations declares the same comments on its read table for its typed columns. """ database = django_settings.CLICKHOUSE_DATABASE sql = """ @@ -535,9 +537,10 @@ def _refuse_unsweepable( cluster: ClickhouseCluster, targets: list[DeletionTarget], deletion_request: DeletionRequestContext, - predicate_for: Callable[[DeletionTarget], tuple[str, dict] | None], + predicate_for: Callable[[DeletionTarget], tuple[str, dict] | NotCounted], *, reason: str, + log: logging.Logger | None = None, ) -> None: """Raise a dagster.Failure when the request would strand rows on any of ``targets``.""" try: @@ -547,36 +550,73 @@ def _refuse_unsweepable( predicate_for, events=[] if deletion_request.delete_all_events else deletion_request.events, reason=reason, + log=log, ) except UnsweepableRowsError as exc: raise dagster.Failure(description=f"Deletion request {deletion_request.request_id}: {exc}") from exc +_STORES_NONE_OF_THE_PROPERTIES = "the request names only person properties, and this table stores none" + + +def _deletion_target(data_table: str) -> DeletionTarget: + return next(target for target in PERSONAL_DATA_TARGETS if target.data_table == data_table) + + +def _scoped_to(target: DeletionTarget, deletion_request: DeletionRequestContext) -> DeletionRequestContext: + """The property-removal request narrowed to the property columns ``target`` stores. + + A target without ``stores_person_properties`` (today only flag_evaluations) drops the + request's ``person_properties`` half, since that column cannot be queried or assigned there; + see the field's own comment on ``DeletionTarget`` for what that costs. + """ + return deletion_request if target.stores_person_properties else replace(deletion_request, person_properties=[]) + + +def _names_any_property(request: DeletionRequestContext) -> bool: + return bool(request.properties or request.person_properties) + + +def _hogql_excludes(target: DeletionTarget, deletion_request: DeletionRequestContext) -> bool: + """Whether the request carries a HogQL predicate that does not compile against ``target``.""" + return bool(deletion_request.hogql_predicate) and not target.accepts_hogql_predicate + + def _refuse_property_removal_unsweepable( cluster: ClickhouseCluster, - unsweepable: list[DeletionTarget], + targets: list[DeletionTarget], deletion_request: DeletionRequestContext, marker_str: str, + *, + log: logging.Logger, ) -> None: - """Gate a property-removal request against every unsweepable target. + """Gate a property-removal request against every one of ``targets`` the rewrite cannot complete. - A target without ``stores_person_properties`` (today only flag_evaluations) drops the - request's ``person_properties`` half from its presence check, since the gate cannot query that - column there; see the field's own comment on ``DeletionTarget`` for what that costs. A target - left with no criteria at all, nothing in ``properties`` either, is skipped rather than - queried. + That is a target the job does not rewrite, and a target the request's HogQL predicate excludes, + which ``_property_rewrite_targets`` leaves out of the sweep. """ - def predicate_for(target: DeletionTarget) -> tuple[str, dict] | None: - request = ( - deletion_request if target.stores_person_properties else replace(deletion_request, person_properties=[]) - ) - if not request.properties and not request.person_properties: - return None + def predicate_for(target: DeletionTarget) -> tuple[str, dict] | NotCounted: + request = _scoped_to(target, deletion_request) + if not _names_any_property(request): + return NotCounted(reason=_STORES_NONE_OF_THE_PROPERTIES) return _property_removal_where(request, inserted_at_max=marker_str, json_schema=target.uses_new_events_schema) _refuse_unsweepable( - cluster, unsweepable, deletion_request, predicate_for, reason=_PROPERTY_REWRITE_UNSWEEPABLE_REASON + cluster, + [t for t in targets if not t.accepts_property_rewrite], + deletion_request, + predicate_for, + reason=_PROPERTY_REWRITE_UNSWEEPABLE_REASON, + log=log, + ) + _refuse_unsweepable( + cluster, + [t for t in targets if t.accepts_property_rewrite and _hogql_excludes(t, deletion_request)], + deletion_request, + predicate_for, + reason=_HOGQL_UNSWEEPABLE_REASON, + log=log, ) @@ -639,7 +679,7 @@ class EventRemovalShard: def _placement_for_table(cluster: ClickhouseCluster, data_table: str) -> TargetPlacement: - target = next(t for t in PERSONAL_DATA_TARGETS if t.data_table == data_table) + target = _deletion_target(data_table) placements = resolve_placements(cluster, [target]) if not placements: raise dagster.Failure(description=f"{data_table} is not present on any reachable cluster") @@ -653,7 +693,7 @@ def _verify_immediate_event_deletion( deleted_shards: list[EventRemovalShard], ) -> None: tables = list(dict.fromkeys(shard.data_table for shard in deleted_shards)) - targets = [next(t for t in PERSONAL_DATA_TARGETS if t.data_table == table) for table in tables] + targets = [_deletion_target(table) for table in tables] if any(target.uses_patch_parts for target in targets): wait_for_patch_part_replication() _verify_swept( @@ -823,12 +863,9 @@ def complete_event_deletion( _PROPERTY_REWRITE_UNSWEEPABLE_REASON = ( - "the property-rewrite machinery is scoped to the events tables and does not reach it; a " - "request naming $feature_flag additionally cannot " - "be honored by mutation at all, because flag_key sits in the table's sort key where no UPDATE " - "can reset it. There is no way to complete this request today: " - "either narrow its events to ones this table never stores, or wait out the table's TTL. " - f"See {COVERAGE_DOC}." + "the property rewrite does not clean every copy of a property this table keeps. The native-JSON " + "events table also retains temporary properties and quarantine diagnostics, which the rewrite " + f"leaves in place. See {COVERAGE_DOC}." ) @@ -920,6 +957,10 @@ class PropertyRemovalTarget: def mapping_key(self) -> str: return f"{self.table}_shard_{self.shard}" + @property + def deletion_target(self) -> DeletionTarget: + return _deletion_target(self.table) + @frozen class _ShardStaging: @@ -965,15 +1006,25 @@ def finish_step(self, client: Client, step: str, payload: dict) -> None: {"step": step, "payload": json.dumps(payload)}, ) - def count_staged_uuids(self, client: Client, months: list[str]) -> dict[str, int]: + def count_staged_uuids( + self, client: Client, months: list[str], where: str = "1", params: dict | None = None + ) -> dict[str, int]: if not months: return {} rows = client.execute( - f"SELECT _file, uniqExact(uuid) FROM s3({self.data_args(months)}) GROUP BY _file", + f"SELECT _file, uniqExact(uuid) FROM s3({self.data_args(months)}) WHERE {where} GROUP BY _file", + params, settings=_LONG_QUERY_SETTINGS, ) return {file.removesuffix(".native"): count for file, count in rows} + def unexpired_uuids(self, client: Client, months: dict[str, int], where: str, params: dict) -> dict[str, int]: + """Per month, the staged uuids that a restore check must find back in the table.""" + if self.target.deletion_target.ttl_days is None: + # The copy and the delete already matched the staged files to these counts. + return months + return self.count_staged_uuids(client, sorted(months), where, params) + def discard_step(self, client: Client, step: str) -> None: # ClickHouse cannot delete an S3 object, so the progress file is overwritten with zero rows. client.execute( @@ -1011,6 +1062,27 @@ def empty_data_files(self, client: Client) -> int: _LONG_QUERY_SETTINGS = {"max_execution_time": 86400} +# The TTL expires a row at the start of a day, so a row this many days short of it has a day left. +_TTL_CHECK_MARGIN_DAYS = 2 + + +def _unexpired_rows(client: Client, target: DeletionTarget) -> tuple[str, dict]: + """A filter for the rows that the target's TTL cannot drop while a restore check runs. + + With ttl_only_drop_parts, a reingested part that holds only expired rows can drop as soon as it + lands. The uuids it held then go missing from the count. A part that holds a row this filter + keeps cannot drop, so every kept row is still there to count. Each call fixes one cutoff, so the + table side and the staged side of a comparison leave out the same rows. + """ + if target.ttl_days is None: + return "1", {} + [[since]] = client.execute( + "SELECT toString(now64(6, 'UTC') - toIntervalDay(%(days)s))", + {"days": target.ttl_days - _TTL_CHECK_MARGIN_DAYS}, + ) + return "timestamp >= toDateTime64(%(unexpired_since)s, 6, 'UTC')", {"unexpired_since": since} + + def _datetime64_str(value: datetime) -> str: value = value.replace(tzinfo=UTC) if value.tzinfo is None else value.astimezone(UTC) return value.strftime("%Y-%m-%d %H:%M:%S.%f") @@ -1024,11 +1096,11 @@ def _marker_str(deletion_request: DeletionRequestContext) -> str: return _datetime64_str(deletion_request.inserted_at_marker) -def _compile_predicate(deletion_request: DeletionRequestContext, target: PropertyRemovalTarget) -> tuple[str, dict]: +def _compile_predicate(deletion_request: DeletionRequestContext, json_schema: bool) -> tuple[str, dict]: # HogQL compilation reaches into the Django ORM (Team lookup), so it runs on the op's own thread # before the shard work is dispatched to a cluster worker thread. Property access lowers # differently on the legacy and native-JSON tables, hence one compilation per schema. - return compile_hogql_predicate(deletion_request, use_new_events_schema=target.json_schema) + return compile_hogql_predicate(deletion_request, use_new_events_schema=json_schema) @frozen @@ -1043,10 +1115,10 @@ class _ShardPredicate: def _shard_predicate( client: Client, deletion_request: DeletionRequestContext, - target: PropertyRemovalTarget, + target: DeletionTarget, marker_str: str, hogql_compiled: tuple[str, dict], - log: QueryLogger, + log: QueryLogger | None = None, ) -> _ShardPredicate: """Build the predicate that selects this shard's originals. @@ -1058,14 +1130,14 @@ def _shard_predicate( # through JSON subcolumns. mat_cols: list[tuple[str, bool]] = [] person_mat_cols: list[tuple[str, bool]] = [] - if not target.json_schema: + if not target.uses_new_events_schema: if deletion_request.properties: mat_cols = _get_affected_mat_columns( - client, "events", deletion_request.properties, table_column="properties", log=log + client, target.read_table, deletion_request.properties, table_column="properties", log=log ) if deletion_request.person_properties: person_mat_cols = _get_affected_mat_columns( - client, "events", deletion_request.person_properties, table_column="person_properties", log=log + client, target.read_table, deletion_request.person_properties, table_column="person_properties", log=log ) sql, params = _property_removal_where( deletion_request, @@ -1073,7 +1145,7 @@ def _shard_predicate( person_mat_cols=person_mat_cols, inserted_at_max=marker_str, hogql_compiled=hogql_compiled, - json_schema=target.json_schema, + json_schema=target.uses_new_events_schema, ) return _ShardPredicate(sql=sql, params=params, mat_cols=mat_cols + person_mat_cols) @@ -1121,14 +1193,19 @@ def _cleaned_select_list( "AND default_kind NOT IN ('MATERIALIZED', 'ALIAS', 'EPHEMERAL') ORDER BY position", {"db": django_settings.CLICKHOUSE_DATABASE, "table": target.table}, ) + [[engine]] = client.execute( + "SELECT engine FROM system.tables WHERE database = %(db)s AND name = %(table)s", + {"db": django_settings.CLICKHOUSE_DATABASE, "table": target.table}, + ) params: dict = {"inserted_at_marker": marker_str} - replacements: dict[str, str] = { - "inserted_at": "toDateTime64(%(inserted_at_marker)s, 6, 'UTC')", - # Bump the ReplacingMergeTree version (ver=_timestamp) past the original's, so a merge that - # meets a cleaned row and an original with the same sorting key keeps the cleaned row. +1 - # second because _timestamp is second-precision while the marker is microsecond-precision. - "_timestamp": "toDateTime(toDateTime64(%(inserted_at_marker)s, 6, 'UTC')) + 1", - } + replacements: dict[str, str] = {"inserted_at": "toDateTime64(%(inserted_at_marker)s, 6, 'UTC')"} + # Bump the ReplacingMergeTree version (ver=_timestamp) past the original's, so a merge that + # meets a cleaned row and an original with the same sorting key keeps the cleaned row. +1 + # second because _timestamp is second-precision while the marker is microsecond-precision. + # A plain MergeTree such as flag_evaluations keeps the original value, because there _timestamp + # is the Kafka message time that flag_evaluations_backfill reads as the consumer position. + if "ReplacingMergeTree" in engine: + replacements["_timestamp"] = "toDateTime(toDateTime64(%(inserted_at_marker)s, 6, 'UTC')) + 1" # On the JSON table the column round-trips through a string: serialize it, drop the keys, and # let the cast below turn the cleaned string back into the JSON column type. if deletion_request.properties: @@ -1178,13 +1255,9 @@ def _run_on_shard(cluster: ClickhouseCluster, target: PropertyRemovalTarget, fn: return next(iter(result.values())) -def _deletion_target(target: PropertyRemovalTarget) -> DeletionTarget: - return next(t for t in PERSONAL_DATA_TARGETS if t.data_table == target.table) - - def _cluster_for(cluster: ClickhouseCluster, target: PropertyRemovalTarget) -> ClickhouseCluster: """The handle whose shards carry ``target``'s table, which for sharded_events_json is the events cluster.""" - placement = placement_for(cluster, _deletion_target(target)) + placement = placement_for(cluster, target.deletion_target) if placement is None: raise dagster.Failure(description=f"{target.table} is not present on any reachable cluster") return placement.cluster @@ -1197,6 +1270,30 @@ def log(label: str, sql: str) -> None: return log +def _property_rewrite_targets( + targets: list[DeletionTarget], deletion_request: DeletionRequestContext, *, log: logging.Logger +) -> list[DeletionTarget]: + """The tables out of ``targets`` that this request's sweep rewrites and its verification counts. + + A target the request's HogQL predicate excludes is left out, and + ``_refuse_property_removal_unsweepable`` refuses the request while such a target holds rows the + request names. + """ + events = [] if deletion_request.delete_all_events else deletion_request.events + swept: list[DeletionTarget] = [] + for target in targets: + if not target.accepts_property_rewrite or _hogql_excludes(target, deletion_request): + continue + if not target.may_hold_any_of(events): + log.info(f"{target.read_table}: not rewritten, because it stores none of the request's events") + continue + if not _names_any_property(_scoped_to(target, deletion_request)): + log.info(f"{target.read_table}: not rewritten, because {_STORES_NONE_OF_THE_PROPERTIES}") + continue + swept.append(target) + return swept + + @dagster.op(out=dagster.DynamicOut(PropertyRemovalTarget), tags=OWNER_TAG) def get_property_removal_shards( context: dagster.OpExecutionContext, @@ -1213,13 +1310,14 @@ def get_property_removal_shards( in this op rather than the load op because this is the first one holding a cluster handle. """ placements = resolve_placements(cluster) - unsweepable = [p.target for p in placements if not p.target.accepts_property_rewrite] - if unsweepable: - # Bound by the same marker as the sweep and the verify gate, so a row ingested after the - # marker, which the sweep would never touch, cannot refuse the request forever. - _refuse_property_removal_unsweepable(cluster, unsweepable, deletion_request, _marker_str(deletion_request)) + # Bound by the same marker as the sweep and the verify gate, so a row ingested after the + # marker, which the sweep would never touch, cannot refuse the request forever. + _refuse_property_removal_unsweepable( + cluster, [p.target for p in placements], deletion_request, _marker_str(deletion_request), log=context.log + ) - rewritten = [p for p in placements if p.target.accepts_property_rewrite] + swept = _property_rewrite_targets([p.target for p in placements], deletion_request, log=context.log) + rewritten = [p for p in placements if p.target in swept] context.log.info( f"Fanning out property removal {deletion_request.request_id} to " + ", ".join(f"{p.target.data_table} x {len(p.cluster.shards)} shard(s)" for p in rewritten) @@ -1255,7 +1353,7 @@ def _copy_property_removal_target( ) _sync_replica(client, target, log) - predicate = _shard_predicate(client, deletion_request, target, marker_str, hogql_compiled, log) + predicate = _shard_predicate(client, deletion_request, target.deletion_target, marker_str, hogql_compiled, log) inserted_at_sql = f"SELECT maxOrNull(inserted_at) FROM {db}.{target.table} WHERE {predicate.sql}" log("max-inserted-at", inserted_at_sql) [[inserted_at_max]] = client.execute(inserted_at_sql, predicate.params, settings=_LONG_QUERY_SETTINGS) @@ -1323,12 +1421,13 @@ def copy_property_removal_shard( ClickHouse, and the write overwrites each monthly file. """ marker_str = _marker_str(deletion_request) - hogql_compiled = _compile_predicate(deletion_request, target) + hogql_compiled = _compile_predicate(deletion_request, target.json_schema) + request = _scoped_to(target.deletion_target, deletion_request) staging = _ShardStaging(request_id=deletion_request.request_id, target=target) log = _query_logger(context, target) def copy(client: Client) -> dict: - return _copy_property_removal_target(client, deletion_request, target, marker_str, hogql_compiled, staging, log) + return _copy_property_removal_target(client, request, target, marker_str, hogql_compiled, staging, log) copied = _run_on_shard(_cluster_for(cluster, target), target, copy) context.add_output_metadata({"copied": dagster.MetadataValue.int(copied["rows"])}) @@ -1348,7 +1447,8 @@ def delete_property_removal_shard( """ db = django_settings.CLICKHOUSE_DATABASE marker_str = _marker_str(deletion_request) - hogql_compiled = _compile_predicate(deletion_request, target) + hogql_compiled = _compile_predicate(deletion_request, target.json_schema) + request = _scoped_to(target.deletion_target, deletion_request) staging = _ShardStaging(request_id=deletion_request.request_id, target=target) log = _query_logger(context, target) @@ -1372,7 +1472,7 @@ def delete(client: Client) -> int: ) _sync_replica(client, target, log) - predicate = _shard_predicate(client, deletion_request, target, marker_str, hogql_compiled, log) + predicate = _shard_predicate(client, request, target.deletion_target, marker_str, hogql_compiled, log) predicate = _with_copied_inserted_at_bound(predicate, copied["inserted_at_max"]) count_sql = ( f"SELECT toString(toYYYYMM(timestamp)) AS month, uniqExact(uuid) " @@ -1425,7 +1525,7 @@ def delete(client: Client) -> int: parameters=predicate.params, settings={"lightweight_deletes_sync": 2, "mutations_sync": 2}, reuse_since=delete_since, - patch_parts=_deletion_target(target).uses_patch_parts, + patch_parts=target.deletion_target.uses_patch_parts, ) log("delete-originals", delete_runner.get_statement(delete_runner.get_all_commands())) # mutations_sync = 2 blocks on every replica of this shard; the explicit wait is a backstop. @@ -1471,6 +1571,10 @@ def reingest(client: Client) -> int: copied = steps[_COPIED] columns = ", ".join(f"`{name}`" for name in copied["columns"]) + # A failed earlier attempt may have inserted a month on another replica of this shard. Without + # that part here, the leftover count below misses those rows and the insert adds them a second + # time, which a plain MergeTree such as flag_evaluations never merges away. + _sync_replica(client, target, log) for month, expected in sorted(copied["months"].items()): if f"{_REINGESTED}_{month}" in steps: continue @@ -1498,7 +1602,7 @@ def reingest(client: Client) -> int: parameters=month_params, settings={"lightweight_deletes_sync": 2, "mutations_sync": 2}, reuse_since=clear_since, - patch_parts=_deletion_target(target).uses_patch_parts, + patch_parts=target.deletion_target.uses_patch_parts, ) log("clear-partial-reingest", clear_runner.get_statement(clear_runner.get_all_commands())) clear_runner(client).wait(client) @@ -1511,9 +1615,11 @@ def reingest(client: Client) -> int: # same blocks, so deduplication would silently drop the rows this month needs. client.execute(insert_sql, settings={**_LONG_QUERY_SETTINGS, "insert_deduplicate": 0}) + unexpired_sql, unexpired_params = _unexpired_rows(client, target.deletion_target) + expected = staging.unexpired_uuids(client, {month: expected}, unexpired_sql, unexpired_params).get(month, 0) stamped = client.execute( - f"SELECT uniqExact(uuid) FROM {db}.{target.table} WHERE {partial_rows}", - month_params, + f"SELECT uniqExact(uuid) FROM {db}.{target.table} WHERE {partial_rows} AND {unexpired_sql}", + {**month_params, **unexpired_params}, settings=_LONG_QUERY_SETTINGS, )[0][0] if stamped != expected: @@ -1541,7 +1647,8 @@ def verify_property_removal_shard( """Fail when this shard kept an original, lost a uuid, or holds a target property on a cleaned row.""" db = django_settings.CLICKHOUSE_DATABASE marker_str = _marker_str(deletion_request) - hogql_compiled = _compile_predicate(deletion_request, target) + hogql_compiled = _compile_predicate(deletion_request, target.json_schema) + request = _scoped_to(target.deletion_target, deletion_request) staging = _ShardStaging(request_id=deletion_request.request_id, target=target) log = _query_logger(context, target) @@ -1561,7 +1668,7 @@ def verify(client: Client) -> dict: return stats _sync_replica(client, target, log) - predicate = _shard_predicate(client, deletion_request, target, marker_str, hogql_compiled, log) + predicate = _shard_predicate(client, request, target.deletion_target, marker_str, hogql_compiled, log) predicate = _with_copied_inserted_at_bound(predicate, copied["inserted_at_max"]) remaining = client.execute( f"SELECT count() FROM {db}.{target.table} WHERE {predicate.sql}", @@ -1577,21 +1684,23 @@ def verify(client: Client) -> dict: "start_time": deletion_request.start_time, "end_time": deletion_request.end_time, "marker": marker_str, - **_presence_params(deletion_request), + **_presence_params(request), } - presence = _target_presence_clause(deletion_request, target, predicate.mat_cols) + presence = _target_presence_clause(request, target, predicate.mat_cols) + unexpired_sql, unexpired_params = _unexpired_rows(client, target.deletion_target) + expected_months = staging.unexpired_uuids(client, copied["months"], unexpired_sql, unexpired_params) cleaned_stats = client.execute( - f"SELECT toString(toYYYYMM(timestamp)) AS month, uniqExact(uuid), countIf({presence}) " + f"SELECT toString(toYYYYMM(timestamp)) AS month, uniqExactIf(uuid, {unexpired_sql}), countIf({presence}) " f"FROM {db}.{target.table} WHERE {cleaned_rows} GROUP BY month", - cleaned_params, + {**cleaned_params, **unexpired_params}, settings=_LONG_QUERY_SETTINGS, ) - cleaned_months = {month: count for month, count, _ in cleaned_stats} + cleaned_months = {month: count for month, count, _ in cleaned_stats if count} still_present = sum(present for _, _, present in cleaned_stats) - if remaining or still_present or cleaned_months != copied["months"]: + if remaining or still_present or cleaned_months != expected_months: raise dagster.Failure( description=f"[{target.mapping_key}] verification failed: {remaining} originals remain, " - f"cleaned={cleaned_months}, copied={copied['months']}, {still_present} cleaned rows still carry " + f"cleaned={cleaned_months}, expected={expected_months}, {still_present} cleaned rows still carry " "a target property. Investigate before re-running." ) staging.finish_step(client, _VERIFIED, {"rows": sum(cleaned_months.values())}) @@ -1646,78 +1755,47 @@ def verify_property_removal( Takes ``shard_stats`` (one dict per shard op) purely to sequence verification after every shard op has finished — a Dagster fan-in. - Checks each distributed events table for rows that still match the full removal predicate. - It uses the same ``inserted_at_max`` bound as the copy and delete passes, so post-marker - ingestion cannot wedge verification. A non-zero result means an original survived. + Checks the Distributed read table of every table the sweep rewrote for rows that still match + the full removal predicate. It uses the same ``inserted_at_max`` bound as the copy and delete + passes, so post-marker ingestion cannot wedge verification. A non-zero result means an original + survived. """ total_copied = sum(stats["copied"] for stats in shard_stats) context.log.info(f"All {len(shard_stats)} shard op(s) finished; {total_copied} events copied+cleaned in total") - - marker = deletion_request.inserted_at_marker - if marker is None: - raise dagster.Failure(description="property_removal_marker missing; load_property_removal_request must set it") - marker_str = marker.astimezone(UTC).strftime("%Y-%m-%d %H:%M:%S.%f") + marker_str = _marker_str(deletion_request) # Repeat the fan-out gate here. That one is point-in-time: rows can land between it and now, and # a re-execution from a failed shard reuses the fan-out op's cached output without re-running it. # Bounded by the same marker as the checks below so post-marker ingestion can't wedge the run. placements = resolve_placements(cluster) - unsweepable = [p.target for p in placements if not p.target.accepts_property_rewrite] - if unsweepable: - _refuse_property_removal_unsweepable(cluster, unsweepable, deletion_request, marker_str) - - properties = deletion_request.properties - person_properties = deletion_request.person_properties - targets: list[tuple[str, bool, tuple[str, dict]]] = [ - ("events", False, compile_hogql_predicate(deletion_request)), - ] - if any(p.target is EVENTS_JSON for p in placements): - targets.append( - ( - DISTRIBUTED_EVENTS_JSON_TABLE, - True, - compile_hogql_predicate(deletion_request, use_new_events_schema=True), - ) - ) + _refuse_property_removal_unsweepable( + cluster, [p.target for p in placements], deletion_request, marker_str, log=context.log + ) - def check( - client: Client, - table: str, - json_schema: bool, - hogql_compiled: tuple[str, dict], - ) -> int: - mat_cols = ( - _get_affected_mat_columns(client, table, properties, table_column="properties") - if properties and not json_schema - else [] - ) - person_mat_cols = ( - _get_affected_mat_columns(client, table, person_properties, table_column="person_properties") - if person_properties and not json_schema - else [] - ) - predicate, params = _property_removal_where( - deletion_request, - mat_cols=mat_cols, - person_mat_cols=person_mat_cols, - inserted_at_max=marker_str, - hogql_compiled=hogql_compiled, - json_schema=json_schema, - ) - remaining = client.execute( - f"SELECT count() FROM {table} WHERE {predicate} AND _row_exists = 1", - params, + def check(client: Client, target: DeletionTarget, hogql_compiled: tuple[str, dict]) -> int: + request = _scoped_to(target, deletion_request) + predicate = _shard_predicate(client, request, target, marker_str, hogql_compiled) + return client.execute( + surviving_rows_sql(target.read_table, predicate.sql), + predicate.params, settings={"max_execution_time": 1800}, )[0][0] - return remaining - if any(p.target.uses_patch_parts for p in placements): + swept = _property_rewrite_targets([p.target for p in placements], deletion_request, log=context.log) + if any(target.uses_patch_parts for target in swept): wait_for_patch_part_replication() - results = [ - cluster.any_host(partial(check, table=table, json_schema=json_schema, hogql_compiled=hogql_compiled)).result() - for table, json_schema, hogql_compiled in targets + pending = [ + cluster.any_host_by_role( + partial( + check, + target=target, + hogql_compiled=_compile_predicate(deletion_request, target.uses_new_events_schema), + ), + NodeRole.DATA, + ) + for target in swept ] - remaining = sum(results) + remaining = sum(future.result() for future in pending) context.add_output_metadata( { "remaining_originals": dagster.MetadataValue.int(remaining), diff --git a/posthog/dags/tests/test_data_deletion_requests.py b/posthog/dags/tests/test_data_deletion_requests.py index c0dcacb67d4f..48f955f50854 100644 --- a/posthog/dags/tests/test_data_deletion_requests.py +++ b/posthog/dags/tests/test_data_deletion_requests.py @@ -1,4 +1,5 @@ import json +import logging from concurrent.futures import Future from contextlib import nullcontext from dataclasses import replace @@ -69,6 +70,7 @@ RequestStatus, RequestType, auto_approve_pending_requests, + count_remaining_for_request, ) from posthog.models.deletion_targets import ( EVENTS, @@ -84,7 +86,11 @@ PERSON_PROPERTIES_JSON_TYPE, json_property_presence_expr, ) -from posthog.models.flag_evaluations.sql import FLAG_EVALUATIONS_DATA_TABLE, FLAG_EVALUATIONS_SOURCE_EVENT +from posthog.models.flag_evaluations.sql import ( + FLAG_EVALUATIONS_DATA_TABLE, + FLAG_EVALUATIONS_SOURCE_EVENT, + FLAG_EVALUATIONS_TTL_DAYS, +) from posthog.models.person.bulk_delete import PersonDeletionFailure, PersonDeletionStep, PersonProfileDeletionResult from posthog.test.persons import create_person @@ -134,13 +140,15 @@ def _truncate_writable_events(client: Client) -> None: def _insert_flag_evaluations_with_properties(rows: list[tuple], client: Client) -> None: - # Rows of (team_id, distinct_id, properties_json, uuid, timestamp, inserted_at). The event name is - # stamped here because a property-removal request narrows on it. inserted_at is set explicitly - # rather than left to its `DEFAULT timestamp` so a test can place a row before or after the - # removal marker, which is what the marker-bounded gate keys on. + # Rows of (team_id, distinct_id, properties_json, uuid, timestamp, inserted_at[, _timestamp]). The + # event name is stamped here because a property-removal request narrows on it. inserted_at is set + # explicitly rather than left to its `DEFAULT timestamp` so a test can place a row before or after + # the removal marker, which is what the marker-bounded gate keys on. + columns = "team_id, distinct_id, properties, uuid, timestamp, inserted_at" + if rows and len(rows[0]) == 7: + columns += ", _timestamp" client.execute( - "INSERT INTO writable_flag_evaluations " - "(team_id, distinct_id, properties, uuid, timestamp, inserted_at, event) VALUES", + f"INSERT INTO writable_flag_evaluations ({columns}, event) VALUES", [(*row, FLAG_EVALUATIONS_SOURCE_EVENT) for row in rows], ) @@ -1034,11 +1042,14 @@ def test_auto_approve_schedule_launches_a_run_on_tick(): def test_property_removal_cleaning_uses_unversioned_pool_udf() -> None: client = Mock(spec=Client) - client.execute.return_value = [ - ("properties", "String"), - ("person_properties", "String"), - ("inserted_at", "Nullable(DateTime64(6, 'UTC'))"), - ("_timestamp", "DateTime"), + client.execute.side_effect = [ + [ + ("properties", "String"), + ("person_properties", "String"), + ("inserted_at", "Nullable(DateTime64(6, 'UTC'))"), + ("_timestamp", "DateTime"), + ], + [("ReplicatedReplacingMergeTree",)], ] request = DeletionRequestContext( request_id=str(uuid4()), @@ -2534,16 +2545,18 @@ def test_deferred_event_removal_queues_flag_evaluations_and_blocks_promotion(clu @pytest.mark.django_db -def test_get_property_removal_shards_refuses_when_flag_evaluations_holds_matching_rows(cluster: ClickhouseCluster): - # Property removal cannot rewrite flag_evaluations: the rewrite machinery is scoped to the - # events tables, and flag_key sits in the sort key where no mutation can reset it, so even - # that machinery could not fully honor a request naming $feature_flag. Completing the - # request anyway would report the property erased while a copy of it survived. +def test_get_property_removal_shards_refuses_a_hogql_predicate_when_flag_evaluations_holds_matching_rows( + cluster: ClickhouseCluster, +): + # A HogQL predicate compiles only against the events schema, so the sweep cannot apply it to + # flag_evaluations. Sweeping the table without it would erase rows the predicate excludes, and + # skipping the table would report the property erased while a copy of it survived. request = DataDeletionRequest.objects.create( team_id=PROP_TEAM_ID, request_type=RequestType.PROPERTY_REMOVAL, events=[FLAG_EVALUATIONS_SOURCE_EVENT], properties=["$ip"], + hogql_predicate="properties.$browser = 'Chrome'", start_time=datetime.now() - timedelta(days=7), end_time=datetime.now() + timedelta(days=1), status=RequestStatus.APPROVED, @@ -2565,7 +2578,7 @@ def test_get_property_removal_shards_refuses_when_flag_evaluations_holds_matchin # Match the refusal itself, not just the table name: an UNKNOWN_TABLE error would also carry # "flag_evaluations" and would let a broken gate pass this test. - with pytest.raises(dagster.Failure, match="cannot be deleted"): + with pytest.raises(dagster.Failure, match="cannot be deleted: the request carries a HogQL predicate"): list(get_property_removal_shards(build_op_context(), cluster, deletion_request)) # A row ingested after the marker is outside the deletion snapshot, so the gate must ignore it @@ -2587,30 +2600,59 @@ def test_get_property_removal_shards_refuses_when_flag_evaluations_holds_matchin @pytest.mark.django_db @pytest.mark.parametrize( - "properties, person_properties, insert_value, expect_refusal", + "hogql_predicate, properties, person_properties, insert_value, error", [ - pytest.param([], ["email"], '{"email": "a@example.com"}', False, id="person_properties_only"), - pytest.param(["$ip"], ["email"], '{"email": "a@example.com"}', False, id="mixed_row_holds_only_the_person_key"), - pytest.param(["$ip"], ["email"], '{"$ip": "1.2.3.4"}', True, id="mixed_matches_event_property"), + pytest.param("", [], ["email"], '{"email": "a@example.com"}', None, id="person_properties_only"), + pytest.param( + "", ["$ip"], ["email"], '{"email": "a@example.com"}', None, id="mixed_row_holds_only_the_person_key" + ), + pytest.param( + "", + ["$ip"], + ["email"], + '{"$ip": "1.2.3.4"}', + "still match the removal predicate", + id="mixed_matches_event_property", + ), + pytest.param( + "properties.$browser = 'Chrome'", + [], + ["email"], + '{"email": "a@example.com"}', + None, + id="gated_person_properties_only", + ), + pytest.param( + "properties.$browser = 'Chrome'", + ["$ip"], + ["email"], + '{"$ip": "1.2.3.4"}', + "cannot be deleted", + id="gated_mixed_matches_event_property", + ), ], ) -def test_get_property_removal_shards_narrows_person_properties_on_flag_evaluations( +def test_verify_property_removal_narrows_person_properties_on_flag_evaluations( cluster: ClickhouseCluster, + team, + hogql_predicate: str, properties: list[str], person_properties: list[str], insert_value: str, - expect_refusal: bool, + error: str | None, ) -> None: - # flag_evaluations has no person_properties column (#95693), so the gate drops the - # person_properties half of its check. A row that holds only the key named there survives, which - # is the accepted cost, and a gate that kept that half would query a column the table does not - # have. The event-property half still applies, so a row matching it must still refuse. + # verify runs after the shard ops have rewritten rows, so a query error here leaves an erasure + # half done. flag_evaluations has no person_properties column (#95693), so both of its checks on + # the table drop that half, and a row matching the event-property half still fails the request. + # The admin Verify button marks a request completed when its count is zero, so that count must + # hold the same rows the job's verify does. request = DataDeletionRequest.objects.create( - team_id=PROP_TEAM_ID, + team_id=team.id, request_type=RequestType.PROPERTY_REMOVAL, events=[FLAG_EVALUATIONS_SOURCE_EVENT], properties=properties, person_properties=person_properties, + hogql_predicate=hogql_predicate, start_time=datetime.now() - timedelta(days=7), end_time=datetime.now() + timedelta(days=1), status=RequestStatus.APPROVED, @@ -2625,15 +2667,133 @@ def test_get_property_removal_shards_narrows_person_properties_on_flag_evaluatio cluster.any_host( partial( _insert_flag_evaluations_with_properties, - [(PROP_TEAM_ID, "someone", insert_value, str(uuid4()), before_marker, before_marker)], + [(team.id, "someone", insert_value, str(uuid4()), before_marker, before_marker)], ) ).result() - if expect_refusal: - with pytest.raises(dagster.Failure, match="cannot be deleted"): - list(get_property_removal_shards(build_op_context(), cluster, deletion_request)) - else: - assert list(get_property_removal_shards(build_op_context(), cluster, deletion_request)) + with pytest.raises(dagster.Failure, match=error) if error else nullcontext(): + verify_property_removal(build_op_context(), cluster, deletion_request, []) + assert count_remaining_for_request(request) == (1 if error else 0) + + cluster.any_host(_truncate_flag_evaluations).result() + + +def _flag_evaluation_rows(team_id: int, client: Client) -> list[tuple[str, dict, str, str, int]]: + rows = client.execute( + "SELECT toString(uuid), properties, flag_key, session_id, toUnixTimestamp(_timestamp) FROM flag_evaluations " + "WHERE team_id = %(team_id)s AND _row_exists = 1", + {"team_id": team_id}, + ) + return sorted( + ( + (uuid, json.loads(properties), flag_key, session_id, kafka_time) + for uuid, properties, flag_key, session_id, kafka_time in rows + ), + key=lambda row: row[0], + ) + + +@pytest.mark.django_db +@pytest.mark.parametrize( + "properties, person_properties", + [ + pytest.param(["$session_id"], [], id="typed_column"), + pytest.param(["$feature_flag"], [], id="sort_key_column"), + pytest.param(["$session_id"], ["email"], id="with_person_properties"), + ], +) +def test_full_job_property_removal_rewrites_flag_evaluations( + cluster: ClickhouseCluster, properties: list[str], person_properties: list[str] +) -> None: + now = timezone.now() + ingested = now - timedelta(hours=1) + kafka_time = ingested.replace(microsecond=0) + stored = {"$feature_flag": "beta", "$session_id": "s1", "keep": "yes"} + replayed_uuid, single_uuid = str(uuid4()), str(uuid4()) + rows = [ + (PROP_TEAM_ID, "someone", json.dumps(stored), uuid, ingested, ingested, kafka_time) + for uuid in (replayed_uuid, replayed_uuid, single_uuid) + ] + cluster.any_host(_truncate_flag_evaluations).result() + cluster.any_host(partial(_insert_flag_evaluations_with_properties, rows)).result() + + request = DataDeletionRequest.objects.create( + team_id=PROP_TEAM_ID, + request_type=RequestType.PROPERTY_REMOVAL, + events=[FLAG_EVALUATIONS_SOURCE_EVENT], + properties=properties, + person_properties=person_properties, + start_time=now - timedelta(days=7), + end_time=now + timedelta(minutes=1), + status=RequestStatus.APPROVED, + ) + result = data_deletion_request_property_removal.execute_in_process( + run_config={"ops": {"load_property_removal_request": {"config": {"request_id": str(request.pk)}}}}, + resources={"cluster": cluster}, + ) + assert result.success + + cleaned = {key: value for key, value in stored.items() if key not in properties} + # _timestamp keeps the Kafka message time, which flag_evaluations_backfill reads as the consumer position. + expected = [ + (uuid, cleaned, cleaned.get("$feature_flag", ""), cleaned.get("$session_id", ""), int(kafka_time.timestamp())) + for uuid in sorted((replayed_uuid, replayed_uuid, single_uuid)) + ] + assert cluster.any_host(partial(_flag_evaluation_rows, PROP_TEAM_ID)).result() == expected + request.refresh_from_db() + assert request.status == RequestStatus.COMPLETED + + cluster.any_host(_truncate_flag_evaluations).result() + + +@pytest.mark.django_db +def test_full_job_property_removal_survives_the_ttl_dropping_reingested_flag_evaluations( + cluster: ClickhouseCluster, +) -> None: + # Tests build the table without a TTL, so the patch deletes the expired row where the drop would. + now = timezone.now() + live_at = now - timedelta(hours=1) + expired_at = now - timedelta(days=FLAG_EVALUATIONS_TTL_DAYS + 5) + stored = json.dumps({"$session_id": "s1", "keep": "yes"}) + live_uuid, expired_uuid = str(uuid4()), str(uuid4()) + rows = [ + (PROP_TEAM_ID, "someone", stored, live_uuid, live_at, live_at), + (PROP_TEAM_ID, "someone", stored, expired_uuid, expired_at, expired_at), + ] + cluster.any_host(_truncate_flag_evaluations).result() + cluster.any_host(partial(_insert_flag_evaluations_with_properties, rows)).result() + + request = DataDeletionRequest.objects.create( + team_id=PROP_TEAM_ID, + request_type=RequestType.PROPERTY_REMOVAL, + events=[FLAG_EVALUATIONS_SOURCE_EVENT], + properties=["$session_id"], + start_time=now - timedelta(days=FLAG_EVALUATIONS_TTL_DAYS + 30), + end_time=now + timedelta(minutes=1), + status=RequestStatus.APPROVED, + ) + original = _ShardStaging.finish_step + + def drop_expired_after_reingest(self: _ShardStaging, client: Client, name: str, payload: dict) -> None: + original(self, client, name, payload) + if self.target.table == FLAG_EVALUATIONS_DATA_TABLE and name == "reingested": + client.execute( + f"DELETE FROM {FLAG_EVALUATIONS_DATA_TABLE} WHERE uuid = %(uuid)s", + {"uuid": expired_uuid}, + settings={"lightweight_deletes_sync": 2, "mutations_sync": 2}, + ) + + with patch.object(_ShardStaging, "finish_step", drop_expired_after_reingest): + result = data_deletion_request_property_removal.execute_in_process( + run_config={"ops": {"load_property_removal_request": {"config": {"request_id": str(request.pk)}}}}, + resources={"cluster": cluster}, + ) + assert result.success + + remaining = cluster.any_host(partial(_flag_evaluation_rows, PROP_TEAM_ID)).result() + assert [(uuid, properties) for uuid, properties, *_ in remaining] == [(live_uuid, {"keep": "yes"})] + request.refresh_from_db() + assert request.status == RequestStatus.COMPLETED cluster.any_host(_truncate_flag_evaluations).result() @@ -3020,7 +3180,10 @@ def execute_query(query: Query, _role: NodeRole) -> Future[list[tuple[object, .. cluster = Mock(spec=ClickhouseCluster) cluster.any_host_by_role.side_effect = execute_query with pytest.raises(dagster.Failure, match="cannot be deleted") if refuses else nullcontext(): - _refuse_property_removal_unsweepable(cluster, [EVENTS_JSON], request, marker) + # EVENTS_JSON accepts the rewrite, so the gate skips it. A copy without the flag reaches the + # gate's predicate. + unsweepable = replace(EVENTS_JSON, accepts_property_rewrite=False) + _refuse_property_removal_unsweepable(cluster, [unsweepable], request, marker, log=logging.getLogger(__name__)) @pytest.mark.parametrize( diff --git a/posthog/models/data_deletion_request.py b/posthog/models/data_deletion_request.py index 456e5dae5c89..2722d8bdebea 100644 --- a/posthog/models/data_deletion_request.py +++ b/posthog/models/data_deletion_request.py @@ -869,16 +869,13 @@ def _mat_col_presence_clauses(mat_cols: list[tuple[str, bool]]) -> list[str]: return [f"`{name}` != ''" for name, _ in mat_cols] -def discover_affected_mat_columns(properties: list[str], table_column: str) -> list[tuple[str, bool]]: - """DEFAULT-materialized columns on the distributed ``events`` table for the given properties. +def discover_affected_mat_columns(properties: list[str], table_column: str, table: str) -> list[tuple[str, bool]]: + """DEFAULT-materialized columns on the Distributed read table ``table`` for the given properties. Returns ``(column_name, is_nullable)`` for columns whose comment follows the ``column_materializer::::`` convention. Mirrors ``_get_affected_mat_columns`` in the deletion job so verification counts a row as dirty on the same terms the deletion does — a value left in a materialized column after its JSON key is gone still counts. - - Scoped to ``events`` deliberately, matching the deletion job; see - docs/internal/clickhouse-deletion-coverage.md. """ if not properties: return [] @@ -895,11 +892,11 @@ def discover_affected_mat_columns(properties: list[str], table_column: str) -> l SELECT name, comment, type LIKE 'Nullable(%%)' FROM system.columns WHERE database = %(database)s - AND table = 'events' + AND table = %(table)s AND comment LIKE '%%column_materializer::%%' AND comment NOT LIKE '%%column_materializer::elements_chain::%%' """, - {"database": django_settings.CLICKHOUSE_DATABASE}, + {"database": django_settings.CLICKHOUSE_DATABASE, "table": table}, readonly=True, ch_user=ClickHouseUser.META, ) @@ -916,6 +913,9 @@ def _property_presence_where( request: "DataDeletionRequest", mat_cols: list[tuple[str, bool]] | None = None, person_mat_cols: list[tuple[str, bool]] | None = None, + *, + with_person_properties: bool = True, + with_hogql: bool = True, ) -> tuple[str, dict]: """WHERE predicate + params matching events that still carry any target (person_)property. @@ -924,6 +924,9 @@ def _property_presence_where( property_removal request: once the property has been stripped from the JSON and its materialized column reset on every matching event, this count reaches zero. The presence set must match the deletion path (``_property_removal_where``) or a row it still considers dirty reads as clean here. + + ``with_person_properties`` and ``with_hogql`` drop the parts a table without the events schema + cannot run: its ``person_properties`` column and the compiled HogQL fragment. """ parts = [_EVENT_REMOVAL_TIME_PREDICATE, event_match_sql_fragment(request)] params = event_match_params(request) @@ -935,7 +938,7 @@ def _property_presence_where( params[f"fp_{i}_{j}"] = part if mat_cols: presence.extend(_mat_col_presence_clauses(mat_cols)) - for i, prop in enumerate(request.person_properties or []): + for i, prop in enumerate((request.person_properties or []) if with_person_properties else []): presence.append(jsonhas_expr(prop, f"pp_{i}", column="person_properties")) for j, part in enumerate(prop.split(".")): params[f"pp_{i}_{j}"] = part @@ -944,7 +947,7 @@ def _property_presence_where( if presence: parts.append(f"AND ({' OR '.join(presence)})") - hogql_sql, hogql_values = compile_hogql_predicate(request) + hogql_sql, hogql_values = compile_hogql_predicate(request) if with_hogql else ("", {}) if hogql_sql: parts.append(f"AND ({hogql_sql})") params.update(hogql_values) @@ -952,15 +955,28 @@ def _property_presence_where( def count_remaining_property_events(request: "DataDeletionRequest") -> int: - """Count events that still carry any of a property-removal request's target properties.""" + """Count rows that still carry any of a property-removal request's target properties. + + Counts across every default deletion target the property-removal job rewrites, because a request + is only complete once each of those tables is clean. A target without ``stores_person_properties`` + is checked for the event ``properties`` only, and skipped when the request names only person + properties, as the job does. A target that cannot take the compiled HogQL fragment is counted + without it, which matches a superset. That can only hold a request open, never complete it early. + The presence predicate is built for the legacy schema, so the native-JSON table is left to the + job's own verification. + """ from posthog.clickhouse.client import sync_execute from posthog.clickhouse.client.connection import ClickHouseUser from posthog.clickhouse.query_tagging import Feature, Product, tags_context from posthog.clickhouse.workload import Workload + from posthog.models.deletion_targets import ( + DEFAULT_DELETION_TARGETS, + resolve_read_targets_via_sync_execute, + surviving_rows_sql, + ) - mat_cols = discover_affected_mat_columns(request.properties or [], "properties") - person_mat_cols = discover_affected_mat_columns(request.person_properties or [], "person_properties") - predicate, params = _property_presence_where(request, mat_cols, person_mat_cols) + events = [] if request.delete_all_events else request.events + total = 0 with tags_context( product=Product.INTERNAL, feature=Feature.DATA_DELETION, @@ -968,22 +984,41 @@ def count_remaining_property_events(request: "DataDeletionRequest") -> int: workload=Workload.OFFLINE, query_type="data_deletion_request_verify_property", ): - # nosemgrep: clickhouse-fstring-param-audit (predicate built from internal helper, not user input) - result = sync_execute( - f"SELECT count() FROM events WHERE {predicate} AND _row_exists = 1", - params, - team_id=request.team_id, - readonly=True, - workload=Workload.OFFLINE, - ch_user=ClickHouseUser.META, - ) - return int(result[0][0]) if result else 0 + for target in resolve_read_targets_via_sync_execute(DEFAULT_DELETION_TARGETS): + if ( + not target.accepts_property_rewrite + or target.uses_new_events_schema + or not target.may_hold_any_of(events) + ): + continue + person_properties = (request.person_properties or []) if target.stores_person_properties else [] + if not request.properties and not person_properties: + continue + mat_cols = discover_affected_mat_columns(request.properties or [], "properties", target.read_table) + person_mat_cols = discover_affected_mat_columns(person_properties, "person_properties", target.read_table) + predicate, params = _property_presence_where( + request, + mat_cols, + person_mat_cols, + with_person_properties=target.stores_person_properties, + with_hogql=target.accepts_hogql_predicate, + ) + result = sync_execute( + surviving_rows_sql(target.read_table, predicate), + params, + team_id=request.team_id, + readonly=True, + workload=Workload.OFFLINE, + ch_user=ClickHouseUser.META, + ) + total += int(result[0][0]) if result else 0 + return total def count_remaining_for_request(request: "DataDeletionRequest") -> int | None: """Count rows still matching a deletion request's criteria in ClickHouse. - Dispatches on request type: matching events for event_removal, events still carrying the + Dispatches on request type: matching events for event_removal, rows still carrying the target property for property_removal. Returns ``None`` for person_removal, which has no automated remaining-count. """ diff --git a/posthog/models/deletion_targets.py b/posthog/models/deletion_targets.py index 9f52c4642cea..03ac042e3211 100644 --- a/posthog/models/deletion_targets.py +++ b/posthog/models/deletion_targets.py @@ -11,6 +11,7 @@ docs/internal/clickhouse-deletion-coverage.md. """ +import logging from collections.abc import Callable, Sequence from dataclasses import dataclass from enum import Enum @@ -34,6 +35,7 @@ FLAG_EVALUATIONS_DATA_TABLE, FLAG_EVALUATIONS_SOURCE_EVENT, FLAG_EVALUATIONS_TABLE, + FLAG_EVALUATIONS_TTL_DAYS, ) COVERAGE_DOC = "docs/internal/clickhouse-deletion-coverage.md" @@ -99,21 +101,17 @@ class DeletionTarget: # physical columns (mat_*, the property-group maps, or JSON subcolumns), so it only runs # against the schema it was compiled for. hogql_schema: HogQLSchema | None = None - # properties/person_properties can be rewritten in place. True needs both halves: the property - # columns are DEFAULT-kind (assignable by ALTER UPDATE, as materialize() mints them), and the - # property-rewrite machinery in posthog/dags/data_deletion_requests.py actually sweeps the - # table, which today is hardcoded to the events tables. flag_evaluations satisfies only the - # schema half, so flipping this without extending the sweep silently under-deletes. + # True where the property-removal job sweeps the table. Its verification fails while any copy of + # a named property that the table keeps survives the rewrite. A property removal refuses while a + # target without it holds rows the request names. accepts_property_rewrite: bool = False - # Whether the property-removal gate may build a person_properties predicate here. False where - # the column has been dropped out of band on PostHog Cloud, which makes the predicate an - # unknown-identifier error rather than a count. The flag narrows the gate on every deployment - # regardless of whether the column is actually present there. The cost is a blind spot: rows - # this table stored before its producer stopped sending person_properties hold real values - # wherever the column survives, and this gate no longer sees them. Bounded by the table's TTL; - # see COVERAGE_DOC. - # accepts_property_rewrite=True implies this must stay True too, since the rewrite assumes the - # column holds real data; __post_init__ below enforces that pairing. + # Whether property removal may build a person_properties predicate or replacement here. False + # where the column has been dropped out of band on PostHog Cloud, which makes the predicate an + # unknown-identifier error rather than a count. The flag narrows the gate and the rewrite on + # every deployment regardless of whether the column is actually present there. The cost is a + # blind spot: rows this table stored before its producer stopped sending person_properties hold + # real values wherever the column survives, and property removal no longer sees them. Bounded + # by the table's TTL; see COVERAGE_DOC. stores_person_properties: bool = True # Whether the person-overrides squash rewrites person_id here. A merge moves a distinct_id to # another person, and a later deletion names only the survivor, so a table the squash skips @@ -128,15 +126,20 @@ class DeletionTarget: # The event names this table can hold, None meaning unconstrained. Lets a request naming other # events skip this table without querying it. stored_events: frozenset[str] | None = None + # Days the TTL keeps a row past toDate(timestamp). Property removal's restore checks leave out + # the rows that the TTL can drop. + ttl_days: int | None = None # Deletes and person_id rewrites on this table write patch parts instead of mutations; see # MutationRunner.patch_parts. uses_patch_parts: bool = False def __post_init__(self) -> None: - if self.accepts_property_rewrite and not self.stores_person_properties: + # A table that takes a HogQL predicate has the events schema, including person_properties. + if self.accepts_property_rewrite and self.accepts_hogql_predicate and not self.stores_person_properties: raise ValueError( - f"{self.data_table}: accepts_property_rewrite needs stores_person_properties, " - f"because the rewrite writes the person_properties column back. See {COVERAGE_DOC}." + f"{self.data_table}: accepts_property_rewrite on an events-schema table needs " + f"stores_person_properties, because the rewrite must clean its person_properties " + f"column too. See {COVERAGE_DOC}." ) @property @@ -200,17 +203,20 @@ class TargetPlacement: # Flag-evaluation telemetry carries the same person_id and group payload as events, so team and # queued-uuid sweeps must reach it, and a person sweep matches on person_id like every other -# events-shaped table. Its producer stopped sending person_properties on 2026-09-05 (#95693); a row -# the table stored before then is out of the property-removal gate's reach until its TTL passes. It -# takes neither of the richer sweeps; both exclusions are explained in +# events-shaped table. Property removal rewrites its event properties. Its producer stopped sending +# person_properties on 2026-09-05 (#95693); a row the table stored before then is out of property +# removal's reach until its TTL passes. A HogQL predicate does not compile against the table, which +# limits immediate event removal and property removal there; both limits are explained in # docs/internal/clickhouse-deletion-coverage.md. FLAG_EVALUATIONS = DeletionTarget( data_table=FLAG_EVALUATIONS_DATA_TABLE, read_table=FLAG_EVALUATIONS_TABLE, optional=True, + accepts_property_rewrite=True, stores_person_properties=False, accepts_person_id_rewrite=True, stored_events=frozenset({FLAG_EVALUATIONS_SOURCE_EVENT}), + ttl_days=FLAG_EVALUATIONS_TTL_DAYS, ) EVENTS_TARGETS: tuple[DeletionTarget, ...] = (EVENTS, EVENTS_JSON) @@ -434,34 +440,48 @@ def count_surviving_rows(cluster: ClickhouseCluster, target: DeletionTarget, pre return int(rows[0][0]) if rows else 0 +_STORES_NONE_OF_THE_EVENTS = "it stores none of the request's events" + + +@dataclass(frozen=True) +class NotCounted: + """What ``predicate_for`` returns for a target that none of the request's criteria apply to.""" + + reason: str + + def assert_no_unsweepable_rows( cluster: ClickhouseCluster, targets: Sequence[DeletionTarget], - predicate_for: Callable[[DeletionTarget], tuple[str, dict] | None], + predicate_for: Callable[[DeletionTarget], tuple[str, dict] | NotCounted], *, events: Sequence[str], reason: str, + log: logging.Logger | None = None, ) -> None: """Fail when a target this request cannot sweep actually holds rows matching it. Pass targets already narrowed by ``resolve_targets``; this does not re-check presence. - ``predicate_for`` returns the criteria to count on a target, or ``None`` to skip it. + ``predicate_for`` returns the criteria to count on a target, or ``NotCounted`` to skip it. The criteria carry no HogQL fragment and no materialized-column arms, so each count is a superset of what the deletion would have removed. A caller may drop a criterion the target cannot hold, and the count is then a superset of the remainder. ``events`` is the request's event filter, empty meaning every event. + ``log`` gets an info line for each target skipped without a count, with the reason, so a + completed erasure records which targets went unchecked. Deliberately gated on rows existing rather than on the request's shape: an unconditional refusal would block every request of that shape from the day it lands, including the ones with nothing to strand. """ for target in targets: - if not target.may_hold_any_of(events): - continue - - criteria = predicate_for(target) - if criteria is None: + criteria = ( + predicate_for(target) if target.may_hold_any_of(events) else NotCounted(reason=_STORES_NONE_OF_THE_EVENTS) + ) + if isinstance(criteria, NotCounted): + if log: + log.info(f"{target.read_table}: not counted, because {criteria.reason}") continue predicate, params = criteria diff --git a/posthog/models/flag_evaluations/sql.py b/posthog/models/flag_evaluations/sql.py index c390cf44f82e..e9596cdef5c2 100644 --- a/posthog/models/flag_evaluations/sql.py +++ b/posthog/models/flag_evaluations/sql.py @@ -113,14 +113,13 @@ # # DEFAULT rather than MATERIALIZED, the kind materialize() mints on sharded_events: # both compute the expression when an insert omits the column, but only a DEFAULT -# column accepts ALTER UPDATE, which the events property-removal path relies on to -# reset extracted values whose source property was erased (see -# docs/internal/clickhouse-deletion-coverage.md). An UPDATE of properties does not -# recompute these columns, so a rewrite must reset each affected column in the -# same mutation. The cost is a footgun MATERIALIZED did not have: an insert that -# names one of these columns stores the given value even when it contradicts -# properties. Producers must omit them, which the Kafka path enforces by -# writable_flag_evaluations not declaring them. +# column accepts ALTER UPDATE or an explicit value on insert. An UPDATE of +# properties does not recompute these columns, so any rewrite of properties must +# reset each affected column itself. Property removal does that as it copies a row +# out (see docs/internal/clickhouse-deletion-coverage.md). The cost is a footgun +# MATERIALIZED did not have: an insert that names one of these columns stores the +# given value even when it contradicts properties. Producers must omit them, which +# the Kafka path enforces by writable_flag_evaluations not declaring them. _FLAG_EVALUATIONS_TYPED_COLUMNS = f""" , $group_0 String DEFAULT {trim_quotes_expr("JSONExtractRaw(properties, '$group_0')")} COMMENT 'column_materializer::$group_0' , $group_1 String DEFAULT {trim_quotes_expr("JSONExtractRaw(properties, '$group_1')")} COMMENT 'column_materializer::$group_1' diff --git a/posthog/models/test/test_deletion_targets.py b/posthog/models/test/test_deletion_targets.py index 675873aa2b28..c1b3acaca30b 100644 --- a/posthog/models/test/test_deletion_targets.py +++ b/posthog/models/test/test_deletion_targets.py @@ -10,6 +10,7 @@ from posthog.clickhouse.cluster import ClickhouseCluster, HostInfo from posthog.models.deletion_targets import ( DeletionTarget, + HogQLSchema, UnreachableTargetError, dispatchable_here, placement_for, @@ -134,3 +135,14 @@ def test_a_sibling_shards_over_the_role_its_target_declares() -> None: assert placement is not None assert sorted(placement.cluster.shards) == [1, 2] assert placement.cluster.shard_role == NodeRole.EVENTS + + +def test_an_events_schema_target_cannot_take_the_property_rewrite_without_person_properties() -> None: + with pytest.raises(ValueError, match="stores_person_properties"): + DeletionTarget( + data_table="sharded_events", + read_table="events", + hogql_schema=HogQLSchema.LEGACY, + accepts_property_rewrite=True, + stores_person_properties=False, + )