fix(sessions): rebuild sessions v3 tables on self-hosted upgrade by pauldambra · Pull Request #127889 · PostHog/posthog

pauldambra · 2026-11-02T23:44:41Z

Problem

Self-hosted installs fail to start after an upgrade, because migrate_clickhouse stops at 0364_sessions_v3_more_ad_ids.
ClickHouse reports: Conversion from AggregateFunction(uniq, Nullable(UUID)) to AggregateFunction(uniqExact, Nullable(UUID)) is not supported.
#104566 changed the v3 counts to uniq. Off cloud, 0364 creates the new MV on top of the old v3 table, which still has uniqExact columns.

Changes

Self-hosted upgrades now get past 0364. The migration drops the v3 tables, view and MVs, then creates them from the current definitions.
The dropped rows are at most what the 2025 MVs wrote before 0178 removed them. Sessions reads default to v2, so no query result changes.
The rebuilt MVs turn v3 ingestion on for self-hosted installs, which adds write load from sharded_events and session_replay_events.
The rebuild also removes flag_keys and the old bloom filters, so the three unused DROP_FLAG_* constants go.
Cloud is unchanged: 0364 still has no operations there.

How did you test this code?

Against a local ClickHouse: an old-shape v3 table makes the old 0364 fail with the error above. The new 0364 passes, gives uniq columns, and passes again on a second run.
posthog/clickhouse/test/test_migrations.py passes.
Not run: the multinode HCL smoke. CI runs it.

Test rationale: no new test. The bug needs an old-shape table, which the test database never has.

Release status

No feature flag controls this change

Docs update

None.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, claude-opus-5-5

Skills: /clickhouse-migrations, /writing-pr-descriptions, /simplify

More context for AI agents

Session: PostHog Desktop task, from a self-hosted docker logs trace.
Decisions: the change edits 0364 and adds no new migration, because a failed 0364 re-runs and must stop failing. It follows the drop-and-create order of 0158.
Duplicate search: no open PR fixes this.
Read closely: the drop of sharded_raw_sessions_v3 off cloud, and that the cloud gate relies on CLOUD_DEPLOYMENT being set on the pod.

Created with PostHog Desktop from a PostHog Desktop task

🤖 Generated with Claude Code

…installs Migration 0364 created the v3 MVs on top of the existing v3 table. An install that created v3 before the shape change still had uniqExact count columns, so ClickHouse refused the MV and migrate_clickhouse failed on upgrade. Off cloud, 0364 now drops the v3 tables and views and creates them again from the current definitions. Nothing wrote to v3 there since 0178, and v2 stays the default, so only stale rows go. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: be030037-bc82-4088-8a1d-fe367ee20c39

trunk-io · 2026-11-02T23:44:46Z

😎 Merged successfully - details.

github-actions · 2026-11-02T23:45:47Z

🤖 CI report

⚠️

Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

✅ Duplication (Python) — clean

New Python code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

✅ Duplication (TypeScript) — clean

New TypeScript code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

✅ Playwright — all passed

All tests passed.

View test results →

ℹ️ ClickHouse migration SQL — 1 migration(s)

ClickHouse migration SQL per cloud environment

unset

all

DROP TABLE IF EXISTS raw_sessions_v3_mv

DROP TABLE IF EXISTS raw_sessions_v3_recordings_mv

DROP VIEW IF EXISTS raw_sessions_v3_v

DROP TABLE IF EXISTS raw_sessions_v3

DROP TABLE IF EXISTS writable_raw_sessions_v3

DROP TABLE IF EXISTS sharded_raw_sessions_v3 SYNC

CREATE TABLE IF NOT EXISTS sharded_raw_sessions_v3
(
    team_id Int64,

    -- Both UInt128 and UUID are imperfect choices here
    -- see https://michcioperz.com/wiki/clickhouse-uuid-ordering/
    -- but also see https://github.com/ClickHouse/ClickHouse/issues/77226 and hope
    -- right now choose UInt128 as that's the type of events.$session_id_uuid, but in the future we will probably want to switch everything to the new CH UUID type (when it's released)
    session_id_v7 UInt128,
    -- Ideally we would not need to store this separately, as the ID *is* the timestamp
    -- Unfortunately for now, chaining clickhouse functions to extract the timestamp will break indexes / partition pruning, so do this workaround
    -- again, when the new CH UUID type is released, we should try to switch to that and remove the separate timestamp column
    session_timestamp DateTime64 DEFAULT fromUnixTimestamp64Milli(toUInt64(bitShiftRight(session_id_v7, 80))),

    -- ClickHouse will pick the latest value of distinct_id for the session
    -- this is fine since even if the distinct_id changes during a session
    -- it will still (or should still) map to the same person
    distinct_id AggregateFunction(argMax, String, DateTime64(6, 'UTC')),
    distinct_ids AggregateFunction(groupUniqArray, String),

    min_timestamp SimpleAggregateFunction(min, DateTime64(6, 'UTC')),
    max_timestamp SimpleAggregateFunction(max, DateTime64(6, 'UTC')),
    max_inserted_at SimpleAggregateFunction(max, DateTime64(6, 'UTC')),

    -- urls
    urls SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),
    entry_url AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    end_url AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC')),
    last_external_click_url AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC')),

    -- device
    browser AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    browser_version AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    os AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    os_version AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    device_type AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    viewport_width AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC')),
    viewport_height AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC')),

    -- geoip
    -- only store the properties we actually use, as there's tons, see https://posthog.com/docs/cdp/geoip-enrichment
    geoip_country_code AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    geoip_subdivision_1_code AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    geoip_subdivision_1_name AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    geoip_subdivision_city_name AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    geoip_time_zone AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),

    -- attribution
    entry_referring_domain AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_utm_source AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_utm_campaign AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_utm_medium AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_utm_term AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_utm_content AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_gclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_gad_source AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),

    -- for channel type calculation, it's enough to know if these were present
    entry_has_gclid AggregateFunction(argMin, Boolean, DateTime64(6, 'UTC')),
    entry_has_fbclid AggregateFunction(argMin, Boolean, DateTime64(6, 'UTC')),

    -- lower-tier ad ids as a flat 'key=value' array - like flag_key_values, a per-key map
    -- state made merges much more expensive
    entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),

    -- channel type properties tuple - to reduce redundant reading of the timestamp when loading all of these columns
    -- utm_source, utm_medium, utm_campaign, referring domain, has_gclid, has_fbclid, gad_source
    entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Boolean, Boolean, Nullable(String)), DateTime64(6, 'UTC')),

    -- Count pageview, autocapture, and screen events for providing totals.
    -- Use uniq instead of count, so that inserting events can be idempotent. This is necessary as sometimes we see
    -- events being inserted multiple times to be deduped later, but that can trigger multiple rows here.
    -- Additionally, idempotency is useful for backfilling, as we can just reinsert the same events without worrying.
    -- uniq (not uniqExact) keeps the state small; per-session counts are tiny so it is exact in practice.
    pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
    autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
    screen_uniq AggregateFunction(uniq, Nullable(UUID)),

    -- As a performance optimisation, also keep track of the uniq events for all of these combined.
    -- This is a much more efficient way of calculating the bounce rate, as >2 means not a bounce
    page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
    has_autocapture SimpleAggregateFunction(max, Boolean),

    -- Flags - store every seen 'key=value' pair per flag. A flat array instead of a
    -- groupUniqArrayMap state because per-key sub-aggregators made merges ~8-37x more expensive.
    -- Key-only searches use the text index below (tokenized on '='), so no separate keys column.
    flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),

    -- Event names - store unique event names seen in this session
    event_names SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),

    -- Hosts - store unique hostnames seen in this session (extracted from $host property)
    hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),

    -- Emails - store unique emails seen in this session (extracted from person_properties.email)
    emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),

    -- Replay
    has_replay_events SimpleAggregateFunction(max, Boolean)
,

    -- Indexes
    INDEX event_names_bloom_filter event_names TYPE bloom_filter() GRANULARITY 1,
    INDEX flag_key_values_text flag_key_values TYPE text(tokenizer = splitByString(['='])),
    INDEX hosts_bloom_filter hosts TYPE bloom_filter() GRANULARITY 1,
    INDEX emails_bloom_filter emails TYPE bloom_filter() GRANULARITY 1
) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/{shard}/posthog.raw_sessions_v3', '{replica}')

PARTITION BY toYYYYMM(session_timestamp)
ORDER BY (
    team_id,
    session_timestamp,
    session_id_v7
)
SETTINGS parts_to_delay_insert = 250, max_delay_to_insert = 10, parts_to_throw_insert = 1000

CREATE TABLE IF NOT EXISTS writable_raw_sessions_v3
(
    team_id Int64,

    -- Both UInt128 and UUID are imperfect choices here
    -- see https://michcioperz.com/wiki/clickhouse-uuid-ordering/
    -- but also see https://github.com/ClickHouse/ClickHouse/issues/77226 and hope
    -- right now choose UInt128 as that's the type of events.$session_id_uuid, but in the future we will probably want to switch everything to the new CH UUID type (when it's released)
    session_id_v7 UInt128,
    -- Ideally we would not need to store this separately, as the ID *is* the timestamp
    -- Unfortunately for now, chaining clickhouse functions to extract the timestamp will break indexes / partition pruning, so do this workaround
    -- again, when the new CH UUID type is released, we should try to switch to that and remove the separate timestamp column
    session_timestamp DateTime64 DEFAULT fromUnixTimestamp64Milli(toUInt64(bitShiftRight(session_id_v7, 80))),

    -- ClickHouse will pick the latest value of distinct_id for the session
    -- this is fine since even if the distinct_id changes during a session
    -- it will still (or should still) map to the same person
    distinct_id AggregateFunction(argMax, String, DateTime64(6, 'UTC')),
    distinct_ids AggregateFunction(groupUniqArray, String),

    min_timestamp SimpleAggregateFunction(min, DateTime64(6, 'UTC')),
    max_timestamp SimpleAggregateFunction(max, DateTime64(6, 'UTC')),
    max_inserted_at SimpleAggregateFunction(max, DateTime64(6, 'UTC')),

    -- urls
    urls SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),
    entry_url AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    end_url AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC')),
    last_external_click_url AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC')),

    -- device
    browser AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    browser_version AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    os AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    os_version AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    device_type AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    viewport_width AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC')),
    viewport_height AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC')),

    -- geoip
    -- only store the properties we actually use, as there's tons, see https://posthog.com/docs/cdp/geoip-enrichment
    geoip_country_code AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    geoip_subdivision_1_code AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    geoip_subdivision_1_name AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    geoip_subdivision_city_name AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    geoip_time_zone AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),

    -- attribution
    entry_referring_domain AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_utm_source AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_utm_campaign AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_utm_medium AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_utm_term AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_utm_content AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_gclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_gad_source AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),

    -- for channel type calculation, it's enough to know if these were present
    entry_has_gclid AggregateFunction(argMin, Boolean, DateTime64(6, 'UTC')),
    entry_has_fbclid AggregateFunction(argMin, Boolean, DateTime64(6, 'UTC')),

    -- lower-tier ad ids as a flat 'key=value' array - like flag_key_values, a per-key map
    -- state made merges much more expensive
    entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),

    -- channel type properties tuple - to reduce redundant reading of the timestamp when loading all of these columns
    -- utm_source, utm_medium, utm_campaign, referring domain, has_gclid, has_fbclid, gad_source
    entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Boolean, Boolean, Nullable(String)), DateTime64(6, 'UTC')),

    -- Count pageview, autocapture, and screen events for providing totals.
    -- Use uniq instead of count, so that inserting events can be idempotent. This is necessary as sometimes we see
    -- events being inserted multiple times to be deduped later, but that can trigger multiple rows here.
    -- Additionally, idempotency is useful for backfilling, as we can just reinsert the same events without worrying.
    -- uniq (not uniqExact) keeps the state small; per-session counts are tiny so it is exact in practice.
    pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
    autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
    screen_uniq AggregateFunction(uniq, Nullable(UUID)),

    -- As a performance optimisation, also keep track of the uniq events for all of these combined.
    -- This is a much more efficient way of calculating the bounce rate, as >2 means not a bounce
    page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
    has_autocapture SimpleAggregateFunction(max, Boolean),

    -- Flags - store every seen 'key=value' pair per flag. A flat array instead of a
    -- groupUniqArrayMap state because per-key sub-aggregators made merges ~8-37x more expensive.
    -- Key-only searches use the text index below (tokenized on '='), so no separate keys column.
    flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),

    -- Event names - store unique event names seen in this session
    event_names SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),

    -- Hosts - store unique hostnames seen in this session (extracted from $host property)
    hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),

    -- Emails - store unique emails seen in this session (extracted from person_properties.email)
    emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),

    -- Replay
    has_replay_events SimpleAggregateFunction(max, Boolean)
) ENGINE = Distributed('posthog', 'posthog_test', 'sharded_raw_sessions_v3', cityHash64(session_id_v7))

CREATE TABLE IF NOT EXISTS raw_sessions_v3
(
    team_id Int64,

    -- Both UInt128 and UUID are imperfect choices here
    -- see https://michcioperz.com/wiki/clickhouse-uuid-ordering/
    -- but also see https://github.com/ClickHouse/ClickHouse/issues/77226 and hope
    -- right now choose UInt128 as that's the type of events.$session_id_uuid, but in the future we will probably want to switch everything to the new CH UUID type (when it's released)
    session_id_v7 UInt128,
    -- Ideally we would not need to store this separately, as the ID *is* the timestamp
    -- Unfortunately for now, chaining clickhouse functions to extract the timestamp will break indexes / partition pruning, so do this workaround
    -- again, when the new CH UUID type is released, we should try to switch to that and remove the separate timestamp column
    session_timestamp DateTime64 MATERIALIZED fromUnixTimestamp64Milli(toUInt64(bitShiftRight(session_id_v7, 80))),

    -- ClickHouse will pick the latest value of distinct_id for the session
    -- this is fine since even if the distinct_id changes during a session
    -- it will still (or should still) map to the same person
    distinct_id AggregateFunction(argMax, String, DateTime64(6, 'UTC')),
    distinct_ids AggregateFunction(groupUniqArray, String),

    min_timestamp SimpleAggregateFunction(min, DateTime64(6, 'UTC')),
    max_timestamp SimpleAggregateFunction(max, DateTime64(6, 'UTC')),
    max_inserted_at SimpleAggregateFunction(max, DateTime64(6, 'UTC')),

    -- urls
    urls SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),
    entry_url AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    end_url AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC')),
    last_external_click_url AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC')),

    -- device
    browser AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    browser_version AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    os AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    os_version AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    device_type AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    viewport_width AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC')),
    viewport_height AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC')),

    -- geoip
    -- only store the properties we actually use, as there's tons, see https://posthog.com/docs/cdp/geoip-enrichment
    geoip_country_code AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    geoip_subdivision_1_code AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    geoip_subdivision_1_name AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    geoip_subdivision_city_name AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    geoip_time_zone AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),

    -- attribution
    entry_referring_domain AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_utm_source AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_utm_campaign AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_utm_medium AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_utm_term AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_utm_content AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_gclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_gad_source AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
    entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),

    -- for channel type calculation, it's enough to know if these were present
    entry_has_gclid AggregateFunction(argMin, Boolean, DateTime64(6, 'UTC')),
    entry_has_fbclid AggregateFunction(argMin, Boolean, DateTime64(6, 'UTC')),

    -- lower-tier ad ids as a flat 'key=value' array - like flag_key_values, a per-key map
    -- state made merges much more expensive
    entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),

    -- channel type properties tuple - to reduce redundant reading of the timestamp when loading all of these columns
    -- utm_source, utm_medium, utm_campaign, referring domain, has_gclid, has_fbclid, gad_source
    entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Boolean, Boolean, Nullable(String)), DateTime64(6, 'UTC')),

    -- Count pageview, autocapture, and screen events for providing totals.
    -- Use uniq instead of count, so that inserting events can be idempotent. This is necessary as sometimes we see
    -- events being inserted multiple times to be deduped later, but that can trigger multiple rows here.
    -- Additionally, idempotency is useful for backfilling, as we can just reinsert the same events without worrying.
    -- uniq (not uniqExact) keeps the state small; per-session counts are tiny so it is exact in practice.
    pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
    autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
    screen_uniq AggregateFunction(uniq, Nullable(UUID)),

    -- As a performance optimisation, also keep track of the uniq events for all of these combined.
    -- This is a much more efficient way of calculating the bounce rate, as >2 means not a bounce
    page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
    has_autocapture SimpleAggregateFunction(max, Boolean),

    -- Flags - store every seen 'key=value' pair per flag. A flat array instead of a
    -- groupUniqArrayMap state because per-key sub-aggregators made merges ~8-37x more expensive.
    -- Key-only searches use the text index below (tokenized on '='), so no separate keys column.
    flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),

    -- Event names - store unique event names seen in this session
    event_names SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),

    -- Hosts - store unique hostnames seen in this session (extracted from $host property)
    hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),

    -- Emails - store unique emails seen in this session (extracted from person_properties.email)
    emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),

    -- Replay
    has_replay_events SimpleAggregateFunction(max, Boolean)
) ENGINE = Distributed('posthog', 'posthog_test', 'sharded_raw_sessions_v3', cityHash64(session_id_v7))

CREATE OR REPLACE VIEW raw_sessions_v3_v AS
SELECT
    session_id_v7,
    session_timestamp,
    team_id,

    argMaxMerge(distinct_id) as distinct_id,
    groupUniqArrayMerge(distinct_ids) AS distinct_ids,

    min(min_timestamp) as min_timestamp,
    max(max_timestamp) as max_timestamp,
    max(max_inserted_at) as max_inserted_at,

    -- urls
    groupUniqArrayArray(2000)(urls) AS urls,
    argMinMerge(entry_url) as entry_url,
    argMaxMerge(end_url) as end_url,
    argMaxMerge(last_external_click_url) as last_external_click_url,

    -- device
    argMinMerge(browser) as browser,
    argMinMerge(browser_version) as browser_version,
    argMinMerge(os) as os,
    argMinMerge(os_version) as os_version,
    argMinMerge(device_type) as device_type,
    argMinMerge(viewport_width) as viewport_width,
    argMinMerge(viewport_height) as viewport_height,

    -- geoip
    argMinMerge(geoip_country_code) as geoip_country_code,
    argMinMerge(geoip_subdivision_1_code) as geoip_subdivision_1_code,
    argMinMerge(geoip_subdivision_1_name) as geoip_subdivision_1_name,
    argMinMerge(geoip_subdivision_city_name) as geoip_subdivision_city_name,
    argMinMerge(geoip_time_zone) as geoip_time_zone,

    -- attribution
    argMinMerge(entry_utm_source) as entry_utm_source,
    argMinMerge(entry_utm_campaign) as entry_utm_campaign,
    argMinMerge(entry_utm_medium) as entry_utm_medium,
    argMinMerge(entry_utm_term) as entry_utm_term,
    argMinMerge(entry_utm_content) as entry_utm_content,
    argMinMerge(entry_referring_domain) as entry_referring_domain,
    argMinMerge(entry_gclid) as entry_gclid,
    argMinMerge(entry_gad_source) as entry_gad_source,
    argMinMerge(entry_fbclid) as entry_fbclid,

    argMinMerge(entry_has_gclid) as entry_has_gclid,
    argMinMerge(entry_has_fbclid) as entry_has_fbclid,

    argMinMerge(entry_ad_ids_set) as entry_ad_ids_set,

    argMinMerge(entry_channel_type_properties) as entry_channel_type_properties,

    -- counts
    uniqMerge(pageview_uniq) as pageview_uniq,
    uniqMerge(autocapture_uniq) as autocapture_uniq,
    uniqMerge(screen_uniq) as screen_uniq,

    -- perf
    uniqUpToMerge(1)(page_screen_uniq_up_to) as page_screen_uniq_up_to,
    max(has_autocapture) as has_autocapture,

    -- flags
    groupUniqArrayArray(10000)(flag_key_values) as flag_key_values,

    -- event names
    groupUniqArrayArray(2000)(event_names) as event_names,

    -- hosts
    groupUniqArrayArray(100)(hosts) as hosts,

    -- emails
    groupUniqArrayArray(10)(emails) as emails,

    -- replay
    max(has_replay_events) as has_replay_events
FROM posthog_test.raw_sessions_v3
GROUP BY session_id_v7, session_timestamp, team_id

CREATE MATERIALIZED VIEW IF NOT EXISTS raw_sessions_v3_mv
TO posthog_test.writable_raw_sessions_v3
AS

WITH
  
        JSONExtract(properties, 'Tuple(
            `$current_url` Nullable(String),
            `$external_click_url` Nullable(String),
            `$browser` Nullable(String),
            `$browser_version` Nullable(String),
            `$os` Nullable(String),
            `$os_version` Nullable(String),
            `$device_type` Nullable(String),
            `$viewport_width` Nullable(Int64),
            `$viewport_height` Nullable(Int64),
            `$geoip_country_code` Nullable(String),
            `$geoip_subdivision_1_code` Nullable(String),
            `$geoip_subdivision_1_name` Nullable(String),
            `$geoip_city_name` Nullable(String),
            `$geoip_time_zone` Nullable(String),
            `$referring_domain` Nullable(String),
            `utm_source` Nullable(String),
            `utm_campaign` Nullable(String),
            `utm_medium` Nullable(String),
            `utm_term` Nullable(String),
            `utm_content` Nullable(String),
            `gclid` Nullable(String),
            `gad_source` Nullable(String),
            `fbclid` Nullable(String),
            `$host` Nullable(String),
            `gclsrc` Nullable(String),
            `dclid` Nullable(String),
            `gbraid` Nullable(String),
            `wbraid` Nullable(String),
            `gad_campaignid` Nullable(String),
            `srsltid` Nullable(String),
            `msclkid` Nullable(String),
            `twclid` Nullable(String),
            `li_fat_id` Nullable(String),
            `mc_cid` Nullable(String),
            `igshid` Nullable(String),
            `ttclid` Nullable(String),
            `rdt_cid` Nullable(String),
            `epik` Nullable(String),
            `qclid` Nullable(String),
            `sccid` Nullable(String),
            `_kx` Nullable(String),
            `irclid` Nullable(String),
            `irclickid` Nullable(String),
            `utm_id` Nullable(String),
            `utm_source_platform` Nullable(String),
            `utm_creative_format` Nullable(String),
            `utm_marketing_tactic` Nullable(String),
            `yclid` Nullable(String),
            `tblci` Nullable(String),
            `dicbo` Nullable(String),
            `ndclid` Nullable(String),
            `cjevent` Nullable(String),
            `awc` Nullable(String),
            `syclid` Nullable(String),
            `_bhlid` Nullable(String),
            `rtid` Nullable(String),
            `oppref` Nullable(String),
            `bfcid` Nullable(String)
        )') as p,
        JSONExtractString(person_properties, 'email') as _person_email,
        tupleElement(p, '$current_url') as _current_url,
        if(event = '$pageview' OR event = '$screen', left(coalesce(_current_url, ''), 16384), '') as _attribution_url,
        tupleElement(p, '$external_click_url') as _external_click_url,
        tupleElement(p, '$browser') as _browser,
        tupleElement(p, '$browser_version') as _browser_version,
        tupleElement(p, '$os') as _os,
        tupleElement(p, '$os_version') as _os_version,
        tupleElement(p, '$device_type') as _device_type,
        tupleElement(p, '$viewport_width') as _viewport_width,
        tupleElement(p, '$viewport_height') as _viewport_height,
        tupleElement(p, '$geoip_country_code') as _geoip_country_code,
        tupleElement(p, '$geoip_subdivision_1_code') as _geoip_subdivision_1_code,
        tupleElement(p, '$geoip_subdivision_1_name') as _geoip_subdivision_1_name,
        tupleE

…truncated. See the full SQL in the workflow logs.

posthog

PostHog Review · Standard · reviewed f49fb74: 1 Should fix.

This is the only automatic review while this pull request is a draft. Comment @posthog review to review new changes sooner. Otherwise, a review of the new changes runs when you mark it ready for review.

See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.

…essions_v3

parameterai · 2026-11-03T13:03:03Z

Risk: No findings

A ClickHouse-migration-only fix: off-cloud, migration 0364 now drops and recreates the sessions v3 tables/view/MVs from current definitions (fixing the uniq/uniqExact conversion failure on self-hosted upgrades) and lifts the drop-size guard on the sharded table drop. I traced the data-loss claim (0178 unconditionally dropped the v3 ingestion MVs, and the replacement pipeline in 0328 is cloud-only, so off-cloud v3 rows are stale), verified drop/create idempotency and ordering against the 0158 precedent, and confirmed no later migration or code path depends on the old shape. No security issues found.

_{Sentinel reviewed e6d5077 · Review settings}

…ons_v3

pauldambra · 2026-11-03T13:26:37Z

/trunk merge

deployment-status-posthog · 2026-11-03T14:20:05Z

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-11-03 14:19 UTC	Run
prod-us	✅ Deployed	2026-11-03 14:59 UTC	Run
prod-eu	✅ Deployed	2026-11-03 14:58 UTC	Run

pauldambra self-assigned this Nov 2, 2026

posthog Bot reviewed Nov 2, 2026

View reviewed changes

Comment thread posthog/clickhouse/migrations/0364_sessions_v3_more_ad_ids.py Resolved

rorylshanks approved these changes Nov 3, 2026

View reviewed changes

fix(sessions): lift the drop-size guard when rebuilding sharded_raw_s…

e6d5077

…essions_v3

lricoy marked this pull request as ready for review November 3, 2026 13:02

lricoy requested a review from a team as a code owner November 3, 2026 13:02

pr-assigner-resolver-posthog Bot requested a review from a team November 3, 2026 13:03

lricoy approved these changes Nov 3, 2026

View reviewed changes

fix(sessions): revert the drop-size guard change on sharded_raw_sessi…

e78a6cb

…ons_v3

trunk-io Bot merged commit e20ef9c into master Nov 3, 2026
265 of 266 checks passed

trunk-io Bot deleted the posthog/fix-sessions-v3-hobby-upgrade branch November 3, 2026 13:52

Navigation Menu

Repository navigation

fix(sessions): rebuild sessions v3 tables on self-hosted upgrade - #127889

fix(sessions): rebuild sessions v3 tables on self-hosted upgrade#127889
trunk-io[bot] merged 3 commits into
masterfrom
posthog/fix-sessions-v3-hobby-upgrade

pauldambra commented Nov 2, 2026 •

edited by lricoy

Loading

Uh oh!

trunk-io Bot commented Nov 2, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 2, 2026 •

edited

Loading

ClickHouse migration SQL per cloud environment

Uh oh!

posthog Bot left a comment

Uh oh!

Uh oh!

parameterai Bot commented Nov 3, 2026 •

edited

Loading

Uh oh!

pauldambra commented Nov 3, 2026

Uh oh!

Uh oh!

deployment-status-posthog Bot commented Nov 3, 2026 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

pauldambra commented Nov 2, 2026 • edited by lricoy Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Changes

How did you test this code?

Release status

Docs update

🤖 Agent context

Uh oh!

trunk-io Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 CI report

ClickHouse migration SQL per cloud environment

Uh oh!

posthog Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

parameterai Bot commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pauldambra commented Nov 3, 2026

Uh oh!

Uh oh!

deployment-status-posthog Bot commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploy status

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pauldambra commented Nov 2, 2026 •

edited by lricoy

Loading

trunk-io Bot commented Nov 2, 2026 •

edited

Loading

github-actions Bot commented Nov 2, 2026 •

edited

Loading

parameterai Bot commented Nov 3, 2026 •

edited

Loading

deployment-status-posthog Bot commented Nov 3, 2026 •

edited

Loading