Skip to content

fix(insights): Fix timezone date issues #9678

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
timgl merged 38 commits into from
May 20, 2022
Merged

fix(insights): Fix timezone date issues #9678

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
62a80a7
fix(insights): Fix timezone date issues
timgl May 6, 2022
a03215d
Merge branch 'master' into fix-timezone-date-issues
timgl May 6, 2022
388f83a
fix formatting
timgl May 6, 2022
019c1b9
fix
timgl May 10, 2022
9e45c3e
fix
timgl May 10, 2022
c266d35
fix
timgl May 10, 2022
3a347a8
fix
timgl May 10, 2022
827a7b2
fix
timgl May 10, 2022
541dfe2
fix
timgl May 10, 2022
541d1f5
fix
timgl May 10, 2022
79e02f3
fix
timgl May 10, 2022
16d4efd
fix
timgl May 10, 2022
41ea450
fix
timgl May 10, 2022
313b9a6
Merge branch 'master' into fix-timezone-date-issues
timgl May 10, 2022
74bf573
fix typescript
timgl May 10, 2022
1264171
Merge branch 'master' into fix-timezone-date-issues
timgl May 10, 2022
5649258
Merge branch 'master' into fix-timezone-date-issues
timgl May 11, 2022
863f7f4
Add snapshots for timezone tests
timgl May 11, 2022
7753eb9
Update snapshots
timgl May 11, 2022
4764fb7
fix with_data issue
timgl May 11, 2022
4a2c7cc
Update snapshots
github-actions[bot] May 11, 2022
4d2a0cc
Update snapshots
github-actions[bot] May 11, 2022
0e617ae
fix date_to and fix retention
timgl May 12, 2022
7fb7e82
Fix retention
timgl May 12, 2022
80a0b16
Merge branch 'fix-timezone-date-issues' of github.com:PostHog/posthog…
timgl May 12, 2022
6393419
Merge branch 'master' into fix-timezone-date-issues
timgl May 12, 2022
0339b64
Update snapshots
github-actions[bot] May 12, 2022
b4209cb
Always set hourly intervals to 23:59:59, unless no date is given
timgl May 13, 2022
724a5fb
Merge branch 'fix-timezone-date-issues' of github.com:PostHog/posthog…
timgl May 13, 2022
622925b
Update snapshots
github-actions[bot] May 13, 2022
ae60a32
Merge branch 'master' into fix-timezone-date-issues
timgl May 17, 2022
185a5f0
try always passing hourly through
timgl May 17, 2022
34c70ec
Merge branch 'fix-timezone-date-issues' of github.com:PostHog/posthog…
timgl May 17, 2022
f320ec5
ifix
timgl May 17, 2022
6be4afc
fix
timgl May 17, 2022
6392a9c
fix mayb
timgl May 17, 2022
66e8e30
other fix
timgl May 17, 2022
98f36e2
fix
timgl May 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions ee/clickhouse/queries/funnels/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
from abc import ABC
from typing import Any, Dict, List, Optional, Tuple, Union, cast

from django.utils import timezone
from rest_framework.exceptions import ValidationError

from ee.clickhouse.materialized_columns.columns import ColumnName
Expand Down Expand Up @@ -116,9 +115,6 @@ def _update_filters(self):
if not self._filter._date_from:
data.update({"date_from": relative_date_parse("-7d")})

if not self._filter._date_to:
data.update({"date_to": timezone.now()})

if self._filter.breakdown and not self._filter.breakdown_type:
data.update({"breakdown_type": "event"})

Expand Down
8 changes: 5 additions & 3 deletions ee/clickhouse/queries/funnels/funnel_trends.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,8 +110,10 @@ def get_query(self) -> str:
_date_from = self._filter.date_from

breakdown_clause = self._get_breakdown_prop()
formatted_date_from = format_ch_timestamp(_date_from, self._filter)
formatted_date_to = format_ch_timestamp(self._filter.date_to, self._filter)
formatted_date_from = format_ch_timestamp(_date_from, convert_to_timezone=self._team.timezone_for_charts)
formatted_date_to = format_ch_timestamp(
self._filter.date_to, convert_to_timezone=self._team.timezone_for_charts
)

self.params.update(
{
Expand Down Expand Up @@ -140,7 +142,7 @@ def get_query(self) -> str:
) data
RIGHT OUTER JOIN (
SELECT
{trunc_func}(toDateTime(%(formatted_date_from)s, %(timezone)s) + {interval_func}(number), {start_of_week_fix(self._filter)} %(timezone)s) AS entrance_period_start
{trunc_func}(toDateTime(%(formatted_date_from)s) + {interval_func}(number), {start_of_week_fix(self._filter)} %(timezone)s) AS entrance_period_start
{', breakdown_value as prop' if breakdown_clause else ''}
FROM numbers(dateDiff(%(interval)s, toDateTime(%(formatted_date_from)s), toDateTime(%(formatted_date_to)s)) + 1) AS period_offsets
{'ARRAY JOIN (%(breakdown_values)s) AS breakdown_value' if breakdown_clause else ''}
Expand Down
217 changes: 102 additions & 115 deletions ee/clickhouse/queries/funnels/test/__snapshots__/test_funnel.ambr
View file
Open in desktop

Large diffs are not rendered by default.

132 changes: 66 additions & 66 deletions ee/clickhouse/queries/funnels/test/__snapshots__/test_funnel_correlation.ambr
View file
Open in desktop

Large diffs are not rendered by default.

20 changes: 10 additions & 10 deletions ee/clickhouse/queries/funnels/test/__snapshots__/test_funnel_correlations_persons.ambr
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,8 +99,8 @@
HAVING argMax(is_deleted, version) = 0) AS pdi ON e.distinct_id = pdi.distinct_id
WHERE team_id = 2
AND event IN ['$pageview', 'insight analyzed']
AND timestamp >= toDateTime('2021-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2021-01-08 23:59:59', 'UTC') ) events
AND timestamp >= toDateTime('2021-01-01 00:00:00')
AND timestamp <= toDateTime('2021-01-08 23:59:59') ) events
WHERE (step_0 = 1
OR step_1 = 1) ))
WHERE step_0 = 1 SETTINGS allow_experimental_window_functions = 1 ))
Expand Down Expand Up @@ -320,8 +320,8 @@
HAVING argMax(is_deleted, version) = 0) AS pdi ON e.distinct_id = pdi.distinct_id
WHERE team_id = 2
AND event IN ['$pageview', 'insight analyzed', 'insight updated']
AND timestamp >= toDateTime('2021-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2021-01-08 23:59:59', 'UTC') ) events
AND timestamp >= toDateTime('2021-01-01 00:00:00')
AND timestamp <= toDateTime('2021-01-08 23:59:59') ) events
WHERE (step_0 = 1
OR step_1 = 1
OR step_2 = 1) ))))
Expand Down Expand Up @@ -471,8 +471,8 @@
HAVING argMax(is_deleted, version) = 0) AS pdi ON e.distinct_id = pdi.distinct_id
WHERE team_id = 2
AND event IN ['$pageview', 'insight analyzed']
AND timestamp >= toDateTime('2021-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2021-01-08 23:59:59', 'UTC') ) events
AND timestamp >= toDateTime('2021-01-01 00:00:00')
AND timestamp <= toDateTime('2021-01-08 23:59:59') ) events
WHERE (step_0 = 1
OR step_1 = 1) ))
WHERE step_0 = 1 SETTINGS allow_experimental_window_functions = 1 ))
Expand Down Expand Up @@ -609,8 +609,8 @@
GROUP BY distinct_id
HAVING argMax(is_deleted, version) = 0) AS pdi ON e.distinct_id = pdi.distinct_id
WHERE team_id = 2
AND timestamp >= toDateTime('2021-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2021-01-08 23:59:59', 'UTC') ) events
AND timestamp >= toDateTime('2021-01-01 00:00:00')
AND timestamp <= toDateTime('2021-01-08 23:59:59') ) events
WHERE (1=1) ))
WHERE step_0 = 1 ))
GROUP BY aggregation_target,
Expand Down Expand Up @@ -746,8 +746,8 @@
GROUP BY distinct_id
HAVING argMax(is_deleted, version) = 0) AS pdi ON e.distinct_id = pdi.distinct_id
WHERE team_id = 2
AND timestamp >= toDateTime('2021-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2021-01-08 23:59:59', 'UTC') ) events
AND timestamp >= toDateTime('2021-01-01 00:00:00')
AND timestamp <= toDateTime('2021-01-08 23:59:59') ) events
WHERE (1=1) ))
WHERE step_0 = 1 ))
GROUP BY aggregation_target,
Expand Down
12 changes: 6 additions & 6 deletions ee/clickhouse/queries/funnels/test/__snapshots__/test_funnel_persons.ambr
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,8 +160,8 @@
HAVING argMax(is_deleted, version) = 0) AS pdi ON e.distinct_id = pdi.distinct_id
WHERE team_id = 2
AND event IN ['step one', 'step three', 'step two']
AND timestamp >= toDateTime('2021-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2021-01-08 23:59:59', 'UTC') ) events
AND timestamp >= toDateTime('2021-01-01 00:00:00')
AND timestamp <= toDateTime('2021-01-08 23:59:59') ) events
WHERE (step_0 = 1
OR step_1 = 1
OR step_2 = 1) ))))
Expand Down Expand Up @@ -347,8 +347,8 @@
HAVING argMax(is_deleted, version) = 0) AS pdi ON e.distinct_id = pdi.distinct_id
WHERE team_id = 2
AND event IN ['step one', 'step three', 'step two']
AND timestamp >= toDateTime('2021-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2021-01-08 23:59:59', 'UTC') ) events
AND timestamp >= toDateTime('2021-01-01 00:00:00')
AND timestamp <= toDateTime('2021-01-08 23:59:59') ) events
WHERE (step_0 = 1
OR step_1 = 1
OR step_2 = 1) ))))
Expand Down Expand Up @@ -534,8 +534,8 @@
HAVING argMax(is_deleted, version) = 0) AS pdi ON e.distinct_id = pdi.distinct_id
WHERE team_id = 2
AND event IN ['step one', 'step three', 'step two']
AND timestamp >= toDateTime('2021-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2021-01-08 23:59:59', 'UTC') ) events
AND timestamp >= toDateTime('2021-01-01 00:00:00')
AND timestamp <= toDateTime('2021-01-08 23:59:59') ) events
WHERE (step_0 = 1
OR step_1 = 1
OR step_2 = 1) ))))
Expand Down
48 changes: 24 additions & 24 deletions ee/clickhouse/queries/funnels/test/__snapshots__/test_funnel_strict.ambr
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@
GROUP BY group_key) groups_0 ON "$group_0" == groups_0.group_key
WHERE team_id = 2
AND event = 'sign up'
AND timestamp >= toDateTime('2020-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2020-01-08 23:59:59', 'UTC')
AND timestamp >= toDateTime('2020-01-01 00:00:00')
AND timestamp <= toDateTime('2020-01-08 23:59:59')
AND (NOT has([''], "$group_0"))
GROUP BY value
ORDER BY count DESC
Expand Down Expand Up @@ -103,8 +103,8 @@
AND group_type_index = 0
GROUP BY group_key) groups_0 ON "$group_0" == groups_0.group_key
WHERE team_id = 2
AND timestamp >= toDateTime('2020-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2020-01-08 23:59:59', 'UTC')
AND timestamp >= toDateTime('2020-01-01 00:00:00')
AND timestamp <= toDateTime('2020-01-08 23:59:59')
AND (NOT has([''], "$group_0")) ) events
WHERE (1=1) ))
WHERE step_0 = 1 ))
Expand Down Expand Up @@ -132,8 +132,8 @@
GROUP BY group_key) groups_0 ON "$group_0" == groups_0.group_key
WHERE team_id = 2
AND event = 'sign up'
AND timestamp >= toDateTime('2020-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2020-01-08 23:59:59', 'UTC')
AND timestamp >= toDateTime('2020-01-01 00:00:00')
AND timestamp <= toDateTime('2020-01-08 23:59:59')
GROUP BY value
ORDER BY count DESC
LIMIT 25
Expand Down Expand Up @@ -225,8 +225,8 @@
AND group_type_index = 0
GROUP BY group_key) groups_0 ON "$group_0" == groups_0.group_key
WHERE team_id = 2
AND timestamp >= toDateTime('2020-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2020-01-08 23:59:59', 'UTC') ) events
AND timestamp >= toDateTime('2020-01-01 00:00:00')
AND timestamp <= toDateTime('2020-01-08 23:59:59') ) events
WHERE (1=1) ))
WHERE step_0 = 1 ))
GROUP BY aggregation_target,
Expand All @@ -253,8 +253,8 @@
GROUP BY group_key) groups_0 ON "$group_0" == groups_0.group_key
WHERE team_id = 2
AND event = 'sign up'
AND timestamp >= toDateTime('2020-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2020-01-08 23:59:59', 'UTC')
AND timestamp >= toDateTime('2020-01-01 00:00:00')
AND timestamp <= toDateTime('2020-01-08 23:59:59')
GROUP BY value
ORDER BY count DESC
LIMIT 25
Expand Down Expand Up @@ -339,8 +339,8 @@
AND group_type_index = 0
GROUP BY group_key) groups_0 ON "$group_0" == groups_0.group_key
WHERE team_id = 2
AND timestamp >= toDateTime('2020-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2020-01-08 23:59:59', 'UTC') ) events
AND timestamp >= toDateTime('2020-01-01 00:00:00')
AND timestamp <= toDateTime('2020-01-08 23:59:59') ) events
WHERE (1=1) ))
WHERE step_0 = 1 ))
GROUP BY aggregation_target,
Expand Down Expand Up @@ -371,8 +371,8 @@
GROUP BY group_key) groups_0 ON "$group_0" == groups_0.group_key
WHERE team_id = 2
AND event = 'sign up'
AND timestamp >= toDateTime('2020-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2020-01-08 23:59:59', 'UTC')
AND timestamp >= toDateTime('2020-01-01 00:00:00')
AND timestamp <= toDateTime('2020-01-08 23:59:59')
GROUP BY value
ORDER BY count DESC
LIMIT 25
Expand Down Expand Up @@ -457,8 +457,8 @@
AND group_type_index = 0
GROUP BY group_key) groups_0 ON "$group_0" == groups_0.group_key
WHERE team_id = 2
AND timestamp >= toDateTime('2020-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2020-01-08 23:59:59', 'UTC') ) events
AND timestamp >= toDateTime('2020-01-01 00:00:00')
AND timestamp <= toDateTime('2020-01-08 23:59:59') ) events
WHERE (1=1) ))
WHERE step_0 = 1 ))
GROUP BY aggregation_target,
Expand Down Expand Up @@ -489,8 +489,8 @@
GROUP BY group_key) groups_0 ON "$group_0" == groups_0.group_key
WHERE team_id = 2
AND event = 'sign up'
AND timestamp >= toDateTime('2020-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2020-01-08 23:59:59', 'UTC')
AND timestamp >= toDateTime('2020-01-01 00:00:00')
AND timestamp <= toDateTime('2020-01-08 23:59:59')
GROUP BY value
ORDER BY count DESC
LIMIT 25
Expand Down Expand Up @@ -575,8 +575,8 @@
AND group_type_index = 0
GROUP BY group_key) groups_0 ON "$group_0" == groups_0.group_key
WHERE team_id = 2
AND timestamp >= toDateTime('2020-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2020-01-08 23:59:59', 'UTC') ) events
AND timestamp >= toDateTime('2020-01-01 00:00:00')
AND timestamp <= toDateTime('2020-01-08 23:59:59') ) events
WHERE (1=1) ))
WHERE step_0 = 1 ))
GROUP BY aggregation_target,
Expand Down Expand Up @@ -607,8 +607,8 @@
GROUP BY group_key) groups_0 ON "$group_0" == groups_0.group_key
WHERE team_id = 2
AND event = 'sign up'
AND timestamp >= toDateTime('2020-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2020-01-08 23:59:59', 'UTC')
AND timestamp >= toDateTime('2020-01-01 00:00:00')
AND timestamp <= toDateTime('2020-01-08 23:59:59')
GROUP BY value
ORDER BY count DESC
LIMIT 25
Expand Down Expand Up @@ -693,8 +693,8 @@
AND group_type_index = 0
GROUP BY group_key) groups_0 ON "$group_0" == groups_0.group_key
WHERE team_id = 2
AND timestamp >= toDateTime('2020-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2020-01-08 23:59:59', 'UTC') ) events
AND timestamp >= toDateTime('2020-01-01 00:00:00')
AND timestamp <= toDateTime('2020-01-08 23:59:59') ) events
WHERE (1=1) ))
WHERE step_0 = 1 ))
GROUP BY aggregation_target,
Expand Down
12 changes: 6 additions & 6 deletions ee/clickhouse/queries/funnels/test/__snapshots__/test_funnel_strict_persons.ambr
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,8 +119,8 @@
GROUP BY distinct_id
HAVING argMax(is_deleted, version) = 0) AS pdi ON e.distinct_id = pdi.distinct_id
WHERE team_id = 2
AND timestamp >= toDateTime('2021-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2021-01-08 23:59:59', 'UTC') ) events
AND timestamp >= toDateTime('2021-01-01 00:00:00')
AND timestamp <= toDateTime('2021-01-08 23:59:59') ) events
WHERE (1=1) ))
WHERE step_0 = 1 ))
GROUP BY aggregation_target,
Expand Down Expand Up @@ -263,8 +263,8 @@
GROUP BY distinct_id
HAVING argMax(is_deleted, version) = 0) AS pdi ON e.distinct_id = pdi.distinct_id
WHERE team_id = 2
AND timestamp >= toDateTime('2021-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2021-01-08 23:59:59', 'UTC') ) events
AND timestamp >= toDateTime('2021-01-01 00:00:00')
AND timestamp <= toDateTime('2021-01-08 23:59:59') ) events
WHERE (1=1) ))
WHERE step_0 = 1 ))
GROUP BY aggregation_target,
Expand Down Expand Up @@ -407,8 +407,8 @@
GROUP BY distinct_id
HAVING argMax(is_deleted, version) = 0) AS pdi ON e.distinct_id = pdi.distinct_id
WHERE team_id = 2
AND timestamp >= toDateTime('2021-01-01 00:00:00', 'UTC')
AND timestamp <= toDateTime('2021-01-08 23:59:59', 'UTC') ) events
AND timestamp >= toDateTime('2021-01-01 00:00:00')
AND timestamp <= toDateTime('2021-01-08 23:59:59') ) events
WHERE (1=1) ))
WHERE step_0 = 1 ))
GROUP BY aggregation_target,
Expand Down
Loading