Skip to content

Commit afbcc15

Browse files
authored
Updated date filtering in Tinybird pipes to improve performance (#25707)
ref https://linear.app/ghost/issue/NY-865/analytics-sources-not-populating-for-tangle-due-to-408-timeouts ## Problem The filtered_sessions.pipe date filters wrapped the indexed timestamp column in functions, causing "index blindness": -- Before (prevents index usage): ``` toDate(toTimezone(timestamp, timezone)) >= date_from toDate(toTimezone(timestamp, timezone)) <= date_to ``` Since _mv_hits uses ENGINE_SORTING_KEY "site_uuid, timestamp, session_id", wrapping timestamp in functions prevents ClickHouse from using the sparse index, resulting in full scans. ## Solution Transform the comparison values instead of the column, keeping timestamp bare: -- After (index-friendly): ``` timestamp >= toDateTime(date_from, timezone) timestamp < toDateTime(date_to, timezone) + interval 1 day ``` The + interval 1 day syntax (rather than dateAdd()) is required because Tinybird's {{ Date() }} template renders as a string literal, and dateAdd() on strings produces a DateTime with milliseconds that toDateTime() can't parse with a timezone argument. ## Semantic equivalence Old | New | Meaning -- | -- | -- toDate(toTimezone(ts, tz)) >= date | ts >= toDateTime(date, tz) | Events on/after midnight of date in user's timezone toDate(toTimezone(ts, tz)) <= date | ts < toDateTime(date, tz) + 1 day | Events before midnight of date+1 in user's timezone
1 parent baf9a83 commit afbcc15

File tree

1 file changed

+14
-6
lines changed

1 file changed

+14
-6
lines changed

ghost/core/core/server/data/tinybird/pipes/filtered_sessions.pipe

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,16 @@ SQL >
1212
from _mv_hits
1313
where
1414
site_uuid = {{ String(site_uuid, 'mock_site_uuid', description="Tenant ID", required=True) }}
15-
{% if defined(date_from) %} and toDate(toTimezone(timestamp, {{String(timezone, 'Etc/UTC', description="Site timezone", required=True)}})) >= {{ Date(date_from) }} {% else %} and toDate(toTimezone(timestamp, {{String(timezone, 'Etc/UTC', description="Site timezone", required=True)}})) >= timestampAdd(today(), interval -7 day) {% end %}
16-
{% if defined(date_to) %} and toDate(toTimezone(timestamp, {{String(timezone, 'Etc/UTC', description="Site timezone", required=True)}})) <= {{ Date(date_to) }} {% else %} and toDate(toTimezone(timestamp, {{String(timezone, 'Etc/UTC', description="Site timezone", required=True)}})) <= today() {% end %}
15+
{% if defined(date_from) %}
16+
and timestamp >= toDateTime({{ Date(date_from) }}, {{String(timezone, 'Etc/UTC', description="Site timezone", required=True)}})
17+
{% else %}
18+
and timestamp >= toDateTime(dateAdd(day, -7, today()), {{String(timezone, 'Etc/UTC', description="Site timezone", required=True)}})
19+
{% end %}
20+
{% if defined(date_to) %}
21+
and timestamp < toDateTime({{ Date(date_to) }}, {{String(timezone, 'Etc/UTC', description="Site timezone", required=True)}}) + interval 1 day
22+
{% else %}
23+
and timestamp < toDateTime(dateAdd(day, 1, today()), {{String(timezone, 'Etc/UTC', description="Site timezone", required=True)}})
24+
{% end %}
1725
{% if defined(member_status) %}
1826
and member_status IN (
1927
select arrayJoin(
@@ -41,17 +49,17 @@ SQL >
4149
site_uuid = {{ String(site_uuid, 'mock_site_uuid', description="Tenant ID", required=True) }}
4250
{% if defined(date_from) %}
4351
{# Filter from specified start date #}
44-
and toDate(toTimezone(first_pageview,{{ String(timezone, 'Etc/UTC', description="Site timezone", required=True) }})) >= {{ Date(date_from) }}
52+
and first_pageview >= toDateTime({{ Date(date_from) }}, {{ String(timezone, 'Etc/UTC', description="Site timezone", required=True) }})
4553
{% else %}
4654
{# Default to last 7 days if no start date provided #}
47-
and toDate(toTimezone(first_pageview,{{ String(timezone, 'Etc/UTC', description="Site timezone", required=True) }})) >= timestampAdd(today(), interval -7 day)
55+
and first_pageview >= toDateTime(dateAdd(day, -7, today()), {{ String(timezone, 'Etc/UTC', description="Site timezone", required=True) }})
4856
{% end %}
4957
{% if defined(date_to) %}
5058
{# Filter to specified end date #}
51-
and toDate(toTimezone(first_pageview, {{String(timezone, 'Etc/UTC', description="Site timezone", required=True)}})) <= {{ Date(date_to) }}
59+
and first_pageview < toDateTime({{ Date(date_to) }}, {{ String(timezone, 'Etc/UTC', description="Site timezone", required=True) }}) + interval 1 day
5260
{% else %}
5361
{# Default to today if no end date provided #}
54-
and toDate(toTimezone(first_pageview, {{String(timezone, 'Etc/UTC', description="Site timezone", required=True)}})) <= today()
62+
and first_pageview < toDateTime(dateAdd(day, 1, today()), {{ String(timezone, 'Etc/UTC', description="Site timezone", required=True) }})
5563
{% end %}
5664
{% if defined(source) %} and source = {{ String(source, description="Source to filter on", required=False) }} {% end %}
5765
{% if defined(device) %} and device = {{ String(device, description="Device type to filter on", required=False) }} {% end %}

0 commit comments

Comments
 (0)