Commit afbcc15
authored
Updated date filtering in Tinybird pipes to improve performance (#25707)
ref
https://linear.app/ghost/issue/NY-865/analytics-sources-not-populating-for-tangle-due-to-408-timeouts
## Problem
The filtered_sessions.pipe date filters wrapped the indexed timestamp
column in functions, causing "index blindness":
-- Before (prevents index usage):
```
toDate(toTimezone(timestamp, timezone)) >= date_from
toDate(toTimezone(timestamp, timezone)) <= date_to
```
Since _mv_hits uses ENGINE_SORTING_KEY "site_uuid, timestamp,
session_id", wrapping timestamp in functions prevents ClickHouse from
using the sparse index, resulting in full scans.
## Solution
Transform the comparison values instead of the column, keeping timestamp
bare:
-- After (index-friendly):
```
timestamp >= toDateTime(date_from, timezone)
timestamp < toDateTime(date_to, timezone) + interval 1 day
```
The + interval 1 day syntax (rather than dateAdd()) is required because
Tinybird's {{ Date() }} template renders as a string literal, and
dateAdd() on strings produces a DateTime with milliseconds that
toDateTime() can't parse with a timezone argument.
## Semantic equivalence
Old | New | Meaning
-- | -- | --
toDate(toTimezone(ts, tz)) >= date | ts >= toDateTime(date, tz) | Events
on/after midnight of date in user's timezone
toDate(toTimezone(ts, tz)) <= date | ts < toDateTime(date, tz) + 1 day |
Events before midnight of date+1 in user's timezone1 parent baf9a83 commit afbcc15
File tree
1 file changed
+14
-6
lines changed- ghost/core/core/server/data/tinybird/pipes
1 file changed
+14
-6
lines changedLines changed: 14 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | | - | |
16 | | - | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
17 | 25 | | |
18 | 26 | | |
19 | 27 | | |
| |||
41 | 49 | | |
42 | 50 | | |
43 | 51 | | |
44 | | - | |
| 52 | + | |
45 | 53 | | |
46 | 54 | | |
47 | | - | |
| 55 | + | |
48 | 56 | | |
49 | 57 | | |
50 | 58 | | |
51 | | - | |
| 59 | + | |
52 | 60 | | |
53 | 61 | | |
54 | | - | |
| 62 | + | |
55 | 63 | | |
56 | 64 | | |
57 | 65 | | |
| |||
0 commit comments