[SR-3811] optimize high card count distinct

[dirtysalt]

Use another way to serialize/deserialize hahset in distinct agg state:

1. save packet size
2. better cache locality to do merge

Testing following SQL on ssb dataset, query time can be reduced from 60s -> 20s

SELECT COUNT (DISTINCT lo_orderkey), lo_linenumber FROM lineorder_flat group by lo_linenumber;

[imay]

@dirtysalt
I have one question. If this CL has been merged, can it be compatible with old version.
If not this will make our program can not grayscale upgrade

[dirtysalt]

@dirtysalt
I have one question. If this CL has been merged, can it be compatible with old version.
If not this will make our program can not grayscale upgrade

@imay Unfortunately not yet. But I think we can do something hacky to make it compatible.

Right now serialized format is like,

| size | capacity | control array | slot array|

and size is size_t, in most cases it should be less than (1<<63)-1(otherwise this hashtable is too huge). So we can tell this is an old serialized version if msb of size is 0. And for the further version, we put set msb of size to 1.

[dirtysalt]

@dirtysalt
I have one question. If this CL has been merged, can it be compatible with old version.
If not this will make our program can not grayscale upgrade

yeah, it's very hard to be backward compatible. Because old version of BE does not recognize new data format. So basically yes, it's can not be upgraded in a grayscale way.

[imay]

@dirtysalt
I have one question. If this CL has been merged, can it be compatible with old version.
If not this will make our program can not grayscale upgrade

yeah, it's very hard to be backward compatible. Because old version of BE does not recognize new data format. So basically yes, it's can not be upgraded in a grayscale way.

If we can not upgrade in a grayscale way, this would be bad for users.
So we should make it back compatible.

[DeepThinker666]

could you add some background for this pr?

[dirtysalt]

could you add some background for this pr?

Right, we are doing some benchmark comparisons with CK. And we find out that we don't have good performance when we are doing (count/distinct + group by) those cases. There are several optimization points:

1. better hashtable for inserting and finding
2. serialization and deserialization.

[dirtysalt]

running benchmark on ssb_100g with set parallel_fragment_exec_instance_num = 16

Query	SR Master	SR PR
Q01	1110	1086
Q02	1052	1069
Q03	400	360
Q04	176	174
Q05	15840	9526
Q06	2305	2037
Q07	1347	1261
Q08	326	337
Q09	6277	4898
Q10	2449	2189
Q11	6011	4749
Q12	1958	1832
Q13	10079	9475
Q14	6816	6764
Q15	9766	9243
Q16	6003	5844

# 100G
# lo_orderkey = 150,000,000
# lo_partkey = 1000,000
# lo_suppkey = 200,000
# lo_quantity = 50

# lo_linenumber = 7
# lo_supplycost = 15933
# lo_linenumber, lo_supplycost = 111531

-- Q01
select count (distinct lo_orderkey) from lineorder_flat;
-- Q02
SELECT COUNT (DISTINCT lo_partkey) FROM lineorder_flat;
-- Q03
SELECT COUNT (DISTINCT lo_suppkey) FROM lineorder_flat;
-- Q04
SELECT COUNT (DISTINCT lo_quantity) FROM lineorder_flat;

-- Q05
SELECT COUNT (DISTINCT lo_orderkey), lo_linenumber FROM lineorder_flat group by lo_linenumber;
-- Q06
SELECT COUNT (DISTINCT lo_partkey), lo_linenumber FROM lineorder_flat group by lo_linenumber;
-- Q07
SELECT COUNT (DISTINCT lo_suppkey), lo_linenumber FROM lineorder_flat group by lo_linenumber;
-- Q08
SELECT COUNT (DISTINCT lo_quantity), lo_linenumber FROM lineorder_flat group by lo_linenumber;

-- Q09
SELECT COUNT (DISTINCT lo_orderkey), lo_supplycost FROM lineorder_flat group by lo_supplycost;
-- Q10
SELECT COUNT (DISTINCT lo_partkey), lo_supplycost FROM lineorder_flat group by lo_supplycost;
-- Q11
SELECT COUNT (DISTINCT lo_suppkey), lo_supplycost FROM lineorder_flat group by lo_supplycost;
-- Q12
SELECT COUNT (DISTINCT lo_quantity), lo_supplycost FROM lineorder_flat group by lo_supplycost;

-- Q13
SELECT COUNT (DISTINCT lo_orderkey), lo_linenumber, lo_supplycost FROM lineorder_flat group by lo_linenumber, lo_supplycost;
-- Q14
SELECT COUNT (DISTINCT lo_partkey), lo_linenumber, lo_supplycost FROM lineorder_flat group by lo_linenumber, lo_supplycost;
-- Q15
SELECT COUNT (DISTINCT lo_suppkey), lo_linenumber, lo_supplycost FROM lineorder_flat group by lo_linenumber, lo_supplycost;
-- Q16
SELECT COUNT (DISTINCT lo_quantity), lo_linenumber, lo_supplycost FROM lineorder_flat group by lo_linenumber, lo_supplycost;

[SeaAndHillMe]

good to learn