How to configure Doris to treat the entire JSON string as a value for a field in Kafka stream load? #30187
Replies: 4 comments 3 replies
-
|
Beta Was this translation helpful? Give feedback.
-
@LemonLiTree 试验了 "jsonpaths" = "["$.vehicleId","$.timestamp","$.RECORDS"]",不行,不识别,对应表的列写入为null,能否在doris的后续版本中增加一个类似$.RECORDS作为完整json串作为一列的功能呢?谢谢! |
Beta Was this translation helpful? Give feedback.
-
刚好我也需要 |
Beta Was this translation helpful? Give feedback.
-
it's supported now. you need to use the latest version and add this on propeties |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
比如我的kafka主题中json为{"vehicleId":"A1001", "timestamp":1704038400000, "level":1, "name":"mike", "food":"cake"}
我希望整个json的内容作为表的一列mydata写入,表为
CREATE TABLE shops
(
vehicleId
varchar(20) ,timestamp
LARGEINT(20),runDate Datetime,
mydata String
)
ENGINE=olap
UNIQUE KEY(
vehicleId
,timestamp
,runDate
)PARTITION BY RANGE( runDate )
(
)
DISTRIBUTED BY HASH(
vehicleId
) BUCKETS 10下面我配置了stream load任务
CREATE ROUTINE LOAD qdyk.mycarbu ON shops
COLUMNS( vehicleId, timestamp, mydata, runDate=date_trunc(from_unixtime(timestamp/1000, 'yyyy-MM-dd HH:mm:ss'), 'day') ),
PROPERTIES
(
"desired_concurrent_number"="3",
"max_batch_interval" = "20",
"max_batch_rows" = "300000",
"max_batch_size" = "209715200",
"max_error_number"="999999999",
"strict_mode" = "false",
"format" = "json",
"jsonpaths" = "["$.vehicleId","$.timestamp","."]",
"strip_outer_array" = "false"
)
FROM KAFKA
(
"kafka_broker_list" = "10.100.10.101:9092",
"kafka_topic" = "rc_shops",
"property.group.id" = "myshops",
"property.kafka_default_offsets"="OFFSET_BEGINNING"
);
其中 "jsonpaths" = "["$.vehicleId","$.timestamp","."]",中我希望.代表整个JSON串内容,但是怎么试验,最后写入的都是null,
请问怎么设置jsonpath才能使得一条消息会成为mydata这列的内容,,我试验了$.*, $*等各种写法都不行,都是null, 谢谢
Beta Was this translation helpful? Give feedback.
All reactions