Skip to content

Use Correlation ID instead of query id in PerQueryCPUMemResourceUsageAccountant #16015

Open
@vrajat

Description

@vrajat

PerQueryCPUMemResourceUsageAccountant uses request id as the id as the key to accumulate resource usage statistics. However the request id is different b/w broker and server as well as b/w OFFLINE/REALTIME requests in the server. Check the examples below - one from SSE and another from MSE:

Image
Image

  • query id is numerical in the broker.
  • query id has a prefix as well as two types of suffix O | R on the server depending on the table that is being processed.

Due to these differences, aggregation will not happen correctly on the server AND it cumbersome to track resource usage across components.

Use (correlation id)[https://docs.pinot.apache.org/users/user-guide-query/query-correlation-id] as the key. Check the example below where the query id is consistent across all threads.

Image

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions