Summary broker query log for multistage engine queries

[dang-stripe]

We've found it difficult to do post-hoc analysis of query timeouts/failures on the multistage engine without access to the stage stats. Debugging these queries have involved:

1. Piecing together distributed server logs to understand where the query failed or slowed down
2. Re-running the query, raising the timeout as needed, and collecting the stage stats adhoc

Given the complexity of the multistage engine, it'd be ideal for these queries to have a concise summary log from the broker similar to how single stage does it here. Some metadata that'd be helpful for debugging:

1. Query success or failure
2. Query latency
3. Concise representation of the stage graph (1->[2,3],2->4,etc) including which stages are leafs
4. Which stages of the query succeeded or failed
5. For stages that failed, which servers failed the complete the stage
6. How long (wall clock) time did each stage take

This would make it much easier to debug production issues on the fly and answer questions like:

1. Is the query failing due to single server or multiple?

cc @gortiz @Jackie-Jiang @jadami10

[gortiz]

I've created #16007, where I'm modifying log in case of error to include the stats. I've also included another log that creates the same summary in case of successful execution, but that log can be very spammy, so it is logged at DEBUG level to be disabled by default. A new marker can be used to enable just that log instead of all debug logs from the class. See the PR to see an example. If needed, Log4j2's BusterFilter can be used to apply a rate to these logs.

The stats already contain most of the information requested here:

1. Query success or failure: Different messages are logged on each case
2. Query latency: Included in the summary
3. Concise representation of the stage graph (1->[2,3],2->4,etc) including which stages are leafs: This is included in the stats.
4. Which stages of the query succeeded or failed: We cannot include that right now, more info below.
5. For stages that failed, which servers failed the complete the stage: In order to have that info we need to merge MSE error management improvements #15919 first.
6. How long (wall clock) time did each stage take: Stats for each operator include clockTimeMs, which is calculated as executionTimeMs / parallelism. Right now, executionTimeMs is 0 for stages that fail, but that is something we will try to improve.

It is not easy to include how many servers have failed because Pinot aborts the execution as soon as the first error is received. We may decide to change that in the future, but it would require significant changes to the code and likely increase latency in error handling, so it is unlikely that we will invest resources in this area in the short term.

[dang-stripe]

Thanks @gortiz!

Which stages of the query succeeded or failed: We cannot include that right now, more info below.
It is not easy to include how many servers have failed because Pinot aborts the execution as soon as the first error is received.

Is it possible to derive this from the stage stats? Or is that lacking information as well? Does it differ between a hard error (segments unavailable at leaf) vs. timeout case?

For stages that failed, which servers failed the complete the stage: In order to have that info we need to merge #15919 first.

How does this behave when queries timeout at a leaf stage vs. intermediate stage? Edit: Ah, I see some of this is being handled in #16075