[Tracker] Time Series Engine GA

[ankitsultana]

Tracker task for pending items for the Time Series Engine.

• Add integration tests for Time Series Engine. [timeseries] Introducing integration tests for time series engine #15960
• Remove reliance on table-type suffix. Presently users need to specify table name with type. This is because of the logic in TableScanVisitor which calls routing manager directly based on the user provided table name.
• Re-think HTTP API. We had gone with a Prometheus compatible API, but I don't see any value in it. Moreover, there are some Pinot query related stats that we would like to return in the response. Maybe we should consider returning in the Broker response format itself?
• Enable standard auth integrations
• Integrate with query event listeners
• Support Explain Plans (this is dependent on HTTP API)
• Add docs in the Apache Pinot wiki
• Support for Hybrid Tables
• Evaluate merge with Multistage Engine
• Fix the plan serialization scheme. Currently it relies on a hacky Jackson based approach.
• Think about Exemplar support.

... more to be added soon

Footnotes

Merging with MSE

Right now we have very little custom code for the Time Series Engine left. But still, we should see if we can integrate with the MSE Runtime to take advantage of stats propagation, plan serde, etc. We recently also contributed MSE Lite Mode with runInBroker query option, which is similar to the execution model we are using in the Time Series Engine, which would facilitate the integration.

At a high-level, I think these are the main things involved in merging TSE with MSE:

1. Adding new PlanNode types for running chains of BaseTimeSeriesPlanNode. Then in the time series planner could simply output a tree of PlanNode that could be handled in theory the same way by the MSE.
2. Adding new MSE Operator corresponding to the PlanNode mentioned above.
3. Handling compilation of the new PlanNode and the leaf stage in the server.
4. Switching to MseBlock for data transfers. We could still use the TimeSeriesBlock for in-memory computations and allowing users to implement custom operators.

[shauryachats]

@shauryachats thanks, can pick this up.