[otap-dataflow] Failover Control Processor

[lquerel]

The Failover Control Processor is designed to generalize the failover mechanism within the pipeline system. It monitors upstream processors for repeated processing failures (indicated by Nack messages after exhausting retry attempts) and then redirects the affected messages to a secondary pipeline branch, known as the fallback branch, to ensure continued message processing.

Purpose

Currently, each processor or exporter needing failover functionality must individually implement failover logic. The Failover Control Processor centralizes and generalizes this logic, simplifying configuration, reducing redundancy, and ensuring consistent failover behavior across the pipeline.

Behavior

• The Retry Processor emits a Nack control message after exhausting a configurable number of retry attempts.
• The Failover Control Processor captures these Nack messages.
• Upon receiving a Nack, the Failover Processor redirects the failed message(s) to the configured fallback pipeline branch for alternate processing.