[otap-dataflow] Retry Control Processor

[lquerel]

The purpose of this processor is to generalize support for the retry mechanism for recoverable errors. This approach eliminates the need to implement retry logic within exporters and other components that require this functionality.

Benefits: a single configuration, one unified retry logic, less work for exporters.

This processor will utilize the concept of control messages of types Ack and Nack. An exporter that fails to process a message must emit a Nack control message (see details of the fields below). When message processing succeeds, the exporter must emit an Ack control message.

Definition of the Ack and Nack messages.

pub enum ControlMessage {
    Ack {
        msg_ids: Vec<String>,
    },
    Nack {
        msg_ids: Vec<String>,
        error: ErrorDetail,
        retry_policy: Option<RetryPolicy>,
    },
}

pub struct RetryPolicy {
    pub retry_delay_ms: u64,
}

pub struct ErrorDetail {
    pub recoverable: bool,
    pub reason: String,
}