Milestone Dec 2025 - Summary

[lquerel]

Main goals: 1) telemetry data plane engine maturity & system hardening, 2) first deliverable (beta version)

Engine Core

• Flow & Reliability Control
  • Back pressure
    • Backpressure Design & Principles #1252
    • Test Backpressure #1253
  • Ack/Nack and Retry Mechanism - Reliable delivery semantics
    • Retry Processor #509
  • Rate limit (no design yet)
    • Rate Limiter Design #888
    • Rate Limiter Implementation #1256
  • Failover & Recovery - Graceful routing/fallback mechanisms (no design yet)
    • Failover Processor Design #510
    • Failover Processor Implementation #1257
• Resource Control
  • Admission control
  • Per-pipeline CPU allocation
    • Support more advanced cpu allocation syntax #1135
  • Memory limit - Detect and prevent OOM / excessive buffering (no design yet)
    • Add a memorylimiter extension #173
    • Rate & Memory limiter framework #919
    • Container-aware CPU and memory limit detection #1134
    • Integration testing for memory limits #210
• Performance & Efficiency
  • Continuous Benchmark Pipeline - Automated profiling and regressions tracking
    • [otel-arrow-rust] Benchmarking test infrastructure #369
  • Core Engine Optimizations (CPU, memory, and allocator tuning)
  • Node-Level Optimizations (OTLP/OTAP receivers/exporters, Syslog recv, Batch, Attribute, Filter)
• Lifecycle Management & NUMA awareness
  • Graceful Shutdown
    • Support graceful shutdown of pipelines #871
    • Shutdown should gracefully drain remaining metric updates #1139
  • NUMA-Aware execution model - Per-NUMA node metric reporter, pinned memory (stretch goal)

Built-in Nodes

• Processors
  • Attribute Processor:
    • OTAP transform function to insert attributes #1035
    • Attribute Processor handle "invalid" transformation configuration through optimization #1036
    • transform_attributes could create an invalid sequence of encoded parent_ids using rename with no deletes #966
  • Filter Processor
    • Filter Processor #1213
  • Generic KQL|OTTL Processor
• Security Capabilities
  • TLS/mTLS support
    • TLS support for OTLP receiver/exporter #1258
    • TLS support for OTAP receiver/exporter #1259
    • TLS support for Syslog #1260
    • mTLS support for OTAP receiver/exporter #1261
  • Basic authentication (for receivers/exporters)
• Security Protection
  • Validation & Defense against invalid/malformed messages
  • Schema validation, payload length checks, parser hardening

Serviceability & Observability

• Control & Lifecycle

  • Admin APIs with TLS support (control plane)
  • Health endpoints (/readyz, /livez)
    • HTTP endpoints to expose metrics, status and health checks #872
  • Status endpoint (hierarchical observed state)
  • Live reconfiguration (add/remove/update pipelines) (stretch goal)

• Metrics & Telemetry

  • Prometheus-compatible metrics endpoint
  • Engine-level metrics/events
  • Node-level metrics/events
  • OTLP export (logs, metrics, traces)
  • Semantic convention registry for the engine and built-in nodes #1255

Integration & Compatibility

• Full OTel Data Model Support - Metrics, Logs, Events, and Traces (OTLP + OTAP)

Testing & Quality

• >= 70% code coverage
• Simulation-based / chaos testing (Turmoil, Madsim, ...) for the most critical part (stretch goal)

Documentation

• Main README.md and architecture overview
• Design documents (dataflow, node model, config model, ...)
• Built-in receivers/processors/exporters reference pages

Delivery & Packaging

• Linux [static] binary
• Docker container image
• Versioned build artifacts with metadata