Add state/structure to OTAP batch + sorting/delta decoding transforms

[albertlockett]

Part of #449

This PR adds:

• A new structure called OtapBatch that encapsulates all the record batches associated with a BatchArrowRecord. It provides efficient methods for setting & getting the record batches by payload type. This has been integrated into the decoder code.
• Adds new otap::transform module, methods for sorting record batch by parent ID and removing delta encoding.
• Adds helper methods for updating schema/field metadata, and invokes these methods to update the metadata with the current state of the record batches. (state including things like the encoding of the parent ID column, and the column sort order).

The scope of this PR has somewhat changed. I'll leave the original description below, for posterity.

---

Original Description:

Restores some of the code from #447 , but with a slightly different implementation of the "Version 2" attribute store.

• This new version assumes that the record batch it receives is sorted by the parent ID.
  • To do this sort, we first materialize the parent IDs using the code in this PR: feat: add function to materialize transport optimized parent ids #455 and then sort the record batch (see implementation of sort_by_parent_id).
• This new store also lazily materializes the KeyValues, by returning an iterator from it's attribute_by_delta_id method (similar to what we did in [WIP] AttributesStore optimization and/or OTAP pdata research #447).
• Note that this new attribute store implementation does not support random access. It expects attribute_by_delta_id to be called with the delta ID of each parent ID.

The benchmark results are interesting.. The v2 store gets slightly worse performance for small batch sizes (128), but gets much better performance for larger batch sizes (8092), which is encouraging!

materialize_parent_ids/v1_attr_store/128
                        time:   [7.6742 µs 7.6967 µs 7.7190 µs]

materialize_parent_ids/v2_attr_store/128
                        time:   [10.495 µs 10.515 µs 10.534 µs]

materialize_parent_ids/v1_attr_store/1536
                        time:   [118.21 µs 118.76 µs 119.32 µs]

materialize_parent_ids/v2_attr_store/1536
                        time:   [89.421 µs 89.582 µs 89.765 µs]

materialize_parent_ids/v1_attr_store/8092
                        time:   [1.2816 ms 1.2888 ms 1.2966 ms]

materialize_parent_ids/v2_attr_store/8092
                        time:   [464.49 µs 465.22 µs 466.00 µs]

Still very much WIP. One thing that needs to be sorted is where/when we apply this transform materialize parent IDs and sort the record batch, and how we track the state of the batch. This PR is currently doing it decode::decoder::Consumer::consume_bar, which may not be the right place.

We might consider not merging all the code in this PR straight away, as some of this work is exploratory/proof of concept

[codecov]

Codecov Report

Attention: Patch coverage is 93.91144% with 33 lines in your changes missing coverage. Please review.

Project coverage is 62.91%. Comparing base (a629271) to head (de776c9).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #468      +/-   ##
==========================================
+ Coverage   62.27%   62.91%   +0.63%     
==========================================
  Files         190      193       +3     
  Lines       27291    27696     +405     
==========================================
+ Hits        16995    17424     +429     
+ Misses       9760     9737      -23     
+ Partials      536      535       -1     

Components	Coverage Δ
otap-dataflow	73.76% <ø> (ø)
beaubourg	67.19% <ø> (ø)
otel-arrow-rust	67.37% <93.91%> (+3.07%)	⬆️
query_abstraction	81.42% <ø> (ø)
syslog_cef_receivers	99.17% <ø> (ø)
otel-arrow-go	52.93% <ø> (+0.02%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[albertlockett]

@jmacd please take a look when you get a chance? This is some followup from last week's SIG meeting

[albertlockett]

One thing that needs to be sorted is where/when we apply this transform materialize parent IDs and sort the record batch, and how we track the state of the batch.

Link to thread about answering some of these questions: https://cloud-native.slack.com/archives/C08RRSJR7FD/p1747847973678329

[albertlockett]

Update 05/22 -- added some structure around OtapBatch and refactored decoder to use this. Started adding helper methods to update schema metadata when transforming record batches

[jmacd]

How are multiple columns supported? I've seen this set to a single column, so far. Would it make sense to have SORT_COL0, SORT_COL1, ... to indicate the successive sort orders?

[albertlockett]

I was thinking maybe we could have some delimiter. E.g. I'm imagining how we'd represent the default sorting of attributes from the golang exporter, it would be type,key,value . Do you think that'd work?

[albertlockett]

Update 05/23 - after discussion w/ @jmacd , decided it's probably best to merge only what we are pretty sure we'll need in this PR, which does not include the AttributeStoreV2 or the benchmark for it. I'm going to stash that code on a branch called attribute-store-optimization-2 on my fork: https://github.com/open-telemetry/otel-arrow/compare/main...albertlockett:otel-arrow:attribute-store-optimizer-2?expand=1