Reductor pattern to compute multiple statistics in one iterator pass.

[Evian-Zhang]

It is very common when we need multiple statistics of one data (for example, we may need both min and stddev at same time). However, current API design of statrs only allows us to compute those statistics by iterating the same data multiple times, which may be unnecessary for some statistics.

There are already some discussions in the rust forum (How to design a composable iterator adapter/consumer API?, Adding Iterator::extrema, argmin, argmax), and some one has proposed the crate reductor, which seems be able to solve this problem. However, the statistics provided by that crate is far less than the statrs crate.

Would you integrate the reductor pattern into statrs to solve this problem?

[YeungOnion]

If I understand this correctly, we would only need to define the recursive cases of the statistics we have, then we would match the API needed for a reductor pattern. This would allow some users to write the code using fold.

This would help with reducing the repeated computation, but regarding ergonomics, I don't have an idea on how to approach the reductor pattern since there's not a notion of an identity element (the starting element to reduce from) for the moments. And the higher the order the more inputs needed for it to be defined.

Do you have any insight on how you might want this to work for an example like (mean, variance)?

---

Note:

There's an approach to this sample streaming for all moments, Wiki Algorithms for Calculating Variance, Parallel Algorithms. Welford's algorithm is the single sample for the first two moments; also on that same article.

[YeungOnion]

I don't want to propose an idea instead of your, but I don't wish to forget mine,

An approach that's not affecting your input before you share it

Additionally, the update requires the lower order moments, so this could be a const generic case for expressing, "I want the first K moments from an iterator of numbers",

let count, mean, m2, m3 =
    reduce_to_moments<4>(data.iter());

or alternatively,

let count, mean, variance, skewness, kurtosis =
    Moment::standardize(reduce_to_moments<5>(data.iter()));

[Evian-Zhang]

Thank you very much for your kind response!

Do you have any insight on how you might want this to work for an example like (mean, variance)?

The readme code in reductor crate is a very good example (let me simplify a bit more):

use reductor::{Reduce, Reductors, Sum, Min, Max};

fn process_samples(
    samples: &[i32]
) {
    let Reductors((Sum::<i32>(sum), Min::<Option<i32>>(min), Max::<Option<i32>>(max))) = samples
        .iter()
        .reduce_with();

    // ...
}

In a high-level view, this transforms the original imperative programming paradigm into the declarative programming paradigm. Users only need to declare what they want, and they can get, of course, what they want (moreover, the target array is only iterated once, instead of three times).