[Question] Would contribution of logging wrapper and a "readahead" wrapper for ObjectStore be wanted?

[m09526]

Which part is this question about

We have developed two ObjectStore wrapper implementations in a similar style to LimitStore, PrefixStore, etc. which we would like to contribute to this repository if they are desired. Based on the description below, would there be enough of a use-case for the creation of two PRs for the implementations?

Describe your question
The two implementations are as follows:

*** LoggingObjectStore
Our implementation, which a PR would be based off is here.

LoggingObjectStore Wraps another ObjectStore implementation and writes all operations "GETs", "PUTs", "LIST", etc, to Rust's standard logger.

This was extremely helpful when debugging an application we had written that requested many files from Amazon S3 in small chunks. It was helpful to be able to see exactly when, where and what file parts were being requested by our application to gain a better understanding of what was going on deep inside some library code.

*** ReadaheadStore
Our implementation, which a PR would be based off is here.

This wraps another ObjectStore and attempts to re-use opened data streams on an object during GET operations. Inspired by the functionality in Apache Hadoop's Hadoop-AWS module, the readahead store keeps a data stream open once a client has finished reading from it. When a new GET operation is requested on the same object, if the starting read position is within a configurable distance of the last read, we can re-use the existing stream instead of opening a new stream.

This can drastically reduce the number of new GET operations that have to be started against the wrapped object store in sequential reads of objects. This can have a performance improvement due to fewer network requests being created.

If a new GET operation is requested that is before the position of a previous stream, or too far beyond the previous position, a new request is performed on the underlying store. The number of concurrent open streams per object, maximum time-to-live and maximum safe "readahead" are configurable.

Additional context
We would like to contribute these two implementations to the wider community.

[tustvold]

I think a logging store is potentially useful, although it might make more sense as an HttpService wrapper now that they exist.

With regards to read-ahead, I'm a little torn. The design of object_store is to encourage people to avoid using cursor-based access patterns that warrant this sort of incremental read-ahead pattern - https://docs.rs/object_store/0.12.0/object_store/#why-not-a-filesystem-interface. Instead people should identify the range up front, and fetch and process it, potentially in a streaming fashion where applicable.

For example - https://docs.rs/arrow-csv/latest/arrow_csv/reader/index.html#async-usage

That being said I appreciate people might have existing codebases with such access patterns, but I'm not sure to what extent we want to encourage this...

[crepererum]

I think a logging store is potentially useful, although it might make more sense as an HttpService wrapper now that they exist.

Depends what the intention of "logging" is. If it's for network IO, I agree. If it's to log semantic operations, I think it should be an ObjectStore wrapper. This way it could also be used for in-mem or FS-backed stores (or any third-party store).

[m09526]

Depends what the intention of "logging" is. If it's for network IO, I agree. If it's to log semantic operations, I think it should be an ObjectStore wrapper. This way it could also be used for in-mem or FS-backed stores (or any third-party store).

In this case, we want to log the semantic operations, i.e. what ObjectStore level operations has other code called on it. This isn't designed to log at the network layer, and certainly we want it to work across implementations, e.g. local file systems and in-memory backed ones. In our case, we used it to discover how Apache DataFusion was accessing remote object stores. This made analysing it's access patterns on files incredibly quick and easy as we could see how it was retrieving data and how those changed with different DataFusion queries.

With regards to read-ahead, I'm a little torn. The design of object_store is to encourage people to avoid using cursor-based access patterns that warrant this sort of incremental read-ahead pattern - https://docs.rs/object_store/0.12.0/object_store/#why-not-a-filesystem-interface. Instead people should identify the range up front, and fetch and process it, potentially in a streaming fashion where applicable.

Maybe read-ahead isn't the most appropriate, descriptive name for it. It's primary purpose is to reduce the number of network requests generated when talking to a remote object store, especially when those requests are chargeable or carry a small but measurable latency. Implemented as a wrapper around any other ObjectStore implementation, it acts transparently, re-using data streams where it can. When processing large Parquet files with the logger implementation above, we could view our implementation making 1 ObjectStore request for each Parquet row group each of which translated to 1 network request. For a large file with 100's of row groups, using the read-ahead wrapper, we reduced the number of network requests from 100's to a small handful. That is the intent here.

The current implementation doesn't attempt any read-ahead if the GetResultPayload is File, so it avoids pointlessly acting on local file systems.

[alamb]

A related discussion is here:

• [DISCUSSION] [object_store] New crate with object store combinators / utilitles #14

I am personally in favor of such wrappers (I have actually written a logging one myself!)

The read ahead one I am not as sure of -- I need to study it more

I propose we add a new crate (in this repo) with object_store_wrappers for these extensions

That way anyone who wants the wrappers can explicitly opt in and we keep the core crate as simple as possible

[m09526]

We'd be happy to contribute LoggingObjectStore to a separate crate if that's preferred.