added 'storage_transformers' to valid_encodings

[JMorado]

This PR adds "storage_transformers" to the valid encodings of a variable, thus allowing Zarr stores to be written using the
new sharding storage transformers.

Example of usage:

import xarray as xr
import numpy as np
from zarr._storage.v3_storage_transformers import ShardingStorageTransformer
from zarr._storage.v3 import DirectoryStoreV3

# Dummy dataset
da = xr.DataArray(np.random.randn(1000, 1000), dims=("x", "y"))
ds = xr.Dataset(dict(dummy_var=da))
ds = ds.chunk({"x": 100, "y": 100})

# Sharded store and sharding transformer
store = DirectoryStoreV3("/home/user/dummy-sharded-store.zarr")
transformer = ShardingStorageTransformer("indexed", chunks_per_shard=(5, 1))

# Write sharded store
ds.to_zarr(
    store,
    encoding={"dummy_var": {"storage_transformers": [transformer]}},
    zarr_version=3,
)

[jhamman]

This looks great @JMorado! Would you mind adding a test that utilizes this encoding parameter?

[JMorado]

Hi @jhamman,

I have added a test and corrected a problem that I had previously missed. It turns out that a lock must be used to ensure the correct writing of a sharded zarr store. Everything seems to be working as expected now. Here are some comments:

• Currently, a lock is passed to ArrayWriter even if only one variable is being sharded. While I believe that locks would not be needed to write non-sharded variables, I don't think it is possible to restrict their usage to specific variables, since the zarr store is written as a whole. Any thoughts on this?
• Related to the previous point, I think the usage of locks could even be made shard-specific, relaxing the writing of different shard files. I also believe this is not currently possible, but let me know if there is a solution.
• For the test, importing ShardingStorageTransformer is currently a bit awkward, but it can be neated in the future when ShardingStorageTransformer becomes available from the top-level zarr namespace.

Let me know what you think.

[rabernat]

It's great to see this PR get started in Xarray! Thanks @JMorado!

From the perspective of a Zarr developer, the sharding feature is still highly experimental. The API may change significantly. While the sharding code is released in the sense that it is available deep in Zarr, it is not really considered part of the public API yet.

So perhaps it's a bit too early to be doing this?

[rabernat]

Regarding locks, I think we need to think hard about the best way to deal with this across the stack. There are a couple of different options:

• Current status: just use a global lock on the entire array--super inefficient
• A bit better: use per-variable locks
• Even better: have locks at the shard level. This would allow concurrent writing of shards
• Alternative which accomplishes the same thing: expose different virtual chunks when reading vs. writing. When writing, the writer library (e.g. Xarray or Dask) would see the shards as the chunks (with a lower layer of the stack handling breaking the shard down into chunks). When reading, the individual, smaller chunks would be accessible.

Note that there are still some deep inefficiencies in the way zarr-python writes shards (see zarr-developers/zarr-python#1338). I think we should be optimizing things at the Zarr level first, before implementing workarounds in Xarray.

[dcherian]

Does it makes sense to create a new backend in a new project to enable experimentation?

[jhamman]

I think we can close this in favor of #9948. Thanks @JMorado for the early attempt here!