NamedArray tracking issue

[dcherian]

@andersy005 I think it would be good to keep a running list of NamedArray tasks. I'll start with a rough sketch, please update/edit as you like.

• Refactor out NamedArray base class (initial refactor for NamedArray #8075)
• publicize design doc: Scientific Python | Pangeo | NumPy Mailist
• Migrate VariableArithmetic to NamedArrayArithmetic (Migrate VariableArithmetic to NamedArrayArithmetic #8244)
• Migrate ExplicitlyIndexed array classes to array protocols
• MIgrate from *Indexer objects to .oindex and .vindex on ExplicitlyIndexed array classes
• Enable explicit use of key tuples (instead of *Indexer objects) in indexing adapters and explicitly indexed arrays #8870
• Migrate unary ops
• Migrate binary ops
• Migrate nanops.py
• Avoid "injecting" reduce methods potentially by using generate_reductions.py? (Move Variable aggregations to NamedArray #8304)
• reprs and formatting.py
• parallelcompat.py
• pycompat.py (Migrate VariableArithmetic to NamedArrayArithmetic #8244)
• Give NamedArray Generic dimension type #8276
• have test_variable.py test both NamedArray and Variable
• Arrays with unknown shape NamedArray.shape does not support unknown dimensions #8291
• keep_attrs for NamedArray #8306
• pydata/xarray as monorepo for Xarray and NamedArray #8310
• Should NamedArray be interchangeable with other array types? or Should we support the axis kwarg? #8333
• Try to preserve imports from xarray.core/* by importing namedarray functionality into xarray.core/*

xref #3981

[headtr1ck]

For the operators we will run into the following issue:

Currently Variable + DataArray will return a DataArray because Variable will return NotImplemented and then python calls DataArray.__radd__. This is hard coded.

If you want to make NamedArray independent from xarray, you need to find a way to solve this. I assume numpy has the same issues, maybe check over there how it works?

[dcherian]

@headtr1ck or @Illviljan would one of you be able to handle moving over unary and binary ops and maybe even nanops. I think that might involve a bunch of typing wrangling that you two would be best suited to fix

[headtr1ck]

I can have a look at it :)
By the way: do we want to get rid of the fastpath argument to the __init__ and replace it with a custom classmethod? I think it is confusing to have this in a public interface.

[Illviljan]

How much should NamedArray conform to the array api standard?

For example, the array api does not implement the methods, .imag, .real, .astype.
https://data-apis.org/array-api/latest/API_specification/index.html

I'm leaning towards being strict with the array api and to remove them because they are causing some truly difficult typing issues (#8281, #8294).
If all these shape/dim/dtype-changing methods were moved to functions instead the type hinting would be much more straightforward.

[dcherian]

@TomNicholas do you have time to move over parallelcompat.py to namedarray/? It'll be a lot quicker if you did it ;)

[dcherian]

How much should NamedArray conform to the array api standard?

The API is certainly going to be a superset of the array API (that doesn't even have NaN skipping reductions!)

I'm leaning towards being strict with the array api and to remove them because they are causing some truly difficult typing issues

I don't think typing should limit the API we offer.

For example, the array api does not implement the methods, .imag, .real, .astype.
data-apis.org/array-api/latest/API_specification/index.html

Can we add these back please? As of now, our plan is to include them. https://github.com/pydata/xarray/blob/main/design_notes/named_array_design_doc.md#attributes-to-be-preserved-from-xarrayvariable

[TomNicholas]

The API is certainly going to be a superset of the array API (that doesn't even have NaN skipping reductions!)

We should also remember that the array API is not fixed in stone - it can and already has changed to include more functions. We should be raising issues there to get functions we want added. See data-apis/array-api#187 (comment) and data-apis/array-api#669 for example.

I don't think typing should limit the API we offer.

+1

TomNicholas do you have time to move over parallelcompat.py to namedarray/? It'll be a lot quicker if you did it ;)

I'll have a look now!

[headtr1ck]

We should slow down with the PRs, haha. The merge conflicts will be a nightmare

[kgryte]

Hey all! Don't want to hijack this thread, but wanted to respond to a few of the comments above regarding the Array API.

For example, the array api does not implement the methods, .imag, .real, .astype.
data-apis.org/array-api/latest/API_specification/index.html

@Illviljan In the Array API standard, we've preferred functional APIs over methods and have the equivalent of imag(), real(), and astype().

The API is certainly going to be a superset of the array API (that doesn't even have NaN skipping reductions!)

@dcherian There has been some discussion about adding support for NaN reductions to the specification (see data-apis/array-api#621); however, there is still some debate as to what form that should take. E.g., whether we actually want nan* variants, a single NaN reduction wrapper, or kwarg support to existing reductions. You're encouraged to participate in that thread to help provide input regarding what makes the most sense from your POV.

We should also remember that the array API is not fixed in stone - it can and already has changed to include more functions. We should be raising issues there to get functions we want added.

@TomNicholas Indeed! May be good to list all the current Array API pain points (if not done already), so that we can potentially prioritize. We could also potentially arrange for xarray devs to present at one of the Consortium workgroup meetings.

Regardless, thanks for considering the Array API standard. We'd love to get additional feedback on the specification over on the specification repo. :)