DataArray.resample().apply() fails to apply custom function

[st-bender]

What happened?

Hi,
I try to apply a custom function to aggregate a resampled object via .apply().
Maybe it is a documentation issue? But I couldn't find it.

For example calculating the median by passing np.median fails with an error as shown in the log below.

What did you expect to happen?

I would expect the median or any other custom function to be calculated for the resampled data.
It seems to work with pure pandas.

Minimal Complete Verifiable Example

import numpy as np
import pandas as pd
import xarray as xr

idx = pd.date_range("2000-01-01", "2000-12-31")
data = xr.DataArray(np.random.randn(len(idx)), coords={"index": idx})

data.resample(index="M").apply(np.median)

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [425], in <cell line: 3>()
      1 idx = pd.date_range("2000-01-01", "2000-12-31")
      2 data = pd.Series(np.random.randn(len(idx)), index=idx).to_xarray()
----> 3 data.resample(index="M", label="left", loffset="15d").apply(np.median)

File ~/Work/miniconda3/envs/20211123_py39/lib/python3.9/site-packages/xarray/core/resample.py:279, in DataArrayResample.apply(self, func, args, shortcut, **kwargs)
    267 """
    268 Backward compatible implementation of ``map``
    269 
   (...)
    272 DataArrayResample.map
    273 """
    274 warnings.warn(
    275     "Resample.apply may be deprecated in the future. Using Resample.map is encouraged",
    276     PendingDeprecationWarning,
    277     stacklevel=2,
    278 )
--> 279 return self.map(func=func, shortcut=shortcut, args=args, **kwargs)

File ~/Work/miniconda3/envs/20211123_py39/lib/python3.9/site-packages/xarray/core/resample.py:253, in DataArrayResample.map(self, func, args, shortcut, **kwargs)
    210 """Apply a function to each array in the group and concatenate them
    211 together into a new array.
    212 
   (...)
    249     The result of splitting, applying and combining this array.
    250 """
    251 # TODO: the argument order for Resample doesn't match that for its parent,
    252 # GroupBy
--> 253 combined = super().map(func, shortcut=shortcut, args=args, **kwargs)
    255 # If the aggregation function didn't drop the original resampling
    256 # dimension, then we need to do so before we can rename the proxy
    257 # dimension we used.
    258 if self._dim in combined.coords:

File ~/Work/miniconda3/envs/20211123_py39/lib/python3.9/site-packages/xarray/core/groupby.py:1095, in DataArrayGroupByBase.map(self, func, args, shortcut, **kwargs)
   1093 grouped = self._iter_grouped_shortcut() if shortcut else self._iter_grouped()
   1094 applied = (maybe_wrap_array(arr, func(arr, *args, **kwargs)) for arr in grouped)
-> 1095 return self._combine(applied, shortcut=shortcut)

File ~/Work/miniconda3/envs/20211123_py39/lib/python3.9/site-packages/xarray/core/groupby.py:1115, in DataArrayGroupByBase._combine(self, applied, shortcut)
   1113 """Recombine the applied objects like the original."""
   1114 applied_example, applied = peek_at(applied)
-> 1115 coord, dim, positions = self._infer_concat_args(applied_example)
   1116 if shortcut:
   1117     combined = self._concat_shortcut(applied, dim, positions)

File ~/Work/miniconda3/envs/20211123_py39/lib/python3.9/site-packages/xarray/core/groupby.py:559, in GroupBy._infer_concat_args(self, applied_example)
    558 def _infer_concat_args(self, applied_example):
--> 559     if self._group_dim in applied_example.dims:
    560         coord = self._group
    561         positions = self._group_indices

AttributeError: 'numpy.float64' object has no attribute 'dims'

Anything else we need to know?

No response

Environment

INSTALLED VERSIONS

commit: None
python: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:56:21)
[GCC 10.3.0]
python-bits: 64
OS: Linux
OS-release: 4.4.0-210-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: C
LOCALE: ('en_GB', 'UTF-8')
libhdf5: 1.12.2
libnetcdf: 4.8.1

xarray: 2022.6.0
pandas: 1.4.3
numpy: 1.23.2
scipy: 1.9.0
netCDF4: 1.6.0
pydap: installed
h5netcdf: None
h5py: 3.7.0
Nio: None
zarr: 2.12.0
cftime: 1.6.1
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2022.6.1
distributed: 2022.6.1
matplotlib: 3.5.3
cartopy: 0.20.3
seaborn: None
numbagg: None
fsspec: 2022.7.1
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 65.2.0
pip: 22.2.2
conda: 4.14.0
pytest: 7.1.2
IPython: 8.4.0
sphinx: 5.1.1

[mathause]

apply seems to expect that the applied function returns a DataArray. As an alternative could you use data.resample(index="M").reduce(np.median)?

[dcherian]

1114 applied_example, applied = peek_at(applied)

We could raise an error here if applied_example is not a DataArray or Dataset but not sure if its worth it. This is a somewhat common user question

[st-bender]

@mathause Thanks, that works for np.median, but .reduce() passes an axis= keyword argument, which needs to be taken care of by any custom function. I used np.median here just as an example, as it showed the problem.

@dcherian I am not sure if raising an error would be very user friendly. In my opinion, which may be biased by my personal use cases, I would expect .apply() (or .map() for that matter) of a resample object to take any function that returns a scalar: the value at the resampled point. It should be then up to xarray to iterate over any other non-resampled dimensions. I am not sure why it requires to return a DataArray in the first place. As mentioned, it works with pandas series and dataframes, e.g.:

data.to_pandas().resample("M").apply(np.median)

[mathause]

It does work if the array keeps the size:

data.resample(index="M").apply(lambda x: x.values)

As a workaround you could allow your function to consume a dummy axis. Or you could pass dim as ...

data.resample(index="M").reduce(lambda x, axis: 1) # workaround 1
data.resample(index="M").reduce(lambda x: 1, dim=...) # workaround 2

(reduce only passes axis if dim is not None but groupby passes the group_dim per default.

[st-bender]

Hi @mathause

It does work if the array keeps the size:

data.resample(index="M").apply(lambda x: x.values)

Thanks, but I am not sure I find that intuitive, why should the resampled array have the same size as the original? It seems to make only sense for DataArray.apply(), but not for a resampled one. As I indicated in my other reply, returning a scalar or equivalent should be fine, shouldn't it?
At the very least the documentation is lacking, it refers to the pandas method, but clearly the behaviour is different.

As a workaround you could allow your function to consume a dummy axis. Or you could pass dim as ...

data.resample(index="M").reduce(lambda x, axis: 1) # workaround 1
data.resample(index="M").reduce(lambda x: 1, dim=...) # workaround 2

(reduce only passes axis if dim is not None but groupby passes the group_dim per default.

That feels a bit like curing the symptoms instead of the root cause, why not set dim='...' if it is not given?