Cannot roundtrip time in NETCDF4_CLASSIC

[mathause]

MCVE Code Sample

import numpy as np
import xarray as xr
time = xr.cftime_range("2006-01-01", periods=2, calendar="360_day")

da = xr.DataArray(time, dims=["time"])
da.encoding["dtype"] = np.float
da.to_netcdf("tst.nc", format="NETCDF4_CLASSIC")

ds = xr.open_dataset("tst.nc")
ds.to_netcdf("tst2.nc", format="NETCDF4_CLASSIC")

yields:

ValueError: could not safely cast array from dtype int64 to int32

Or an example without to_netcdf:

import numpy as np
import xarray as xr

time = xr.cftime_range("2006-01-01", periods=2, calendar="360_day")

da = xr.DataArray(time, dims=["time"])
da.encoding["_FillValue"] = np.array([np.nan])

xr.backends.netcdf3.encode_nc3_variable(xr.conventions.encode_cf_variable(da))

Expected Output

Xarray can save the dataset/ an xr.Variable.

Problem Description

If there is a time variable that can be encoded using integers only, but that has a _FillValue set to NaN, saving to_netcdf(name, format="NETCDF4_CLASSIC") fails. The problem is that xarray adds a (unnecessary) _FillValue when saving a file.

Note: if the time cannot be encoded using integers only, it works:

da = xr.DataArray(time, dims=["time"])
da.encoding["_FillValue"] = np.array([np.nan])
da.encoding["units"] = "days since 2006-01-01T12:00:00"

xr.backends.netcdf3.encode_nc3_variable(xr.conventions.encode_cf_variable(da))

Another note: when saving with NETCDF4

da = xr.DataArray(time, dims=["time"])
da.encoding["_FillValue"] = np.array([np.nan])

xr.backends.netCDF4_._encode_nc4_variable(xr.conventions.encode_cf_variable(da))

The following is returned:

<xarray.Variable (time: 2)>
array([0, 1])
Attributes:
    units:       days since 2006-01-01 00:00:00.000000
    calendar:    proleptic_gregorian
    _FillValue:  [-9223372036854775808]

Output of xr.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.7.3 | packaged by conda-forge | (default, Jul 1 2019, 21:52:21) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 4.12.14-lp151.28.36-default machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.5 libnetcdf: 4.7.1

xarray: 0.14.1
pandas: 0.25.2
numpy: 1.17.3
scipy: 1.3.1
netCDF4: 1.5.3
pydap: None
h5netcdf: 0.7.4
h5py: 2.10.0
Nio: None
zarr: None
cftime: 1.0.4.2
nc_time_axis: 1.2.0
PseudoNetCDF: None
rasterio: 1.1.1
cfgrib: None
iris: None
bottleneck: 1.3.1
dask: 2.6.0
distributed: 2.6.0
matplotlib: 3.1.2
cartopy: 0.17.0
seaborn: 0.9.0
numbagg: None
setuptools: 41.4.0
pip: 19.3.1
conda: None
pytest: 5.2.2
IPython: 7.9.0
sphinx: 2.2.1

[dcherian]

The problem is that xarray adds a (unnecessary) _FillValue when saving a file.

yes. I think we should fix this. It breaks roundtripping

[mathause]

This happens in:

xarray/xarray/conventions.py

Line 106 in db36c5c

def maybe_default_fill_value(var):

My current workaround is:

if "time" in ds:
    ds.time.encoding.pop("_FillValue", None)

[mathause]

todo: Check if #3652 helps

[mathause]

#3652 helps indeed. Example 1 from above now works without error. Example 2 still throws an error. However, I think this is fine, because you are unlikely to end up with an encoding that defines only a _FillValue and not a dtype.

I close this for now unless you disagree?