Depending upon how compute() is called, python seg faults

[mayanksuman]

I have two large dask arrays (hist_data_ua850 and hist_data_hus850). If I want to calculate the product then (hist_data_ua850 * hist_data_hus850).compute() fails with either seg fault or bus error. However, (hist_data_ua850.compute() * hist_data_hus850.compute()) works. (sample code attached
)

The gdb produce following error message for seg fault.

Thread 25 "python3" received signal SIGBUS, Bus error.
[Switching to Thread 0x7fffbf5e3700 (LWP 31903)]
0x00007fffec92a8de in H5FL_fac_malloc ()
   from /home/user/.local/lib/python3.6/site-packages/netCDF4/.libs/libhdf5-d5cdc8a0.so.101.1.0                          

or

Thread 25 "python3" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffbf5e3700 (LWP 11553)]
0x00007fffeca3cb57 in ?? () from /home/user/.local/lib/python3.6/site-packages/netCDF4/.libs/libhdf5-d5cdc8a0.so.101.1.0

During these seg fault <10% of RAM is being used.

Output of xr.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.6.8.candidate.1 python-bits: 64 OS: Linux OS-release: 4.18.0-3-amd64 machine: x86_64 processor: byteorder: little LC_ALL: en_IN.UTF-8 LANG: en_IN.UTF-8 LOCALE: en_IN.UTF-8

xarray: 0.11.0
pandas: 0.23.4
numpy: 1.15.4
scipy: 1.1.0
netCDF4: 1.4.2
h5netcdf: None
h5py: 2.8.0
Nio: None
zarr: None
cftime: 1.0.3.4
PseudonetCDF: None
rasterio: 1.0.2
iris: None
bottleneck: 1.2.1
cyordereddict: None
dask: 1.0.0
distributed: 1.20.2
matplotlib: 3.0.2
cartopy: 0.17.0
seaborn: 0.9.0
setuptools: 40.2.0
pip: 18.0
conda: None
pytest: 3.4.0
IPython: 7.2.0
sphinx: 1.8.2

[mayanksuman]

The attached code works fine with xarray==0.10.9. The problem seems to exist with xarray==0.11.0 only.

[shoyer]

Can you share how you produced these arrays? In particular, how did you load them from netCDF files?

#2595 may also resolve this issue -- it would be nice if you could give it a try and report back.

[shoyer]

note: I don't think your sample code came through -- there aren't any attachments on this post.

[mayanksuman]

Can you share how you produced these arrays? In particular, how did you load them from netCDF files?

#2595 may also resolve this issue -- it would be nice if you could give it a try and report back.

The xarray dataset was loaded using open_mfdataset(). Further, the behavior is erratic, only happen when you have loaded 10+ xarray dataset using open_mfdataset(). These xarray datasets are not opening same netcdf files.

Sample Code:

import xarray as xr

hist_hus850 = xr.open_mfdataset('data/hus850/*_historical_*.nc')
rcp45_hus850 = xr.open_mfdataset('data/hus850/*_rcp45_*.nc')
rcp85_hus850 = xr.open_mfdataset('data/hus850/*_rcp85_*.nc')

hist_va850 = xr.open_mfdataset('data/va850/*_historical_*.nc')
rcp45_va850 = xr.open_mfdataset('data/va850/*_rcp45_*.nc')
rcp85_va850 = xr.open_mfdataset('data/va850/*_rcp85_*.nc')

hist_ua850 = xr.open_mfdataset('data/ua850/*_historical_*.nc')
rcp45_ua850 = xr.open_mfdataset('data/ua850/*_rcp45_*.nc')
rcp85_ua850 = xr.open_mfdataset('data/ua850/*_rcp85_*.nc')

hist_pr = xr.open_mfdataset('data/pr/*_historical_*.nc')
rcp45_pr = xr.open_mfdataset('data/pr/*_rcp45_*.nc')
rcp85_pr = xr.open_mfdataset('data/pr/*_rcp85_*.nc')

(hist_hus850.compute()*hist_ua850.compute())      # works in both xarray==0.10.9 and  xarray==0.11.0
(hist_hus850*hist_ua850).compute()                # do not works in xarray==0.11.0 but works in earlier version

Python3 Trackback

Traceback (most recent call last):
  File "/media/data/mayank/test_xarray/venv/lib/python3.6/site-packages/xarray/backends/file_manager.py", line 137, in acquire
    file = self._cache[self._key]
  File "/media/data/mayank/test_xarray/venv/lib/python3.6/site-packages/xarray/backends/lru_cache.py", line 43, in __getitem__
    value = self._cache[key]
KeyError: [<function _open_netcdf4_group at 0x7ffff1a81840>, ('/media/data/mayank/test_xarray/data/hus850/hus850_mean_historical_19910101-19951231.nc', CombinedLock([<SerializableLock: 8187e0f4-5e90-46df-9201-4b501cc1548b>, <SerializableLock: a81c2788-14be-46ad-bd4b-227be793260d>])), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('group', None), ('persist', False))]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test_seg_fault.py", line 19, in <module>
    (hist_hus850.compute()*hist_ua850.compute())      # works
  File "/media/data/mayank/test_xarray/venv/lib/python3.6/site-packages/xarray/core/dataset.py", line 597, in compute
    return new.load(**kwargs)
  File "/media/data/mayank/test_xarray/venv/lib/python3.6/site-packages/xarray/core/dataset.py", line 494, in load
    evaluated_data = da.compute(*lazy_data.values(), **kwargs)
  File "/media/data/mayank/test_xarray/venv/lib/python3.6/site-packages/dask/base.py", line 397, in compute

Thread 25 "python3" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffbd5b9700 (LWP 32326)]
0x00007fffec9008de in H5FL_fac_malloc ()
   from /media/data/mayank/test_xarray/venv/lib/python3.6/site-packages/netCDF4/.libs/libhdf5-d5cdc8a0.so.101.1.0

[mayanksuman]

The issue is resolved with #2595. Thank You @shoyer.

[shoyer]

Thanks for testing this @mayanksuman! I'm going to leave this open for now until #2595 is merged. It's always nice to know when fixing a theoretical race condition fixes a real bug :).