segfault with a particular netcdf4 file

[hmaarrfk]

What happened?

The following code yields a segfault on my machine (and many other machines with a similar environment)

import xarray
filename = 'tiny.nc.txt'
engine = "netcdf4"

dataset = xarray.open_dataset(filename, engine=engine)

i = 0
for i in range(60):
    xarray.open_dataset(filename, engine=engine)

tiny.nc.txt
mrc.nc.txt

What did you expect to happen?

Not to segfault.

Minimal Complete Verifiable Example

1. Generate some netcdf4 with my application.
2. Trim the netcdf4 file down (load it, and drop all the vars I can while still reproducing this bug)
3. Try to read it.

import xarray
from tqdm import tqdm
filename = 'mrc.nc.txt'
engine = "h5netcdf"
dataset = xarray.open_dataset(filename, engine=engine)

for i in tqdm(range(60), desc=f"filename={filename}, enine={engine}"):
    xarray.open_dataset(filename, engine=engine)


engine = "netcdf4"

dataset = xarray.open_dataset(filename, engine=engine)
for i in tqdm(range(60), desc=f"filename={filename}, enine={engine}"):
    xarray.open_dataset(filename, engine=engine)

filename = 'tiny.nc.txt'

engine = "h5netcdf"
dataset = xarray.open_dataset(filename, engine=engine)
for i in tqdm(range(60), desc=f"filename={filename}, enine={engine}"):
    xarray.open_dataset(filename, engine=engine)


engine = "netcdf4"

dataset = xarray.open_dataset(filename, engine=engine)
for i in tqdm(range(60), desc=f"filename={filename}, enine={engine}"):
    xarray.open_dataset(filename, engine=engine)

hand crafting the file from start to finish seems to not segfault:

import xarray
import numpy as np
engine = 'netcdf4'

dataset = xarray.Dataset()

coords = {}
coords['image_x'] = np.arange(1, dtype='int')
dataset = dataset.assign_coords(coords)

dataset['image'] = xarray.DataArray(
    np.zeros((1,), dtype='uint8'),
    dims=('image_x',)
)

# %%
dataset.to_netcdf('mrc.nc.txt')
# %%
dataset = xarray.open_dataset('mrc.nc.txt', engine=engine)


for i in range(10):
    xarray.open_dataset('mrc.nc.txt', engine=engine)

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.
• Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

i=0 passes
i=1 mostly segfaults, but sometimes it can take more than 1 iteration

Anything else we need to know?

At first I thought it was deep in hdf5, but I am less convinced now

xref: HDFGroup/hdf5#3649

Environment

INSTALLED VERSIONS
------------------
commit: None
python: 3.10.12 | packaged by Ramona Optics | (main, Jun 27 2023, 02:59:09) [GCC 12.3.0]
python-bits: 64
OS: Linux
OS-release: 6.5.1-060501-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.14.2
libnetcdf: 4.9.2

xarray: 2023.9.1.dev25+g46643bb1.d20231009
pandas: 2.1.1
numpy: 1.24.4
scipy: 1.11.3
netCDF4: 1.6.4
pydap: None
h5netcdf: 1.2.0
h5py: 3.9.0
Nio: None
zarr: 2.16.1
cftime: 1.6.2
nc_time_axis: None
PseudoNetCDF: None
iris: None
bottleneck: None
dask: 2023.3.0
distributed: 2023.3.0
matplotlib: 3.8.0
cartopy: None
seaborn: None
numbagg: None
fsspec: 2023.9.2
cupy: None
pint: 0.22
sparse: None
flox: None
numpy_groupies: None
setuptools: 68.2.2
pip: 23.2.1
conda: 23.7.4
pytest: 7.4.2
mypy: None
IPython: 8.16.1
sphinx: 7.2.6

[hmaarrfk]

Ok this was bugging me enough, but here is a reproducer that just "runs" without any data:

import xarray
engine = 'netcdf4'

dataset = xarray.Dataset()

dataset.coords['x'] = ['a']
dataset.to_netcdf('mrc.nc')
dataset = xarray.open_dataset('mrc.nc', engine=engine)

for i in range(10):
    print(f"i={i}")
    xarray.open_dataset('mrc.nc', engine=engine)

the key was making the coordinate a H5T_STRING type.

string_mrc.nc.txt

[hmaarrfk]

Sorry to rapid fire post, but the following "hack" seems to resolve the issues I am observing:

diff --git a/xarray/backends/netCDF4_.py b/xarray/backends/netCDF4_.py
index f21f15bf..8f1243da 100644
--- a/xarray/backends/netCDF4_.py
+++ b/xarray/backends/netCDF4_.py
@@ -394,8 +394,8 @@ class NetCDF4DataStore(WritableCFDataStore):
         kwargs = dict(
             clobber=clobber, diskless=diskless, persist=persist, format=format
         )
-        manager = CachingFileManager(
-            netCDF4.Dataset, filename, mode=mode, kwargs=kwargs
+        manager = DummyFileManager(
+            netCDF4.Dataset(filename, mode=mode, **kwargs)
         )
         return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)

I have a feeling some reference isn't being kept, and the file is being free'ed somehow during garbage collection.

While this "hack" somewhat "works", if I try to open the same file with two different backends, it really likes to complain.

It may be that libnetcdf4 just expects to be in control of the file at a all times..

[hmaarrfk]

Running a similar segfaulting benchmark on xarray's main branch

import xarray
import numpy as np

write_engine = 'h5netcdf'
hold_engine = 'h5netcdf'
read_engine = 'netcdf4'

filename = f'{write_engine}_mrc.nc'

# %%
dataset = xarray.Dataset()

dataset.coords['x'] = ['a']
dataset.coords['my_version'] = '1.2.3.4.5.6'
dataset['images'] = (('x', ), np.zeros((1,)))

dataset.to_netcdf(filename, engine=write_engine)
# %%
dataset = xarray.open_dataset(filename, engine=hold_engine)
for i in range(100):
    print(f"i={i}")
    xarray.open_dataset(filename, engine=read_engine)

	write engine
hold/read engine	h5netcdf	netcdf4
netcdf4/netcdf4	pass	segfault
netcdf4/h5netcdf	pass	segfault
h5netcdf/h5netcdf	pass	pass
h5netcdf/netcdf4	pass	pass

[hmaarrfk]

While I know these issues are hard, can anybody else confirm that this happens on their system as well? Maybe my machine is really weird....

As a final reproducer:

import xarray
xarray.set_options(warn_for_unclosed_files=True)
# Also needs a small patch....
"""
diff --git a/xarray/backends/file_manager.py b/xarray/backends/file_manager.py
index df901f9a..a2e8af03 100644
--- a/xarray/backends/file_manager.py
+++ b/xarray/backends/file_manager.py
@@ -252,11 +252,10 @@ class CachingFileManager(FileManager):
                     self._lock.release()
 
             if OPTIONS["warn_for_unclosed_files"]:
-                warnings.warn(
+                print(
                     f"deallocating {self}, but file is not already closed. "
                     "This may indicate a bug.",
-                    RuntimeWarning,
-                    stacklevel=2,
+                    flush=True
                 )
 
     def __getstate__(self):

"""

dataset = xarray.Dataset()

dataset.coords['x'] = ['a']
dataset.to_netcdf('mrc.nc', engine='netcdf4')

dataset = xarray.open_dataset('mrc.nc', engine='netcdf4')
for i in range(100):
    print(f"i={i}")
    xarray.open_dataset('mrc.nc', engine='netcdf4')

Gives the output:

i=0
deallocating CachingFileManager(<class 'netCDF4._netCDF4.Dataset'>, '/home/mark/git/wgpu/mrc.nc', mode='r', kwargs={'clobber': True, 'diskless': False, 'persist': False, 'format': 'NETCDF4'}, manager_id='120349e5-9287-4535-a724-588aa78cf9d0'), but file is not already closed. This may indicate a bug.

[jhamman]

I can confirm that your original reproducer segfaults on my system (Linux/x86_64). I also agree with your diagnosis that this seems to be an issue with the caching file manager.

FWIW, adding dataset.close() after the first open_dataset does seem to solve things. That said, I don't think this should be required so, for some reason, we're not using the cached file object.

[hmaarrfk]

thanks for confirming. it has been puzzling me for no end.

[heikoklein]

I'm also struggling with the problem and I have simplified the code a bit more:

import xarray
import numpy as np
import os
import sys

filename = 'test_mrc.nc'

if not os.path.exists(filename):
    dataset_w = xarray.Dataset()
    dataset_w['x'] = ['a']
    dataset_w.to_netcdf(filename)

print("try open 1", file=sys.stderr)
dataset = xarray.open_dataset(filename)
print("try open 2", file=sys.stderr)
dataset2 = xarray.open_dataset(filename)
dataset2 = None
print("try open 3", file=sys.stderr)
dataset3 = xarray.open_dataset(filename)
print("success")

The problem only occurs if certain features of netcdf4 are used on file (e.g. superblock 2, strings), but those are common.
The cache-manager fails to handle the file if it was opened twice and one of these two handles go out of scope (here dataset2). The next open (dataset3) throws a segmentation-fault.

I've tested with v2023.6.0 (latest version in conda-forge / python 3.11.7, linux)

[thorbjoernl]

The above example succeeds in my case (after removing the extra .):

(venv) $ pip freeze | grep xarray
xarray==2024.3.0
(venv) $ python3 --version
Python 3.11.3
(venv) $ python3 test.py 
try open 1
try open 2
try open 3
success

No conda; just a regular python virtual environment

[heikoklein]

I just managed to upgrade my xarray to 2024.03.0 (pinning the version) and still get the error, though it works sometimes?

$ conda install xarray=2024.3.0
...
$ conda list | grep xarray
xarray                    2024.3.0           pyhd8ed1ab_0    conda-forge
$ python3 xarray_segfault.py 
try open 1
try open 2
try open 3
success
$ python3 xarray_segfault.py 
try open 1
try open 2
try open 3
Segmentation fault (core dumped)

(this was independ if the test_mrc.nc file existed or not...)