[Python] DataFrame interchange protocol: NaNs are interchanged as null

[honno]

Describe the bug, including details regarding any error messages, version, and platform.

First of, this might not be a problem—let me know if that's the case! Hard for me to wrap my head around the consequences of Arrow's NA model with interchanging 😅

So, if one were to interchange a pandas.DataFrame containing NaNs to a pyarrow.Table, one gets nulls in place of NaNs.

>>> import pandas as pd
>>> import numpy as np
>>> df = pd.DataFrame({"foo": pd.Series([float("nan")], dtype=np.float64)})
>>> from pyarrow.interchange import from_dataframe
>>> from_dataframe(df)
pyarrow.Table
foo: double
----
foo: [[null]    # expect NaN?

We get the same with modin, which also adopts the interchange protocol.

>>> import modin
>>> import ray
>>> ray.init(local_mode=True)
>>> from modin.config import Engine
>>> Engine.put("ray")
>>> from modin import pandas as mpd
>>> df = mpd.DataFrame({"foo": mpd.Series([float("nan")], dtype=np.float64)})
>>> from_dataframe(df)
pyarrow.Table
foo: double
----
foo: [[null]]

I see interchanging another pa.Table with NaNs works fine, asummedly because from_dataframe() short-circuits when it gets a pa.Table/pa.RecordBatch.

>>> table = pa.Table.from_pydict({"foo": pa.array([float("nan")], type=pa.float64())})
>>> from_dataframe(table)
pyarrow.Table
foo: double
----
foo: [[nan]]

Using the arrow nightly from https://pypi.fury.io/arrow-nightlies/

cc @AlenkaF

Component(s)

Python

[jorisvandenbossche]

So, if one were to interchange a pandas.DataFrame containing NaNs to a pyarrow.Table, one gets nulls in place of NaNs.

AFAIK this is expected, because pandas creates an interchange object explicitly indicating that the NaN is used as the null value:

In [15]: df.__dataframe__().get_column_by_name("foo").describe_null
Out[15]: (<ColumnNullType.USE_NAN: 1>, None)

And thus pyarrow correctly converts this to null on the Arrow side.

I see interchanging another pa.Table with NaNs works fine, asummedly because from_dataframe() short-circuits when it gets a pa.Table/pa.RecordBatch.

We are indeed short-cutting here. But even when not doing that (by explicitly using the lower level function that doesn't have this short-circuit), we preserve NaNs:

In [22]: from pyarrow.interchange.from_dataframe import _from_dataframe

In [23]: _from_dataframe(table.__dataframe__())
Out[23]: 
pyarrow.Table
foo: double
----
foo: [[nan]]

That's because pyarrow does not set that the NaN is the missing value indicator, and thus NaNs in the input are preserved.