BUG: behaviour of fillna(NaN) for nullable dtype (or, how to fill nullable array with NaN)

[cynddl]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

s1 = pd.Series([pd.NA, -1], dtype='Int64')
s2 = pd.Series([pd.NA, -1])  # the dtype is now object

Problem description

s1.fillna(np.nan) and s2.fillna(np.nan) return different series. Specifically, the first one keeps the pd.NA cell untouched and does not replace it with np.nan. Replacing with any other value that np.nan does not trigger this bug.

Expected Output

> s2.fillna(np.nan)
0    NaN
1   -1.0
dtype: float64

Output of pd.show_versions()

INSTALLED VERSIONS

commit : f00ed8f
python : 3.8.5.final.0
python-bits : 64
OS : Linux
OS-release : 4.4.0-210-generic
Version : #242-Ubuntu SMP Fri Apr 16 09:57:56 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : en_US.UTF-8
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.3.0
numpy : 1.19.1
pytz : 2019.3
dateutil : 2.8.1
pip : 20.2.2
setuptools : 50.3.0
Cython : 0.29.17
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.18.1
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.2
numexpr : 2.7.1
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.5.2
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[mzeitlin11]

Thanks for reporting this @cynddl! While certainly confusing, I believe this is intended behavior. For object type, pd.NA and np.nan are both valid missing values and treated as distinct.

For nullable types like Int64, there is a notion of a single missing value, pd.NA. Generally (though there are probably inconsistencies here), using other missing values than pd.NA with these types will still treat that value as pd.NA. For example using a missing value in construction,

In [3]: pd.Series([np.nan, -1], dtype="Int64")
Out[3]:
0    <NA>
1      -1
dtype: Int64

If you want pandas to treat pd.NA and np.nan differently, your best bet would be object type data (or perhaps nullable float data depending on how #32265 goes)

[simonjayhawkins]

If you want pandas to treat pd.NA and np.nan differently, your best bet would be object type data (or perhaps nullable float data depending on how #32265 goes)

adding no action label for now. needs to considered as part of discussion in #32265. cc @jorisvandenbossche

[simonjayhawkins]

if the outcome of the discussion is that fillna() on a FloatingArray should be changed...

>>> s = pd.Series([pd.NA, -1], dtype='Float64')
>>> s
0    <NA>
1    -1.0
dtype: Float64
>>> 
>>> s.fillna(np.nan)
0    <NA>
1    -1.0
dtype: Float64

then for the IntegerArray, which does not support floats, then the expected behavior would need to be revisted, but for now the behaviour of IntegerArray and FloatingArray are consistent.