Series dtype changes when a new row is added

[asdf8601]

The problem

If we create a Series with a defined dtype and then a new row is added into that Series the dtype changes. I have left an example below:

• https://pandas.pydata.org/pandas-docs/stable/missing_data.html#inserting-missing-data
• https://pandas.pydata.org/pandas-docs/stable/indexing.html#setting-with-enlargement

Example

import pandas as pd
import numpy as np

pd.__version__  # '0.23.1'

# with Series
s = pd.Series([1,2], dtype=np.float64)
print(s.dtype)  # -> float64
s[3] = None
print(s.dtype)  # -> object

# with DataFrames
d = pd.DataFrame([1,2], dtype=np.float64)
print(d.dtypes)  # -> float64
d.loc[3, 0] = None
print(d.dtypes)  # -> float64

However, this doesn't happen when the row is already present:

In[12]: s = pd.Series([1,2]).astype(np.float64)
In[13]: s[3] = None
In[14]: s
Out[14]: 
0       1
1       2
3    None
dtype: object

In[15]: s = s.astype(np.float64)
In[16]: s
Out[16]: 
0    1.0
1    2.0
3    NaN
dtype: float64

# row 3 (position 2) is already present in s
In[18]: s.iloc[2] = None
In[19]: s
Out[19]: 
0    1.0
1    2.0
3    NaN
dtype: float64

In[20]: s.loc[3] = None
In[21]: s
Out[21]: 
0    1.0
1    2.0
3    NaN
dtype: float64

[WillAyd]

None is not a valid float64 value, hence the coercion to object. If you wanted to preserve that dtype you should be inserting np.nan

[WillAyd]

Sorry misunderstood this on first glance. I see now that the append of None coerces to object but assigning to an existing value preserves type, automatically converting None to np.nan.

Investigation and PRs into the behavior are certainly welcome

[asdf8601]

@WillAyd Don't worry, my fault, I should have gone into more detail.

What would be the expected behavior?

My opinion:

On the one hand, DataFrame and Series behave differently when they perform the same operation, I refer to the first example above: If I insert None in a new field in the DataFrame the dtype is maintained while in the Series it does not.

On the other hand, if the field already exists, both (Series and DataFrame) behave the same, turning None into NaN preserving dtype=float64.

From your first answer I understand that the first is what is expected and the second is not, that is, if a field exists and the content is changed to None, the dtype should change to object?

[WillAyd]

I think any insert or assignment of None should coerce to object. If the user wanted to maintain float64 then they should have used np.nan instead, so that its not up to pandas to make that inference.

cc @TomAugspurger @jreback if they disagree

[jreback]

there is an issue about this already iirc

[asdf8601]

Probably this issue: #20442 (comment)
I found it relevant.

[rhshadrach]

All of these now return float64 on main. There has been recent work in this regard (e.g. PDEP-6), I don't believe we need a test. Closing.