Purpose
I have a large DataFrame with varied dtypes where I have to perform a global .replace
to turn both NaN, NaT and empty strings into None
. The DataFrame looks like
import pandas as pd
from datetime import datetime
df = pd.DataFrame({
'a': [n*10.0 for n in range(5)],
'b': [datetime.now() if n%3 else None for n in range(5)],
'c': pd.Series([f'D{n}' if n%2 else '' for n in range(5)], dtype='category'),
'd': ['Long text chunk...' if n%3 else None for n in range(5)]
})
Which prints
a b c d
0 0.0 NaT None
1 10.0 2020-08-13 23:35:55.533189 D1 Long text chunk...
2 20.0 2020-08-13 23:35:55.533189 Long text chunk...
3 30.0 NaT D3 None
4 40.0 2020-08-13 23:35:55.533189 Long text chunk...
My purpose is to bulk upload the rows into ElasticSearch, which won't accept NaN - neither NaT nor empty strings for date fields - without some setting changes I'm trying to avoid. I figured this way would be faster than individually checking every row when making the dicts.
Approach
Converting all columns to object
before replacing wasn't even runnable due to the DataFrame size - I'd prefer not to convert any column at all. An approach that once worked was
df.fillna('').replace('', None)
But now, adding some category dtypes in, it raises TypeError: No matching signature found
.
Question
Searching this, nothing I found was related to pandas
at all. It's clearly linked to the category dtype¹, but what I don't know:
What's the most pythonic way of doing this while keeping integrity for all columns, especially the categorical ones?
What happens behind the curtains for pandas to raise this apparently generic error in a.replace
?
¹ Edit:
I later found that the pandas implementation replace in this case reaches up to a Cython-compiled method - pandas._libs.algos.pad_inplace
- which expects to fill any Series dtype except category
. That's why my error mentions a signature mismatch. I still wonder if this is intended behavior, as I'd expect an ffill to work especially well in categorical columns.
Since my numeric columns were filled already, I changed column a
here to reflect that. So my hassle is solely the category
dtype.
None
is an object; a Series has to be object dtype to contain such an object.float64
series gets converted. Actually it's thefillna
that converts all columns - except the category one. Your replace works if the value is a valid 'category', e.g.df.fillna('').replace('','D1')
. I don't know anything abutelasticsearch
. But looking at your sampledf
, I see that the 'c' column is a unique pandas structure, which probably isn't usable outside ofpandas
.complex128
.