69

I can't seem to get a simple dtype check working with Pandas' improved Categoricals in v0.15+. Basically I just want something like is_categorical(column) -> True/False.

import pandas as pd
import numpy as np
import random

df = pd.DataFrame({
    'x': np.linspace(0, 50, 6),
    'y': np.linspace(0, 20, 6),
    'cat_column': random.sample('abcdef', 6)
})
df['cat_column'] = pd.Categorical(df2['cat_column'])

We can see that the dtype for the categorical column is 'category':

df.cat_column.dtype
Out[20]: category

And normally we can do a dtype check by just comparing to the name of the dtype:

df.x.dtype == 'float64'
Out[21]: True

But this doesn't seem to work when trying to check if the x column is categorical:

df.x.dtype == 'category'
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-22-94d2608815c4> in <module>()
----> 1 df.x.dtype == 'category'

TypeError: data type "category" not understood

Is there any way to do these types of checks in pandas v0.15+?

4
  • 7
    so aside from the below solns, the canoncial way to select columns >= 0.15.0 is df.select_dtypes(include=['category'])
    – Jeff
    Nov 14, 2014 at 13:37
  • 3
    This probably has to do with the fact that category is a data type added by pandas, compared to other data types that comes from numpy. Feb 4, 2018 at 23:49
  • 1
    @AntoineGallix Yes, the problem is that numpy.dtype is checking if the datatype name "category" is a recognized category name (like "float64"). Since its not recognized in numpy (no categorical datatype in numpy), numpy assumes you made a typo, rather than telling you its definitely not the datatype you're looking for. Pandas on the other hand has chosen the other approach, typos result in plain-old False.
    – JoseOrtiz3
    Apr 30, 2019 at 6:34
  • i notice that df.x.dtype == 'category' works in pandas 1.3.4 but not in pandas 1.0.3
    – Joris
    Dec 28, 2021 at 14:24

7 Answers 7

90

Use the name property to do the comparison instead, it should always work because it's just a string:

>>> import numpy as np
>>> arr = np.array([1, 2, 3, 4])
>>> arr.dtype.name
'int64'

>>> import pandas as pd
>>> cat = pd.Categorical(['a', 'b', 'c'])
>>> cat.dtype.name
'category'

So, to sum up, you can end up with a simple, straightforward function:

def is_categorical(array_like):
    return array_like.dtype.name == 'category'
31

First, the string representation of the dtype is 'category' and not 'categorical', so this works:

In [41]: df.cat_column.dtype == 'category'
Out[41]: True

But indeed, as you noticed, this comparison gives a TypeError for other dtypes, so you would have to wrap it with a try .. except .. block.


Other ways to check using pandas internals:

In [42]: isinstance(df.cat_column.dtype, pd.api.types.CategoricalDtype)
Out[42]: True

In [43]: pd.api.types.is_categorical_dtype(df.cat_column)
Out[43]: True

For non-categorical columns, those statements will return False instead of raising an error. For example:

In [44]: pd.api.types.is_categorical_dtype(df.x)
Out[44]: False

For much older version of pandas, replace pd.api.types in the above snippet with pd.core.common.

2
  • Which columns would it give an error for? Nov 10, 2021 at 17:01
  • With recent versions of numpy this no longer errors, but previously something like np.dtype("int64" == "category" raised an error instead of returning False.
    – joris
    Nov 13, 2021 at 8:33
6

Just putting this here because pandas.DataFrame.select_dtypes() is what I was actually looking for:

df['column'].name in df.select_dtypes(include='category').columns

Thanks to @Jeff.

4

In my pandas version (v1.0.3), a shorter version of joris' answer is available.

df = pd.DataFrame({'noncat': [1, 2, 3], 'categ': pd.Categorical(['A', 'B', 'C'])})

print(isinstance(df.noncat.dtype, pd.CategoricalDtype))  # False
print(isinstance(df.categ.dtype, pd.CategoricalDtype))   # True

print(pd.CategoricalDtype.is_dtype(df.noncat)) # False
print(pd.CategoricalDtype.is_dtype(df.categ))  # True
1
  • 1
    I get unexpected results with my data `` isinstance(Tmanual['X'], pd.CategoricalDtype) Out[216]: False Tmanual['REVENUES_FAST'].dtype.name == 'category' Out[217]: True Tmanual['X'].dtype Out[218]: CategoricalDtype(categories=['ANY', 'ANYIMPORTANT', 'BX', 'OPTIONAL'], ordered=False) `` Jul 7, 2020 at 8:45
3

I ran into this thread looking for the exact same functionality, and also found out another option, right from the pandas documentation here.

It looks like the canonical way to check if a pandas dataframe column is a categorical Series should be the following:

hasattr(column_to_check, 'cat')

So, as per the example given in the initial question, this would be:

hasattr(df.x, 'cat') #True
1

Nowadays you can use:

pandas.api.types.is_categorical_dtype(series)

Docs here: https://pandas.pydata.org/docs/reference/api/pandas.api.types.is_categorical_dtype.html

Available since at least pandas 1.0

0

Taking a look at @Jeff Tratner answer, since the condition df.cat_column.dtype == 'category' not needs to be True to be considered a column as cataegorical, I propose this considering categorical the dtypes within 'categorical_dtypes' list:

def is_cat(column):
    categorical_dtypes = ['object', 'category', 'bool']
    if column.dtype.name in categorical_dtypes:
        return True
    else:
        return False   

´´´

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.