167

Why does Pandas tell me that I have objects, although every item in the selected column is a string — even after explicit conversion.

This is my DataFrame:

<class 'pandas.core.frame.DataFrame'>
Int64Index: 56992 entries, 0 to 56991
Data columns (total 7 columns):
id            56992  non-null values
attr1         56992  non-null values
attr2         56992  non-null values
attr3         56992  non-null values
attr4         56992  non-null values
attr5         56992  non-null values
attr6         56992  non-null values
dtypes: int64(2), object(5)

Five of them are dtype object. I explicitly convert those objects to strings:

for c in df.columns:
    if df[c].dtype == object:
        print "convert ", df[c].name, " to string"
        df[c] = df[c].astype(str)

Then, df["attr2"] still has dtype object, although type(df["attr2"].ix[0] reveals str, which is correct.

Pandas distinguishes between int64 and float64 and object. What is the logic behind it when there is no dtype str? Why is a str covered by object?

1
  • Came here because joins are failing due to 'object type' even though every "is" string Commented Jan 28, 2020 at 22:51

4 Answers 4

204

The dtype object comes from NumPy, it describes the type of element in a ndarray. Every element in an ndarray must have the same size in bytes. For int64 and float64, they are 8 bytes. But for strings, the length of the string is not fixed. So instead of saving the bytes of strings in the ndarray directly, Pandas uses an object ndarray, which saves pointers to objects; because of this the dtype of this kind ndarray is object.

Here is an example:

  • the int64 array contains 4 int64 value.
  • the object array contains 4 pointers to 3 string objects.

enter image description here

2
  • 7
    Note however than having 'object' type columns has a major impact on the performance of the DataFrame read/write operations
    – erwanp
    Commented Jul 6, 2018 at 15:53
  • 5
    can I get datatype returned as string, somehow. I know I can always use type(df["column"].iloc[0]), but it may so happen that it is nan Commented Apr 28, 2019 at 8:12
72

@HYRY's answer is great. I just want to provide a little more context..

Arrays store data as contiguous, fixed-size memory blocks. The combination of these properties together is what makes arrays lightning fast for data access. For example, consider how your computer might store an array of 32-bit integers, [3,0,1].

enter image description here

If you ask your computer to fetch the 3rd element in the array, it'll start at the beginning and then jump across 64 bits to get to the 3rd element. Knowing exactly how many bits to jump across is what makes arrays fast.

Now consider the sequence of strings ['hello', 'i', 'am', 'a', 'banana']. Strings are objects that vary in size, so if you tried to store them in contiguous memory blocks, it'd end up looking like this.

enter image description here

Now your computer doesn't have a fast way to access a randomly requested element. The key to overcoming this is to use pointers. Basically, store each string in some random memory location, and fill the array with the memory address of each string. (Memory addresses are just integers.) So now, things look like this

enter image description here

Now, if you ask your computer to fetch the 3rd element, just as before, it can jump across 64 bits (assuming the memory addresses are 32-bit integers) and then make one extra step to go fetch the string.

The challenge for NumPy is that there's no guarantee the pointers are actually pointing to strings. That's why it reports the dtype as 'object'.

Shamelessly gonna plug my own course on NumPy where I originally discussed this.

3
  • 4
    @Ben: Thanks for your answer. Unrelated question: How did you create the graphics and animations in your answer???
    – hpy
    Commented Jan 24, 2021 at 16:20
  • 8
    @hpy Apple Keynote
    – Ben
    Commented Jan 26, 2021 at 16:57
  • really informative. thanks! Commented Jun 23 at 18:03
24

The accepted answer is good. I just wanted to reference the documentation. The documentation says:

Pandas uses the object dtype for storing strings.

The accepted answer did a great job explaining the "why"; strings are variable-length:

But for strings, the length of the string is not fixed.

But as the leading comment on the accepted answer once said : "Don't worry about it; it's supposed to be like this."

4
  • 1
    Why do i need to convert every column i pass into scipy or sklearn astype(str) for it to accept it? seems I should be able to apply that to all columns initially.
    – Tinkinc
    Commented Aug 21, 2019 at 12:53
  • I don't understand; @Tinkinc what happens if you don't convert columns to string? And this answer seems an elegant way to convert all columns to astype(str) although I'd still wonder string conversion is necessary Commented Aug 21, 2019 at 14:35
  • 1
    I cant fillna(0) all the objects in my dataframe stay (1,nan) instead of (1,0)
    – Tinkinc
    Commented Aug 21, 2019 at 16:03
  • Sorry @Tinkinc I still don't understand; I want to help, but your issue sounds more complex than a Stack Overflow comment. Consider asking a question, or joining me in chat. (just invited you) Commented Aug 21, 2019 at 16:31
12

As of version 1.0.0 (January 2020), pandas has introduced as an experimental feature providing first-class support for string types through pandas.StringDtype.

While you'll still be seeing object by default, the new type can be used by specifying a dtype of pd.StringDtype or simply 'string':

>>> pd.Series(['abc', None, 'def'])
0     abc
1    None
2     def
dtype: object
>>> pd.Series(['abc', None, 'def'], dtype=pd.StringDtype())
0     abc
1    <NA>
2     def
dtype: string
>>> pd.Series(['abc', None, 'def']).astype('string')
0     abc
1    <NA>
2     def
dtype: string
3
  • 5
    Don't use this.... yet. As they stated, The implementation may change without warning. which means new updates will break your old programs.
    – NoName
    Commented Feb 1, 2020 at 22:55
  • 4
    Well, that all depends on what you're going to use it for. If you want to use it in a production system where continuous package upgrades are necessary, and where API breakage cause an unacceptable maintenance burden, then sure, pay close attention to the word "experimental", but if you're using pandas to perform exploratory analyses in scripts whose life time do not increase a work day, then those concerns should mean little to you.
    – fuglede
    Commented Apr 19, 2020 at 8:57
  • 5
    As of Pandas 1.1, the API seems to be stabilized All dtypes can now be converted to StringDtype.
    – D3f0
    Commented Aug 21, 2020 at 6:53

Not the answer you're looking for? Browse other questions tagged or ask your own question.