63

I have a dataframe like the one displayed below:

# Create an example dataframe about a fictional army
raw_data = {'regiment': ['Nighthawks', 'Nighthawks', 'Nighthawks', 'Nighthawks'],
            'company': ['1st', '1st', '2nd', '2nd'],
            'deaths': ['kkk', 52, '25', 616],
            'battles': [5, '42', 2, 2],
            'size': ['l', 'll', 'l', 'm']}
df = pd.DataFrame(raw_data, columns = ['regiment', 'company', 'deaths', 'battles', 'size'])

enter image description here

My goal is to transform every single string inside of the dataframe to upper case so that it looks like this:

enter image description here

Notice: all data types are objects and must not be changed; the output must contain all objects. I want to avoid to convert every single column one by one... I would like to do it generally over the whole dataframe possibly.

What I tried so far is to do this but without success

df.str.upper()
1
  • 3
    str only works for series...
    – IanS
    Sep 15, 2016 at 13:18

8 Answers 8

82

astype() will cast each series to the dtype object (string) and then call the str() method on the converted series to get the string literally and call the function upper() on it. Note that after this, the dtype of all columns changes to object.

In [17]: df
Out[17]: 
     regiment company deaths battles size
0  Nighthawks     1st    kkk       5    l
1  Nighthawks     1st     52      42   ll
2  Nighthawks     2nd     25       2    l
3  Nighthawks     2nd    616       2    m

In [18]: df.apply(lambda x: x.astype(str).str.upper())
Out[18]: 
     regiment company deaths battles size
0  NIGHTHAWKS     1ST    KKK       5    L
1  NIGHTHAWKS     1ST     52      42   LL
2  NIGHTHAWKS     2ND     25       2    L
3  NIGHTHAWKS     2ND    616       2    M

You can later convert the 'battles' column to numeric again, using to_numeric():

In [42]: df2 = df.apply(lambda x: x.astype(str).str.upper())

In [43]: df2['battles'] = pd.to_numeric(df2['battles'])

In [44]: df2
Out[44]: 
     regiment company deaths  battles size
0  NIGHTHAWKS     1ST    KKK        5    L
1  NIGHTHAWKS     1ST     52       42   LL
2  NIGHTHAWKS     2ND     25        2    L
3  NIGHTHAWKS     2ND    616        2    M

In [45]: df2.dtypes
Out[45]: 
regiment    object
company     object
deaths      object
battles      int64
size        object
dtype: object
0
69

This can be solved by the following applymap method:

df = df.applymap(lambda s: s.lower() if type(s) == str else s)
0
16

Loops are very slow instead of using apply function to each and cell in a row, try to get columns names in a list and then loop over list of columns to convert each column text to lowercase.

Code below is the vector operation which is faster than apply function.

for columns in dataset.columns:
    dataset[columns] = dataset[columns].str.lower() 
1
  • 1
    Nice def m3(dataset): for columns in dataset.columns: dataset[columns] = dataset[columns].str.upper() return dataset %timeit pd.concat([df[col].astype(str).str.upper() for col in df.columns], axis=1) %timeit df.apply(lambda x: x.astype(str).str.upper()) %timeit m3(df) 70.5 ms ± 1.26 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) 82.1 ms ± 1.46 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) 40.8 ms ± 546 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) Aug 21, 2020 at 13:16
9

Since str only works for series, you can apply it to each column individually then concatenate:

In [6]: pd.concat([df[col].astype(str).str.upper() for col in df.columns], axis=1)
Out[6]: 
     regiment company deaths battles size
0  NIGHTHAWKS     1ST    KKK       5    L
1  NIGHTHAWKS     1ST     52      42   LL
2  NIGHTHAWKS     2ND     25       2    L
3  NIGHTHAWKS     2ND    616       2    M

Edit: performance comparison

In [10]: %timeit df.apply(lambda x: x.astype(str).str.upper())
100 loops, best of 3: 3.32 ms per loop

In [11]: %timeit pd.concat([df[col].astype(str).str.upper() for col in df.columns], axis=1)
100 loops, best of 3: 3.32 ms per loop

Both answers perform equally on a small dataframe.

In [15]: df = pd.concat(10000 * [df])

In [16]: %timeit pd.concat([df[col].astype(str).str.upper() for col in df.columns], axis=1)
10 loops, best of 3: 104 ms per loop

In [17]: %timeit df.apply(lambda x: x.astype(str).str.upper())
10 loops, best of 3: 130 ms per loop

On a large dataframe my answer is slightly faster.

0
7

try this

df2 = df2.apply(lambda x: x.str.upper() if x.dtype == "object" else x)  
0
3

If you want to conserve the dtype use isinstance(obj,type)

df.apply(lambda x: x.str.upper().str.strip() if isinstance(x, object) else x)
0

if you want conserv dtype or change only one type.. try for and if:

for x in dados.columns:
    if dados[x].dtype == 'object':
        print('object - allow upper')
        dados[x] = dados[x].str.upper()
    else:
        print('other? - not allow upper')
        dados[x] = dados[x].str.upper()
1
  • Welcome to SO! Please check your answer as the not allow upper in the else part is doing upper anyway.
    – lepsch
    Sep 21, 2022 at 13:23
0

You can apply it for every cols...

oh_df.columns = map(str.lower, oh_df.columns)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.