5

I would like to know if there is a faster and more "pythonic" way of doing the following, e.g. using some built in methods. Given a pandas DataFrame or numpy array of floats, if the value is equal or smaller than 0.5 I need to calculate the reciprocal value and multiply with -1 and replace the old value with the newly calculated one. "Transform" is probably a bad choice of words, please tell me if you have a better/more accurate description.

Thank you for your help and support!!

Data:

import numpy as np
import pandas as pd
dicti = {"A" : np.arange(0.0, 3, 0.1), 
         "B" : np.arange(0, 30, 1),
         "C" : list("ELVISLIVES")*3}
df = pd.DataFrame(dicti)

my function:

def transform_colname(df, colname):
    series = df[colname]    
    newval_list = []
    for val in series:
        if val <= 0.5:
            newval = (1/val)*-1
            newval_list.append(newval)
        else:
            newval_list.append(val)
    df[colname] = newval_list
    return df

function call:

transform_colname(df, colname="A")

**--> I'm summing up the results here, since comments wouldn't allow to post code (or I don't know how to do it).**

Thank you all for your fast and great answers!!

using ipython "%timeit" with "real" data:

my function: 10 loops, best of 3: 24.1 ms per loop

from jojo:

def transform_colname_v2(df, colname):
    series = df[colname]        
    df[colname] = np.where(series <= 0.5, 1/series*-1, series)
    return df

100 loops, best of 3: 2.76 ms per loop

from FooBar:

def transform_colname_v3(df, colname):
    df.loc[df[colname] <= 0.5, colname]  = - 1 / df[colname][df[colname] <= 0.5]
    return df

100 loops, best of 3: 3.32 ms per loop

from dmvianna:

def transform_colname_v4(df, colname):
    df[colname] = df[colname].where(df[colname] <= 0.5, (1/df[colname])*-1)
    return df

100 loops, best of 3: 3.7 ms per loop

Please tell/show me if you would implement your code in a different way!

One final QUESTION: (answered) How could "FooBar" and "dmvianna" 's versions be made "generic"? I mean, I had to write the name of the column into the function (since using it as a variable didn't work). Please explain this last point! --> thanks jojo, ".loc" isn't the right way, but very simple df[colname] is sufficient. changed the functions above to be more "generic". (also changed ">" to be "<=", and updated timing)

Thank you very much!!

3
  • @jojo: Thank you!! Your solution is more pythonic, but slower. My function: 10000 loops, best of 3: 164 µs per loop. Your version: 1000 loops, best of 3: 395 µs per loop.
    – tryptofame
    Aug 15, 2014 at 12:57
  • How does your speed test look like? It's hard to be faster than numpy... at least in python. ;)
    – j-i-l
    Aug 15, 2014 at 13:00
  • 1
    @tryptofame just try to benchmark with a non-trivial sized frame, both solns below will be MUCH faster)
    – Jeff
    Aug 15, 2014 at 13:01

3 Answers 3

6

The typical trick is to write a general mathematical operation to apply to the whole column, but then use indicators to select rows for which we actually apply it:

df.loc[df.A < 0.5, 'A']  = - 1 / df.A[df.A < 0.5] 

In[13]: df
Out[13]: 
            A   B  C
0        -inf   0  E
1  -10.000000   1  L
2   -5.000000   2  V
3   -3.333333   3  I
4   -2.500000   4  S
5    0.500000   5  L
6    0.600000   6  I
7    0.700000   7  V
8    0.800000   8  E
9    0.900000   9  S
10   1.000000  10  E
11   1.100000  11  L
12   1.200000  12  V
13   1.300000  13  I
14   1.400000  14  S
15   1.500000  15  L
16   1.600000  16  I
17   1.700000  17  V
18   1.800000  18  E
19   1.900000  19  S
20   2.000000  20  E
21   2.100000  21  L
22   2.200000  22  V
23   2.300000  23  I
24   2.400000  24  S
25   2.500000  25  L
26   2.600000  26  I
27   2.700000  27  V
28   2.800000  28  E
29   2.900000  29  S
2
  • 1
    don't chain index, use df.loc[..., ...] = value!
    – Jeff
    Aug 15, 2014 at 12:56
  • 1
    True. In my defense, it's the first day after vacations :)
    – FooBar
    Aug 15, 2014 at 12:57
5

If we are talking about arrays:

import numpy as np
a = np.array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6], dtype=np.float)
print 1 / a[a <= 0.5] * (-1)

This will, however only return the values smaller than 0.5.

Alternatively use np.where:

import numpy as np
a = np.array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6], dtype=np.float)
print np.where(a < 0.5, 1 / a * (-1), a)

Talking about pandas DataFrame:

As in @dmvianna's answer (so give some credit to him ;) ), adapting it to pd.DataFrame:

df.a = df.a.where(df.a > 0.5, (1 / df.a) * (-1))
1
  • Hehe, .head() will return only 5 elements or so... I was using it to test with a sample. ;7)
    – dmvianna
    Aug 15, 2014 at 13:15
2

As in @jojo's answer, but using pandas:

df.A = df.A.where(df.A > 0.5, (1/df.A)*-1)

or

df.A.where(df.A > 0.5, (1/df.A)*-1, inplace=True) # this should be faster

.where docstring:

Definition: df.A.where(self, cond, other=nan, inplace=False, axis=None, level=None, try_cast=False, raise_on_error=True)

Docstring: Return an object of same shape as self and whose corresponding entries are from self where cond is True and otherwise are from other.

2
  • FYI, Dataframe is not s sub-class of a np.ndarray (nor has ever been), though it does act like a 2-d ndarray.
    – Jeff
    Aug 15, 2014 at 13:15
  • Thanks @Jeff, I must have skipped history class. I did assume that due to its behaviour. pandas seems to be very compatible with numpy.
    – dmvianna
    Aug 15, 2014 at 13:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.