1

I have pandas data object - data - that is stored as Series of Series. The first series is indexed on ID1 and the second on ID2.

ID1      ID2
1        10259           0.063979
         14166           0.120145
         14167           0.177417
         14244           0.277926
         14245           0.436048
         15021           0.624367
         15260           0.770925
         15433           0.918439
         15763           1.000000
...
1453     812690          0.752274
         813000          0.755041
         813209          0.756425
         814045          0.778434
         814474          0.910647
         814475          1.000000
Length: 19726, dtype: float64

I have a function that uses values from this object for further data processing. Here is the function:

#Function
def getData(ID1, randomDraw): 
    dataID2 = data[ID1]
    value = dataID2.index[np.searchsorted(dataID2, randomDraw, side='left').iloc[0]]
    return value

I use np.vectorize to apply this function on a DataFrame - dataFrame - that has about 22 million rows.

dataFrame['ID2'] = np.vectorize(getData)(dataFrame['ID1'], dataFrame['RAND'])

where ID1 and RAND are columns with values that are feeding into the function.

The code takes about 6 hours to process everything. A similar implementation in Java takes only about 6 minutes to get through 22 million rows of data.

On running a profiler on my program I find that the most expensive call is the indexing into data and the second most expensive is searchsorted.

Function Name: pandas.core.series.Series.__getitem__
Elapsed inclusive time percentage: 54.44

Function Name: numpy.core.fromnumeric.searchsorted  
Elapsed inclusive time percentage: 25.49    

Using data.loc[ID1] to get data makes the program even slower. How can I make this faster? I understand that Python cannot achieve the same efficiency as Java but 6 hours compared to 6 minutes seems too much of a difference. Maybe I should be using a different data structure/ functions? I am using Python 2.7 and PTVS IDE.

Adding a minimum working example:

import numpy as np
import pandas as pd

np.random.seed = 0

#Creating a dummy data object - Series within Series
alt = pd.Series(np.array([ 0.25, 0.50,  0.75,  1.00]), index=np.arange(1,5))
data = pd.Series([alt]*1500, index=np.arange(1,1501))

#Creating dataFrame - 
nRows = 200000
d = {'ID1': np.random.randint(1500, size=nRows) + 1
     ,'RAND': np.random.uniform(low=0.0, high=1.0, size=nRows)}
dataFrame = pd.DataFrame(d)

#Function
def getData(ID1, randomDraw): 
    dataID2 = data[ID1]
    value = dataID2.index[np.searchsorted(dataID2, randomDraw, side='left').iloc[0]]
    return value

dataFrame['ID2'] = np.vectorize(getData)(dataFrame['ID1'], dataFrame['RAND'])
8
  • Try setting the index on the ID values this should speed the lookup significantly
    – EdChum
    Sep 23, 2014 at 16:10
  • It is indexed on ID values.
    – sriramn
    Sep 23, 2014 at 16:11
  • plz show the related code where you apply getData over the data-frame Sep 23, 2014 at 16:17
  • you should show what your actual problem is. it looks like you need to do a simple merge.
    – Jeff
    Sep 23, 2014 at 16:47
  • @Jeff: It is a Monte Carlo prediction procedure - for a random draw [RAND] I am looking up the probability density data[ID1] and seeing which of the alternatives set of ID2s is chosen
    – sriramn
    Sep 23, 2014 at 17:06

1 Answer 1

1

You may get a better performance with this code:

>>> def getData(ts):
...     dataID2 = data[ts.name]
...     i = np.searchsorted(dataID2.values, ts.values, side='left')
...     return dataID2.index[i]
... 
>>> dataFrame['ID2'] = dataFrame.groupby('ID1')['RAND'].transform(getData)
7
  • Great code - works really fast as the lookup order is not on groupby levels. Would it be possible to extend this to multi-column grouping? Say ID0 and ID1 with the dataID2 being data[ID0][ID1].
    – sriramn
    Sep 23, 2014 at 19:05
  • @RazorXsr yes, but would be slower because of number of groups. You may also need to update to master, see this patch Sep 23, 2014 at 19:15
  • OK. So for a multicolumn case I converted dataID2 = data[ts.name[0]][ts.name[1]] since the ts.name object becomes a tuple. That is the way to go right?
    – sriramn
    Sep 23, 2014 at 21:18
  • 1
    @RazorXsr data[ts.name] should work by itself. you can pass a tuple to a series with multi-index Sep 23, 2014 at 21:19
  • Can I update to master by using git clone git://github.com/pydata/pandas.git and then running python setup.py install? I do not see the same performance levels as what @Jeff reports - maybe because your patch is not yet in my build version.
    – sriramn
    Sep 24, 2014 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.