1

I am trying to do the following on a group object:

  • get the interarrival time

  • set the range, flag with vectorized where

  • get cumsum over interarrival time

    def deltat(g):
        try:
            g['tavg'] = g[ g['alert_v']==1 ]['timeindex'].diff(1)
    
            g['iqt'] = np.where( g['value'] > g['value'].quantile(.90) or g['value']< g['value'].quantile(.10),1,0)
            #pd.to_datetime(g[['tavg']], format='%H:%M:%S')
            #print type(g['tavg'] )
    
            g['cumt'] = g['tavg'].cumsum(0) #pd.rolling_sum(g['tavg'],2,0).shift(1)
            print g.head()
            return g
        except:
            pass
    
    d.sort_index(axis=0, inplace=True)
    d=d.groupby(['source','subject_id','alert_t','variable'],as_index=False,group_keys=False).apply(lambda x: deltat(x)) 
    

error: I am getting a StopIteration error. what causes this? why does the exception not just pass?

cumsum Am I using cumsum correctly, or do I need to use a rolling_sum to get the sums over each subsequent two time values rows in the column?

-any help is appreciated

--edit: here is some sample input:

 d = pd.DataFrame({'alert_v': [1]*4 + [0]*4,
              'value': np.random.rand(0,4)*3 + np.random.rand(0,4),
              'timeindex': pd.date_range(end='6/20/2012',periods=8)
             })

what I need is output with columns: tavg which is difference subsequent times where alert_v is 1. iqt need to set 1 for any value above/below quantile levels or 0 cumt this is the cumulative sum of each subsequent tavg value, ie the cumsum

alert_v value timeindex tavg iqt cumt 1 3.1 6/13 NaN 0 NaN 1 2.9 6/14 1 0 1 0 .3 6/15 Nan 0 NaN 1 3.3 6/16 2 0 3 0 .3 6/17 NaN 0 NaN 0 .5 6/18 NaN 0 NaN 0 .2 6/19 NaN 1 NaN 1 3.8 6/20 4 1 7

Attempts at vectorization just still produce the StopIteration error:

 `gg['cumt'] = gg.apply(lambda x: pd.rolling_sum(x['tavg'],2, min_periods=2).shift(1) )`

or gg['cumt'] = gg.apply( lambda x: x['tavg'] + x['tavg'].shift(1)[1:] )

For tavg and iqt, I use these two approaches, but in two different functions... putting them in one function caused a problem.

  g['tavg'] = g[ g['alert_v']==1 ]['timeindex'].diff(1)
  g['iqt'] = g['value'].map(lambda x: x > g['value'].quantile(.90) and 1 or x < g['value'].quantile(.10) and 1 or 0)

alert_v value tavg iqt
timeindex
1984-12-12 13:33:00 0 86 NaT NaN
1984-12-12 14:08:00 0 85 NaT 1
1984-12-12 14:08:00 0 85 NaT 1
1984-12-12 14:08:00 0 84 NaT 1
1984-12-12 14:08:00 0 84 NaT 1
1984-12-12 14:08:00 1 82 NaT 1
1984-12-12 14:25:00 1 83 00:17:00 1
1984-12-12 14:47:00 1 83 00:22:00 1
1984-12-12 16:37:00 0 88 01:50:00 1
1984-12-12 16:37:00 1 82 01:50:00 1
1984-12-12 16:37:00 0 90 01:50:00 1
1984-12-12 17:52:00 0 85 01:15:00 0
1984-12-12 17:52:00 1 95 01:15:00 0
1984-12-12 19:29:00 1 91 01:37:00 0
1984-12-12 19:29:00 0 95 01:37:00 0

anyhow, how do i take the cumsum over filtered rows ? (thanks for the vectorization tip)

2
  • pls show your input and what is the expected output, in a copy-pastable form. What you are doing is very inefficient. A groupby should try to use vectorized functions when possible. Then join them up at the end.
    – Jeff
    Jul 4, 2014 at 15:44
  • The stop iteration error might be due to not returning a dataframe in the except clause. See this post: stackoverflow.com/questions/24974759/… Jul 27, 2014 at 3:38

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.