4

I have a dataframe:

bq_selection_id  bq_balance  bq_market_id  bq_back_price
0         45094462      185.04       155           1.87
1         45094462      185.04       155           1.97
2         45094463      185.04       155           3.05
3         45094463      185.04       156           3.05
4         45094464      185.04       156           5.80
5         45094464      185.04       156           5.80
6         45094466      185.04       157         200.00
7         45094466      185.04       157         200.00
8         45094465      185.04       157            NaN
9         45094465      185.04       157            NaN

I would like to have two extra columns second_lowest, none_values both of them per group, groupby market id. Another words for market id 155 second_lowest 1.97 and there are no NaN values therefore none_values is False. I would like to get something like:

bq_selection_id bq_balance bq_market_id bq_back_price second_lowest none_val
0         45094462      185.04       155           1.87    1.97       False
1         45094462      185.04       155           1.97    1.97       False
2         45094463      185.04       155           3.05    1.97       False
3         45094463      185.04       156           3.05    5.80       False
4         45094464      185.04       156           5.80    5.80       False
5         45094464      185.04       156           6.40    5.80       False
6         45094466      185.04       157           1.00    1.70       True
7         45094466      185.04       157           1.70    1.70       True
8         45094465      185.04       157            NaN    1.70       True
9         45094465      185.04       157            NaN    1.70       True

Can you help me with this, please?

2
  • could you please clarify how did you get 1.70 for bq_market_id == 157? Mar 30, 2016 at 13:56
  • Basically I grouped all values possibly with .groupby('bq_market_id ') and then take the next min/smallest value in group in group with id 157 (1.00,1.70,NaN,NaN) so the min value 1.00 and the second smallest is 1.70 Mar 30, 2016 at 14:04

2 Answers 2

9

Combining ideas used in your previous questions (1, 2), you could use groupby/transform to assign a new value for each row in your DataFrame:

import numpy as np
import pandas as pd
pd.options.display.width = 1000

df = pd.DataFrame(
    {'bq_back_price': [1.87, 1.97, 3.05, 3.05, 5.8, 5.8, 200.0, 200.0, np.nan, np.nan], 
     'bq_balance': [1850.4, 1850.4, 1850.4, 1850.4, 1850.4, 1850.4, 1850.4, 
                    1850.4, 1850.4, 1850.4], 
     'bq_market_id': [155, 155, 155, 156, 156, 156, 157, 157, 157, 157], 
     'bq_selection_id': [45094462, 45094462, 45094463, 45094463, 45094464, 
                         45094464, 45094466, 45094466, 45094465, 45094465]})

grouped = df.groupby('bq_market_id')['bq_back_price']
df['second_lowest'] = grouped.transform(lambda x: x.nsmallest(2).max())
df['has_null'] = grouped.transform(lambda x: pd.isnull(x).any()).astype(bool)
print(df)

yields

   bq_back_price  bq_balance  bq_market_id  bq_selection_id  second_lowest has_null
0           1.87      1850.4           155         45094462           1.97    False
1           1.97      1850.4           155         45094462           1.97    False
2           3.05      1850.4           155         45094463           1.97    False
3           3.05      1850.4           156         45094463           5.80    False
4           5.80      1850.4           156         45094464           5.80    False
5           5.80      1850.4           156         45094464           5.80    False
6         200.00      1850.4           157         45094466         200.00     True
7         200.00      1850.4           157         45094466         200.00     True
8            NaN      1850.4           157         45094465         200.00     True
9            NaN      1850.4           157         45094465         200.00     True
2
  • Thanks. That is exactly what I needed. I was trying to do it through .apply, but of course transform is much better Mar 30, 2016 at 19:18
  • I get "Cannot use method %r with dtype %s" at some point "Cannot use method nsmallest with dtype object. Do you know why I might get this error at some point? @unutbu Apr 2, 2016 at 16:12
0

How about:

gb = df.groupby('bq_market_id')
df['second_lowest'] = gb.bq_back_price.apply(lambda x: x.sort_values(ascending=False).iloc[1])[df.bq_market_id]
df['none_val'] = gb.bq_back_price.apply(lambda x: x.isnull().values.any())[df.bq_market_id]
1
  • 1
    This orders NaN values before real numbers, giving the highest real value in the case there are NaNs in a group.
    – mdurant
    Mar 30, 2016 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.