5

I have a pandas dataframe displaying users' performance on test questions. It looks like this:

userID     questionID   correct
-------------------------------
  1             1          1
  1             5          1
  1             6          0
  1             8          0
  1             10         1
  2             3          1
  2             5          1
  2             6          0
  .             .          .
  .             .          .
  .             .          .   

I want to make a feature vector for each user saying whether or not they got each question right, that looks something like this:

questionID     1     2      3     4     5     6     ...
userID       -------------------------------------------------
  1            1    NaN   NaN    NaN    1     0     ...
  2           NaN   NaN    1     NaN    1     0     ...
  .           ...
  .           ...
  .            

Each user only gets shown a subset of all the questions, so it's a sparse matrix.

How can I make the above table in pandas?

I wanted to do something like below - grouping by userID and questionID and then unstacking, but I'm not sure exactly how it should work.

df = df.groupby(['user_id','question_id'])
df.unstack()

Thanks for your help.

1 Answer 1

6

You're looking for pivot:

In [11]: df.pivot(values='correct', index='userID', columns='questionID')
Out[11]: 
questionID  1   3   5   6   8   10
userID                            
1            1 NaN   1   0   0   1
2          NaN   1   1   0 NaN NaN

You might like to reindex the columns (based on all the questions) if you're not surjective.

In [12]: _.reindex_axis(np.arange(1, 10), 1)
Out[12]: 
         1   2   3   4  5  6   7   8   9
userID                                  
1        1 NaN NaN NaN  1  0 NaN   0 NaN
2      NaN NaN   1 NaN  1  0 NaN NaN NaN

Note: Originally this answer suggested pivot_table (which uses an aggfunc on repeated values, by default mean, and that's not what you want here - as @U2EF1 points out), it offers some other additional features over pivot but is a little slower:

df.pivot_table(values='correct', rows='userID', cols='questionID')

I have this feeling that in older versions of pandas, pivot was sensitive to NaN so you had to use pivot_table...

4
  • @user3591836 Note that this is going to average the correct column, so make sure that (userID, questionID) pairs are unique!
    – U2EF1
    Jun 27, 2014 at 6:21
  • @U2EF1 great point, perhaps pivot is the correct function to use here (in my experience it's a little more sensitive though)... I was surprised it allows the NaN! Jun 27, 2014 at 6:23
  • df.pivot(index='userID', columns='questionID') does exactly the same thing, both work fine. And pandas tries to be NaN friendly all over the place :)
    – U2EF1
    Jun 27, 2014 at 6:27
  • @U2EF1 I have this feeling that pivot used to not play well with missing data, but maybe I'm making that up. Thanks, updated the answer (definitely pivot is correct here) Jun 27, 2014 at 6:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.