50

I am still new to Python pandas' pivot_table and would like to ask a way to count frequencies of values in one column, which is also linked to another column of ID. The DataFrame looks like the following.

import pandas as pd
df = pd.DataFrame({'Account_number':[1,1,2,2,2,3,3],
                   'Product':['A', 'A', 'A', 'B', 'B','A', 'B']
                  })

For the output, I'd like to get something like the following:

                Product
                A      B
Account_number           
      1         2      0
      2         1      2
      3         1      1

So far, I tried this code:

df.pivot_table(rows = 'Account_number', cols= 'Product', aggfunc='count')

This code gives me the two same things. What is the problems with the code above? A part of the reason why I am asking this question is that this DataFrame is just an example. The real data that I am working on has tens of thousands of account_numbers.

1
  • 1
    All the answers on this page will not work for DataFrames with more than 3 columns. The idiomatic solution is to use aggfunc='size'. See my answer below for more details
    – Ted Petrou
    Oct 20, 2018 at 14:44

5 Answers 5

51

You need to specify the aggfunc as len:

In [11]: df.pivot_table(index='Account_number', columns='Product', 
                        aggfunc=len, fill_value=0)
Out[11]:
Product         A  B
Account_number
1               2  0
2               1  2
3               1  1

It looks like count, is counting the instances of each column (Account_number and Product), it's not clear to me whether this is a bug...

3
  • @Andy_Hayden, +1. I don't think it's a bug, but I would wish the behavior to be more consistent, see: df.pivot_table(rows='Account_number', cols='Product', aggfunc=sum, fill_value=0)
    – CT Zhu
    Mar 14, 2014 at 17:48
  • @CTZhu I think it might be a bug (you don't expect the used columns to be included in the aggregation, in fact they make no sense with sum!) Mar 14, 2014 at 17:52
  • InvalidIndexError: Reindexing only valid with uniquely valued Index objects
    – tread
    Mar 16, 2020 at 13:50
45

Solution: Use aggfunc='size'

Using aggfunc=len or aggfunc='count' like all the other answers on this page will not work for DataFrames with more than three columns. By default, pandas will apply this aggfunc to all the columns not found in index or columns parameters.

For instance, if we had two more columns in our original DataFrame defined like this:

df = pd.DataFrame({'Account_number':[1, 1, 2 ,2 ,2 ,3 ,3], 
                   'Product':['A', 'A', 'A', 'B', 'B','A', 'B'], 
                   'Price': [10] * 7,
                   'Quantity': [100] * 7})

Output:

   Account_number Product  Price  Quantity
0               1       A     10       100
1               1       A     10       100
2               2       A     10       100
3               2       B     10       100
4               2       B     10       100
5               3       A     10       100
6               3       B     10       100

If you apply the current solutions to this DataFrame, you would get the following:

df.pivot_table(index='Account_number',
               columns='Product',
               aggfunc=len,
               fill_value=0)

Output:

                  Price    Quantity   
Product            A  B        A  B
Account_number                     
1                  2  0        2  0
2                  1  2        1  2
3                  1  1        1  1

Solution

Instead, use aggfunc='size'. Since size always returns the same number for each column, pandas does not call it on every single column and just does it once.

df.pivot_table(index='Account_number', 
               columns='Product',
               aggfunc='size',
               fill_value=0)

Output:

Product         A  B
Account_number      
1               2  0
2               1  2
3               1  1
3
  • By the way, there's a bug with margins=True github.com/pandas-dev/pandas/issues/27799 Oct 21, 2019 at 3:37
  • 1
    "By default, pandas will apply this aggfunc to all the columns not found in index or columns parameters." >> this feels like a bug
    – zthomas.nc
    Jul 29, 2020 at 17:20
  • Thank you so, so much. This was bothering me for hours until I found your answer.
    – User356
    Dec 2, 2022 at 18:20
19

In new version of Pandas, slight modification is required. I had to spend some time figuring out so just wanted to add that here so that someone can directly use this.

df.pivot_table(index='Account_number', columns='Product', aggfunc=len,
               fill_value=0)
4

You can use count:

df.pivot_table(index='Account_number', columns='Product', aggfunc='count')
1
  • 1
    Use aggfunc='size' because aggfunc='count' does not work.
    – Shane S
    Apr 14, 2022 at 18:40
1

I know this question is about pivot_table but for the problem given in the question, we can use crosstab:

out = pd.crosstab(df['Account_number'], df['Product'])

Output:

Product         A  B
Account_number      
1               2  0
2               1  2
3               1  1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.