7

I have a dataframe df:

ID    Var1     Var2
1     1.2        4
1     2.1        6
1     3.0        7
2     1.3        8
2     2.1        9
2     3.2        13

I want to find the pearson correlation coefficient value between Var1 and Var2 for every ID

So the result should look like this:

ID    Corr_Coef
1     0.98198
2     0.97073

update:

Must make sure all columns of variables are int or float

5
  • If you are are getting that last dataframe with ID Var1 only 1.0 then your input dataframe only has Var1, what are you correlating against? Check your input df. Jul 12, 2017 at 19:03
  • The df is exactly what I am showing up there. I've checked it. Could be a python version thing? When I use other groupby functions like count() for instance, I get two columns in the result Var1 and Var2
    – BKS
    Jul 12, 2017 at 19:15
  • Not likely. You have somthing else really strange going on. Jul 12, 2017 at 19:16
  • I figured it out. It was reading Var2 as a string for some reason. So I just changed the type of the column to an integer and it worked.
    – BKS
    Jul 12, 2017 at 19:24
  • You might want to go back and revisit Brad's solution. Jul 12, 2017 at 19:25

4 Answers 4

15

To get your desired output format you could use .corrwith:

corrs = (df[['Var1', 'ID']]
        .groupby('ID')
        .corrwith(df.Var2)
        .rename(columns={'Var1' : 'Corr_Coef'}))

print(corrs)
    Corr_Coef
ID           
1     0.98198
2     0.97073

Generalized solution:

import numpy as np

def groupby_coef(df, col1, col2, on_index=True, squeeze=True, name='coef',
                 keys=None, **kwargs):
    """Grouped correlation coefficient between two columns

    Flat result structure in contrast to `groupby.corr()`.

    Parameters
    ==========
    df : DataFrame
    col1 & col2: str
        Columns for which to calculate correlation coefs
    on_index : bool, default True
        Specify whether you're grouping on index
    squeeze : bool, default True
        True -> Series; False -> DataFrame
    name : str, default 'coef'
        Name of DataFrame column if squeeze == True
    keys : column label or list of column labels / arrays
        Passed to `pd.DataFrame.set_index`
    **kwargs :
        Passed to `pd.DataFrame.groupby`
    """

    # If we are grouping on something other than the index, then
    #     set as index first to avoid hierarchical result.
    # Kludgy, but safer than trying to infer.
    if not on_index:
        df = df.set_index(keys=keys)
        if not kwargs:
            # Assume we're grouping on 0th level of index
            kwargs = {'level': 0}
    grouped = df[[col1]].groupby(**kwargs)
    res = grouped.corrwith(df[col2])
    res.columns = [name]
    if squeeze:
        res = np.squeeze(res)
    return res

Examples:

df_1 = pd.DataFrame(np.random.randn(10, 2), 
                    index=[1]*5 + [2]*5).add_prefix('var')
df_2 = df_1.reset_index().rename(columns={'index': 'var2'})

print(groupby_coef(df_1, 'var0', 'var1', level=0))
1    7.424e-18
2   -9.481e-19
Name: coef, dtype: float64

print(groupby_coef(df_2, col1='var0', col2='var1', 
                   on_index=False, keys='var2'))
var2
1    7.424e-18
2   -9.481e-19
Name: coef, dtype: float64
6
  • I keep getting an error when using .corrwith that I don't understand /usr/local/lib/python2.7/dist-packages/pandas/core/groupby.pyc in corrwith(self, other, axis, drop) /usr/local/lib/python2.7/dist-packages/pandas/core/groupby.pyc in wrapper(*args, **kwargs) 590 *args, **kwargs) 591 except (AttributeError): --> 592 raise ValueError 593 594 return wrapper ValueError:
    – BKS
    Jul 12, 2017 at 18:31
  • What does import pandas --> pandas.__version__ give you? Secondly, are there other columns to your Dataframe besides the 3 given here? Jul 12, 2017 at 18:31
  • There aren't other columns. But even if there are shouldn't df[['Var1', 'ID']] remove them? Also I'm using pandas 0.19.2
    – BKS
    Jul 12, 2017 at 18:33
  • To your question--yes. I'm a bit stumped, and just ran the script above after switching to 0.19.2 with no issue. I suppose it could be an issue with Python 2.x v 3.x, go ahead and accept @Scott's answer if that does work for you. Jul 12, 2017 at 18:36
  • I did get Brad's solution to work. However, I am too, am on Python 3. Jul 12, 2017 at 18:39
10
df.groupby('ID').corr()

Output:

             Var1      Var2
ID                         
1  Var1  1.000000  0.981981
   Var2  0.981981  1.000000
2  Var1  1.000000  0.970725
   Var2  0.970725  1.000000

With OP output formating.

df_out = df.groupby('ID').corr()
(df_out[~df_out['Var1'].eq(1)]
          .reset_index(1, drop=True)['Var1']
          .rename('Corr_Coef')
          .reset_index())

Output:

   ID  Corr_Coef
0   1   0.981981
1   2   0.970725
7
  • When I use df.groupby('ID').corr() all it does is compare Var1 with itself... so I only get one column output full of 1's.
    – BKS
    Jul 12, 2017 at 18:47
  • @BKS Yes, see the second part after 'With OP (original poster's) output formatting. Jul 12, 2017 at 18:49
  • I'm not quite understanding your answer. What is df_out? the output dataframe from the .corr function? Cause what I'm getting is just random answers.
    – BKS
    Jul 12, 2017 at 18:51
  • df_out = df.groupby('ID').corr() Jul 12, 2017 at 18:52
  • ok so I understood it correctly. Which brings us back to the original comment. I get an error that Var2 doesn't exit because the corr() function only compare Var1 with itself, doesn't consider Var2. The output matrix is only Var1
    – BKS
    Jul 12, 2017 at 18:54
0

Simple solution:

df.groupby('ID').corr().unstack().iloc[:,1]
0

Since both solutions did not work for me I will post one that allows to calculate the correlation between one column in different groups for python3. This is hopefully addressing the same issue @BKS ran into.

data = df[['date', 'group_id', 'var1']]
data_new = data.set_index(['date', 'group_id']).unstack(['group_id'])
final_df = pd.DataFrame(data_new.to_numpy(), columns=data_new.columns)
dfCorr = final_df.corr()

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.