1

How do I merge 2 multi-index dataframes? I get the error below.

df1              cola_df1  colb_df1
Fruit   1/1/20      2         3
        1/2/20      4         5
Apple   1/1/20      8         9
        1/2/20      10        11

df2              cola_df2  colb_df2
Fruit   1/1/20      2         3
        1/2/20      999       5
Apple   1/1/20      8         9
        1/2/20      10        11

df1              cola_df1  colb_df1  cola_df2  colb_df2
Fruit   1/1/20      2         3        2         3
        1/2/20      99         5      999         5
Apple   1/1/20      8         9        8          9
        1/2/20      10        11       10         11

I tried the following:

pd.concat([df1, df2], keys=['x', 'y'], axis=1).swaplevel(0,1, axis=1).sort_index(axis=1)
cannot handle a non-unique multi-index

2 Answers 2

2

First is possible upgrade pandas? Because if using changed sample data like bellow I got different error if using your solution:

df = pd.concat([df1,df2], keys=['x','y'], axis=1).swaplevel(0,1,axis=1).sort_index(axis=1)
print (df)

ValueError: Reindexing only valid with uniquely valued Index objects


Test if indexes are duplicated by:

print (df1.index.is_unique)
print (df2.index.is_unique)

You can deduplicated MutliIndex by GroupBy.cumcount with DataFrame.set_index:

print (df1)
              cola_df1  colb_df1
Fruit 1/1/20         2         3
      1/2/20         4         5
Apple 1/1/20         8         9
Fruit 1/2/20        10        11 <-duplicated index

print (df2)
              cola_df2  colb_df2
Fruit 1/1/20         2         3
      1/2/20       999         5
Apple 1/1/20         8         9
      1/2/20        10        11
      
print (df1.index.is_unique)
False

print (df2.index.is_unique)
True

df1['g'] = df1.groupby(level=[0,1]).cumcount()
df2['g'] = df2.groupby(level=[0,1]).cumcount()
df1 = df1.set_index('g', append=True)
df2 = df2.set_index('g', append=True)

df = pd.concat([df1,df2], keys=['x','y'], axis=1).swaplevel(0,1, axis=1).sort_index(axis=1)
print (df)
               cola_df1 cola_df2 colb_df1 colb_df2
                      x        y        x        y
             g                                    
Apple 1/1/20 0      8.0      8.0      9.0      9.0
      1/2/20 0      NaN     10.0      NaN     11.0
Fruit 1/1/20 0      2.0      2.0      3.0      3.0
      1/2/20 0      4.0    999.0      5.0      5.0
             1     10.0      NaN     11.0      NaN <- added 1 only for duplicated
3
  • Thanks, but this generates a new index column, g, that is 0 throughout? So I still get the same error
    – asd
    Feb 16, 2021 at 15:30
  • @asd - I changed answer with sample data and some test if duplicated index. If still not working is possible upgrade pandas?
    – jezrael
    Feb 17, 2021 at 6:06
  • @asd - You are welcome! It seems you matching wrong answer df1.join(df2, on=["x", "y"], how="inner") , it cannot work here.
    – jezrael
    Feb 17, 2021 at 14:37
1
indx =pd.MultiIndex.from_product([["Fruit", "Apple"],["1/1/20", "1/2/20"]], names=["x", "y"])
df1 = pd.DataFrame({"cola_df1": [2,4,8,10], "colb_df1": [3,5,9,11]}, index=indx)
              cola_df1  colb_df1
x     y                         
Fruit 1/1/20         2         3
      1/2/20         4         5
Apple 1/1/20         8         9
      1/2/20        10        11
df2 = pd.DataFrame({"cola_df2": [2,999,8,10], "colb_df2": [3,5,9,11]}, index=indx)

df1.join(df2, on=["x", "y"], how="inner") 

output

              cola_df1  colb_df1  cola_df2  colb_df2
x     y                                             
Fruit 1/1/20         2         3         2         3
      1/2/20         4         5       999         5
Apple 1/1/20         8         9         8         9
      1/2/20        10        11        10        11

where "x" and "y" are the names of the multi-indexes should work. Check this link for further explanation.
how="outer" enables you to keep the indices (rows) that exist in one dataframe but do not exist in the other one.

3
  • Could you indicate why is that? you can check the link in the answer for more details.
    – sos
    Feb 17, 2021 at 16:46
  • no, I tried it with OP data and it worked fine. I suggest you to have a look at the link in my answer, because there you can see that join can be used to merge on two multiindex and more.
    – sos
    Feb 18, 2021 at 12:24
  • do you test OP solution?
    – jezrael
    Feb 18, 2021 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.