42

Is there any way to merge on a single level of a MultiIndex without resetting the index?

I have a "static" table of time-invariant values, indexed by an ObjectID, and I have a "dynamic" table of time-varying fields, indexed by ObjectID+Date. I'd like to join these tables together.

Right now, the best I can think of is:

dynamic.reset_index().merge(static, left_on=['ObjectID'], right_index=True)

However, the dynamic table is very big, and I don't want to have to muck around with its index in order to combine the values.

3
  • What if you created an additional column with the level of the MultiIndex you want to join on and then merged/joined on that on that? Might not be totally efficient, but at least you maintain the index. May 20, 2013 at 19:50
  • Yes, that would work. It'd cost some memory, and it wouldn't help speed. At that point, though, I think I might as well drop the index entirely, if it's not going to help speed up merges. May 20, 2013 at 21:27
  • 2
    corresponding github issue github.com/pydata/pandas/issues/3662 May 21, 2013 at 19:59

4 Answers 4

33

Yes, since pandas 0.14.0, it is now possible to merge a singly-indexed DataFrame with a level of a multi-indexed DataFrame using .join.

df1.join(df2, how='inner') # how='outer' keeps all records from both data frames

The 0.14 pandas docs describes this as equivalent but more memory efficient and faster than:

merge(df1.reset_index(),
      df2.reset_index(),
      on=['index1'],
      how='inner'
     ).set_index(['index1','index2'])

The docs also mention that .join can not be used to merge two multiindexed DataFrames on a single level and from the GitHub tracker discussion for the previous issue, it seems like this might not of priority to implement:

so I merged in the single join, see #6363; along with some docs on how to do a multi-multi join. That's fairly complicated to actually implement. and IMHO not worth the effort as it really doesn't change the memory usage/speed that much at all.

However, there is a GitHub conversation regarding this, where there has been some recent development https://github.com/pydata/pandas/issues/6360. It is also possible achieve this by resetting the indices as mentioned earlier and described in the docs as well.


Update for pandas >= 0.24.0

It is now possible to merge multiindexed data frames with each other. As per the release notes:

index_left = pd.MultiIndex.from_tuples([('K0', 'X0'), ('K0', 'X1'),
                                        ('K1', 'X2')],
                                        names=['key', 'X'])

left = pd.DataFrame({'A': ['A0', 'A1', 'A2'],
                     'B': ['B0', 'B1', 'B2']}, index=index_left)

index_right = pd.MultiIndex.from_tuples([('K0', 'Y0'), ('K1', 'Y1'),
                                        ('K2', 'Y2'), ('K2', 'Y3')],
                                        names=['key', 'Y'])

right = pd.DataFrame({'C': ['C0', 'C1', 'C2', 'C3'],
                      'D': ['D0', 'D1', 'D2', 'D3']}, index=index_right)

left.join(right)

Out:

            A   B   C   D
key X  Y                 
K0  X0 Y0  A0  B0  C0  D0
    X1 Y0  A1  B1  C0  D0
K1  X2 Y1  A2  B2  C1  D1

[3 rows x 4 columns]
4
  • 2
    For your first example, how do you select the level of the MultiIndex to use for the join?
    – Carl G
    Jul 22, 2016 at 20:06
  • 3
    nvm, the docs say that it joins based upon the index level having the same name as the single-level index.
    – Carl G
    Jul 22, 2016 at 20:18
  • And joins are extremely fast in pandas Apr 10, 2017 at 1:52
  • 1
    @HenryHenrinson If you want use the union of the keys from the data frames (i.e. keep all records from both frames and introduce NaN as needed to fill the resulting table), use how=outer. Since merge was used as an example in the question, I used how=inner, which is the default behavior for merge. You can read more about the different joins available via the how parameter in the pandas documentation and by typing ?pd.DataFrame.join in your Python console. I added a short note as a comment above. Oct 10, 2017 at 12:21
2

I get around this by reindexing the dataframe merging to have the full multiindex so that a left join is possible.

# Create the left data frame
import pandas as pd
idx = pd.MultiIndex(levels=[['a','b'],['c','d']],labels=[[0,0,1,1],[0,1,0,1]], names=['lvl1','lvl2'])
df = pd.DataFrame([1,2,3,4],index=idx,columns=['data'])

#Create the factor to join to the data 'left data frame'
newFactor = pd.DataFrame(['fact:'+str(x) for x in df.index.levels[0]], index=df.index.levels[0], columns=['newFactor'])

Do the join on the subindex by reindexing the newFactor dataframe to contain the index of the left data frame

df.join(newFactor.reindex(df.index,level=0))
2

I would use mapping for a single column:

df1['newcol'] = df1.index.get_level_values(-1).map(lambda x: df2.newcol[x])
1

This works for me!

gData.columns = gData.columns.droplevel(0)

grpData = gData.reset_index()
grpData

pd.merge(grpData,cusData,how='inner')

Here gData is multi index dataframe with two levels and cusData is a single index dataframe.

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.