9

Base scenario

For a recommendation service I am training a matrix factorization model (LightFM) on a set of user-item interactions. For the matrix factorization model to yield the best results, I need to map my user and item IDs to a continuous range of integer IDs starting at 0.

I'm using a pandas DataFrame in the process, and I have found a MultiIndex to be extremely convenient to create this mapping, like so:

ratings = [{'user_id': 1, 'item_id': 1, 'rating': 1.0},
           {'user_id': 1, 'item_id': 3, 'rating': 1.0},
           {'user_id': 3, 'item_id': 1, 'rating': 1.0},
           {'user_id': 3, 'item_id': 3, 'rating': 1.0}]

df = pd.DataFrame(ratings, columns=['user_id', 'item_id', 'rating'])
df = df.set_index(['user_id', 'item_id'])
df

Out:
                 rating
user_id item_id 
1       1        1.0
1       3        1.0
3       1        1.0
3       1        1.0

And then allows me to get the continuous maps like so

df.index.labels[0]    # For users

Out:
FrozenNDArray([0, 0, 1, 1], dtype='int8')

df.index.labels[1]    # For items

Out:
FrozenNDArray([0, 1, 0, 1], dtype='int8')

Afterwards, I can map them back using df.index.levels[0].get_loc method. Great!

Extension

But, now I'm trying to streamline my model training process, ideally by training it incrementally on new data, preserving the old ID mappings. Something like:

new_ratings = [{'user_id': 2, 'item_id': 1, 'rating': 1.0},
               {'user_id': 2, 'item_id': 2, 'rating': 1.0}]

df2 = pd.DataFrame(new_ratings, columns=['user_id', 'item_id', 'rating'])
df2 = df2.set_index(['user_id', 'item_id'])
df2

Out:
                 rating
user_id item_id 
2       1        1.0
2       2        1.0

Then, simply appending the new ratings to the old DataFrame

df3 = df.append(df2)
df3

Out:
                 rating
user_id item_id 
1       1        1.0
1       3        1.0
3       1        1.0
3       3        1.0
2       1        1.0
2       2        1.0

Looks good, but

df3.index.labels[0]    # For users

Out:
FrozenNDArray([0, 0, 2, 2, 1, 1], dtype='int8')

df3.index.labels[1]    # For items

Out:
FrozenNDArray([0, 2, 0, 2, 0, 1], dtype='int8')

I added user_id=2 and item_id=2 in the later DataFrame on purpose, to illustrate where it goes wrong for me. In df3, labels 3 (for both user and item), have moved from integer position 1 to 2. So the mapping is no longer the same. What I'm looking for is [0, 0, 1, 1, 2, 2] and [0, 1, 0, 1, 0, 2] for user and item mappings respectively.

This is probably because of ordering in pandas Index objects, and I'm unsure if what I want is at all possible using a MultiIndex strategy. Looking for help on how most to effectively tackle this problem :)

Some notes:

  • I find using DataFrames convenient for several reasons, but I use the MultiIndex purely for the ID mappings. Alternatives without MultiIndex are completely acceptable.
  • I cannot guarantee that new user_id and item_id entries in new ratings are larger than any values in the old dataset, hence my example of adding id 2 when [1, 3] were present.
  • For my incremental training approach, I will need to store my ID maps somewhere. If I only load new ratings partially, I will have to store the old DataFrame and ID maps somewhere. Would be great if it could all be in one place, like it would be with an index, but columns work too.
  • EDIT: An additional requirement is to allow for row re-ordering of the original DataFrame, as might happen when duplicate ratings exist, and I want to keep the most recent one.

Solution (credits to @jpp for original)

I've made a modification to @jpp's answer to satisfy the additional requirement I've added later (tagged with EDIT). This also truly satisfies the original question as posed in the title, since it preserves the old index integer positions, regardless of rows being reordered for whatever reason. I've also wrapped things into functions:

from itertools import chain
from toolz import unique


def expand_index(source, target, index_cols=['user_id', 'item_id']):

    # Elevate index to series, keeping source with index
    temp = source.reset_index()
    target = target.reset_index()

    # Convert columns to categorical, using the source index and target columns
    for col in index_cols:
        i = source.index.names.index(col)
        col_cats = list(unique(chain(source.index.levels[i], target[col])))

        temp[col] = pd.Categorical(temp[col], categories=col_cats)
        target[col] = pd.Categorical(target[col], categories=col_cats)

    # Convert series back to index
    source = temp.set_index(index_cols)
    target = target.set_index(index_cols)

    return source, target


def concat_expand_index(old, new):
    old, new = expand_index(old, new)
    return pd.concat([old, new])


df3 = concat_expand_index(df, df2)

The result:

df3.index.labels[0]    # For users

Out:
FrozenNDArray([0, 0, 1, 1, 2, 2], dtype='int8')

df3.index.labels[1]    # For items

Out:
FrozenNDArray([0, 1, 0, 1, 0, 2], dtype='int8')
3
  • Is there a specific reason you need consistent labels, e.g. do you use them in logic? If they are important, should they not be part of your dataframe?
    – jpp
    Commented Jun 14, 2018 at 20:13
  • 1
    The thing I am trying to accomplish is to have consistent references to row or column positions in matrices or vectors generated from my DataFrame. This allows me to do incremental training on a machine learning model when introducing new users and items. Having it be part of the DataFrame is an acceptable approach, as noted at the bottom of my question. But since MultiIndex is so convenient for the initial step, I wonder if it can be extended.
    – Fulco
    Commented Jun 14, 2018 at 20:43
  • @Fulco I suggested the numpy tag for a more appropriate audience.
    – akilat90
    Commented Jun 15, 2018 at 9:47

2 Answers 2

4

I think the use of MultiIndex overcomplicates this objective:

I need to map my user and item IDs to a continuous range of integer IDs starting at 0.

This solution falls in to the below category:

Alternatives without MultiIndex are completely acceptable.


def add_mapping(df, df2, df3, column_name='user_id'):

    initial = df.loc[:, column_name].unique()
    new = df2.loc[~df2.loc[:, column_name].isin(initial), column_name].unique()
    maps = np.arange(len(initial))
    mapping = dict(zip(initial, maps))
    maps = np.append(maps, np.arange(np.max(maps)+1, np.max(maps)+1+len(new)))
    total = np.append(initial, new)
    mapping = dict(zip(total, maps))

    df3[column_name+'_map'] = df3.loc[:, column_name].map(mapping) 

    return df3

add_mapping(df, df2, df3, column_name='item_id')
add_mapping(df, df2, df3, column_name='user_id')

 user_id    item_id rating  item_id_map user_id_map
0   1          1    1.0         0           0
1   1          3    1.0         1           0
2   3          1    1.0         0           1
3   3          3    1.0         1           1
0   2          1    1.0         0           2
1   2          2    1.0         2           2

Explanation

This is how to maintain a mapping for the user_id values. Same holds for the item_id values as well.

These are the initial user_id values (unique):

initial_users = df['user_id'].unique()
# initial_users = array([1, 3])

user_map maintains a mapping for user_id values, as per your requirement:

user_id_maps = np.arange(len(initial_users))
# user_id_maps = array([0, 1])

user_map = dict(zip(initial_users, user_id_maps))
# user_map = {1: 0, 3: 1}

These are the new user_id values you got from df2 - ones that you didn't see in df:

new_users = df2[~df2['user_id'].isin(initial_users)]['user_id'].unique()
# new_users = array([2])

Now we update user_map for the total user base with the new users:

user_id_maps = np.append(user_id_maps, np.arange(np.max(user_id_maps)+1, np.max(user_id_maps)+1+len(new_users)))
# array([0, 1, 2])
total_users = np.append(initial_users, new_users)
# array([1, 3, 2])

user_map = dict(zip(total_users, user_id_maps))
# user_map = {1: 0, 2: 2, 3: 1}

Then, just map the values from user_map to df['user_id']:

df3['user_map'] = df3['user_id'].map(user_map)

user_id item_id rating  user_map
0   1   1       1.0          0
1   1   3       1.0          0
2   3   1       1.0          1
3   3   3       1.0          1
0   2   1       1.0          2
1   2   2       1.0          2
7
  • A few points. (1) Please don't use chained indexing, e.g. use .loc instead of df2[~df2[column_name].isin(initial)][column_name].unique(); (2) np.append is expensive, I fear performance issues. Wider point is the mapping is not intrinsically linked (versus for example, categoricals). Nevertheless, this does accomplish the task and you took the effort to wrap in a function. So +1.
    – jpp
    Commented Jun 15, 2018 at 9:07
  • Thanks @jpp ! (1) I had the thought of using loc but just went with chained indexing to make it less verbose. (Why is chained indexing frowned upon though?) (2) Instead of np.append, is list a good idea?
    – akilat90
    Commented Jun 15, 2018 at 9:13
  • (1) See Returning a view versus a copy, (2) I'm not sure, with this method it may be unavoidable.
    – jpp
    Commented Jun 15, 2018 at 9:15
  • Thanks @akilat90. Your answer satisfies the requirements as specified and like jpp I appreciate the effort of providing of providing a copy/paste working function. However, I find jpp's answer more elegant, so I've awarded him the bounty.
    – Fulco
    Commented Jun 20, 2018 at 12:23
  • @Fulco No worries about the bounty - jpp directly answers the question in the heading. I just feel that the MultiIndex is unnecessary for this task - partly, the reason is my unfamiliarity with complex MultiIndex tricks.
    – akilat90
    Commented Jun 20, 2018 at 12:31
2
+50

Forcing alignment of index labels after concatenation does not appear straightforward and, if there is a solution, it is poorly documented.

One option which may appeal to you is Categorical Data. With some careful manipulation, this can achieve the same purpose: each unique index value within a level has a one-to-one mapping to an integer, and this mapping persists even after concatenation with other dataframes.

from itertools import chain
from toolz import unique

# elevate index to series
df = df.reset_index()
df2 = df2.reset_index()

# define columns for reindexing
index_cols = ['user_id', 'item_id']

# convert to categorical with merged categories
for col in index_cols:
    col_cats = list(unique(chain(df[col], df2[col])))
    df[col] = pd.Categorical(df[col], categories=col_cats)
    df2[col] = pd.Categorical(df2[col], categories=col_cats)

# convert series back to index
df = df.set_index(index_cols)
df2 = df2.set_index(index_cols)

I use toolz.unique to return an ordered unique list, but if you don't have access to this library, you can use the identical unique_everseen recipe from the itertool docs.

Now let's have a look at the category codes underlying the 0th index level:

for data in [df, df2]:
    print(data.index.get_level_values(0).codes.tolist())

[0, 0, 1, 1]
[2, 2]

Then perform our concatenation:

df3 = pd.concat([df, df2])

Finally, check that categorical codes are aligned:

print(df3.index.get_level_values(0).codes.tolist())
[0, 0, 1, 1, 2, 2]

For each index level, note we must take the union of all index values across dataframes to form col_cats, otherwise the concatenation will fail.

2
  • Many new things; I didn't even know they existed!
    – akilat90
    Commented Jun 15, 2018 at 9:15
  • 1
    Thanks @jpp. I found your answer to be most elegant and extensible. I made a modification to satisfy a requirement I forgot to mention, and one that truly preserves the old index. I'll put this in my original question.
    – Fulco
    Commented Jun 20, 2018 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.