126

Why do we use loc for pandas dataframes? it seems the following code with or without using loc both compiles and runs at a similar speed:

%timeit df_user1 = df.loc[df.user_id=='5561']

100 loops, best of 3: 11.9 ms per loop

or

%timeit df_user1_noloc = df[df.user_id=='5561']

100 loops, best of 3: 12 ms per loop

So why use loc?

Edit: This has been flagged as a duplicate question. But although pandas iloc vs ix vs loc explanation? does mention that

you can do column retrieval just by using the data frame's __getitem__:

df['time']    # equivalent to df.loc[:, 'time']

it does not say why we use loc, although it does explain lots of features of loc. But my specific question is: why not just omit loc altogether? For this question, I have accepted a very detailed answer below.

Also in the above post, the answer (which I do not think is an answer) is really well hidden in the discussion, and any person searching for what I was, would find it hard to locate the information and would be much better served by the answer provided to my question here.

10
  • Possible duplicate of pandas iloc vs ix vs loc explanation?
    – LinkBerest
    Aug 11, 2016 at 1:54
  • 13
    @JGreenwell - not really, there they are discussing the difference between .loc, .iloc and .ix but here I am simply asking why use .loc at all, not why use it over .iloc or .ix, Im not interested in iloc or ix, Im trying to understand loc first and why we use it as opposed to just leaving it out with nothing in its place. Aug 11, 2016 at 1:58
  • see the end of the answer by ajcr which includes a general use of both .loc and .iloc vs .ix: or the relavent part "if you're only indexing using labels, or only indexing using integer positions, stick with loc or iloc to avoid unexpected results."
    – LinkBerest
    Aug 11, 2016 at 2:00
  • 2
    Essentially, there are fall backs and best guesses that pandas makes when you don't specify the indexing technique. So it goes through each of them. On a DataFrame, the default is use .loc on columns. On Series, the default is use .loc on rows, because there is no columns.
    – Kartik
    Aug 11, 2016 at 2:08
  • 5
    JGreenwell and Kartik - I do not understand. I am not, i repeat, I am not interested in anything to do with .iloc, lets just pretend .iloc does not exist and lets pretend .ix does not exist. I just want to know why I should use .loc rather than just leaving it out all together as in the code in my question. Aug 11, 2016 at 2:08

3 Answers 3

108
  • Explicit is better than implicit.

    df[boolean_mask] selects rows where boolean_mask is True, but there is a corner case when you might not want it to: when df has boolean-valued column labels:

    In [229]: df = pd.DataFrame({True:[1,2,3],False:[3,4,5]}); df
    Out[229]: 
       False  True 
    0      3      1
    1      4      2
    2      5      3
    

    You might want to use df[[True]] to select the True column. Instead it raises a ValueError:

    In [230]: df[[True]]
    ValueError: Item wrong length 1 instead of 3.
    

    Versus using loc:

    In [231]: df.loc[[True]]
    Out[231]: 
       False  True 
    0      3      1
    

    In contrast, the following does not raise ValueError even though the structure of df2 is almost the same as df1 above:

    In [258]: df2 = pd.DataFrame({'A':[1,2,3],'B':[3,4,5]}); df2
    Out[258]: 
       A  B
    0  1  3
    1  2  4
    2  3  5
    
    In [259]: df2[['B']]
    Out[259]: 
       B
    0  3
    1  4
    2  5
    

    Thus, df[boolean_mask] does not always behave the same as df.loc[boolean_mask]. Even though this is arguably an unlikely use case, I would recommend always using df.loc[boolean_mask] instead of df[boolean_mask] because the meaning of df.loc's syntax is explicit. With df.loc[indexer] you know automatically that df.loc is selecting rows. In contrast, it is not clear if df[indexer] will select rows or columns (or raise ValueError) without knowing details about indexer and df.

  • df.loc[row_indexer, column_index] can select rows and columns. df[indexer] can only select rows or columns depending on the type of values in indexer and the type of column values df has (again, are they boolean?).

    In [237]: df2.loc[[True,False,True], 'B']
    Out[237]: 
    0    3
    2    5
    Name: B, dtype: int64
    
  • When a slice is passed to df.loc the end-points are included in the range. When a slice is passed to df[...], the slice is interpreted as a half-open interval:

    In [239]: df2.loc[1:2]
    Out[239]: 
       A  B
    1  2  4
    2  3  5
    
    In [271]: df2[1:2]
    Out[271]: 
       A  B
    1  2  4
    
6
  • 4
    Why didn't you use quote marks around the column name? Wouldn't df[['True']] work correctly? Jun 30, 2018 at 18:20
  • 2
    @LS df[['True']] doesn't appear to work correctly in the example. the column names aren't strings, they're boolean objects. looks like pandas doesn't require column names to be strings (different from e.g. R where names(df) is character and [[]] coerces the input to character) Jul 11, 2019 at 2:17
  • 12
    I would say the last point is the most important and push it to the top, that's actually dangerouly different behaviour Apr 22, 2020 at 6:59
  • 8
    Also one important difference with slices is that loc[1:2] works on index, while df2[1:2] works on simple order of rows regardless of index. That is in the last example above, if index was starting with [3,4], the loc[1:2] would not return anything, while df2[1:2] would still return the first row with index 3. May 31, 2020 at 9:02
  • Another thing is that loc always makes a copy, whereas the non-loc approach does not: see stackoverflow.com/questions/20625582/… so the non-loc approach leads to having to explicitly add .copy() when slicing dataframes, to deal with the settingwithcopywarning Apr 10 at 18:20
19

Performance Consideration on multiple columns "Chained Assignment" with and without using .loc

Let me supplement the already very good answers with the consideration of system performance.

The question itself includes a comparison on the system performance (execution time) of 2 pieces of codes with and without using .loc. The execution times are roughly the same for the code samples quoted. However, for some other code samples, there could be considerable difference on execution times with and without using .loc: e.g. several times difference or more!

A common case of pandas dataframe manipulation is we need to create a new column derived from values of an existing column. We may use the codes below to filter conditions (based on existing column) and set different values to the new column:

df[df['mark'] >= 50]['text_rating'] = 'Pass'

However, this kind of "Chained Assignment" does not work since it could create a "copy" instead of a "view" and assignment to the new column based on this "copy" will not update the original dataframe.

2 options available:

    1. We can either use .loc, or
    1. Code it another way without using .loc

2nd case e.g.:

df['text_rating'][df['mark'] >= 50] = 'Pass'

By placing the filtering at the last (after specifying the new column name), the assignment works well with the original dataframe updated.

The solution using .loc is as follows:

df.loc[df['mark'] >= 50, 'text_rating'] = 'Pass'

Now, let's see their execution time:

Without using .loc:

%%timeit 
df['text_rating'][df['mark'] >= 50] = 'Pass'

2.01 ms ± 105 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

With using .loc:

%%timeit 
df.loc[df['mark'] >= 50, 'text_rating'] = 'Pass'

577 µs ± 5.13 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

As we can see, with using .loc, the execution time is more than 3X times faster!

For a more detailed explanation of "Chained Assignment", you can refer to another related post How to deal with SettingWithCopyWarning in pandas? and in particular the answer of cs95. The post is excellent in explaining the functional differences of using .loc. I just supplement here the system performance (execution time) difference.

5

In addition to what has already been said (issues with having True, False as column name without using loc and ability to select rows and columns with loc and ability to do slicing for row and column selections), another big difference is that you can use loc to assign values to specific rows and columns. If you try to select a subset of the dataframe using boolean series and attempt to change a value of that subset selection you will likely get the SettingWithCopy warning.

Let's say you're trying to change the "upper management" column for all the rows whose salary is bigger than 60000.

This:

mask = df["salary"] > 60000
df[mask]["upper management"] = True

throws the warning that "A value is is trying to be set on a copy of a slice from a Dataframe" and won't work because df[mask] creates a copy and trying to update "upper management" of that copy has no effect on the original df.

But this succeeds:

mask = df["salary"] > 60000
df.loc[mask,"upper management"] = True

Note that in both cases you can do df[df["salary"] > 60000] or df.loc[df["salary"] > 60000], but I think storing boolean condition in a variable first is cleaner.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.