28

Sometimes I want to show all of the rows in a pandas DataFrame, but only for a single command or code-block.

Of course I can set the "max_rows" display option to a large number, but then I have to repeat the command afterwards in order to revert to my preferred setting. (I like 12 rows max, personally).

pd.options.display.max_rows=1000
myDF
pd.options.display.max_rows=12

That's annoying.

I read in the documentation that I can use the pd.option_context() function to accomplish this if I combine my command with a "with" statement:

with pd.option_context("display.max_rows", 1000): myDF

I couldn't get that to work, (no output is returned). But I think such a solution would still be too much typing for routine incidental usage!

I wish there was some quick pythonic way to override the display options!
Does one exist? Have I overlooked something?

I like how one can alter the # of rows that the .head() function outputs by passing it an argument for the # of rows, but it still must be lower than the "display.max_rows" setting...

I know I could keep the "display.max_rows" setting really high all the time, and then tack a .head(12) function on most of the time, but I think most people would agree on how annoying that would be.

I am indeed aware that one can view all (or most of?) the values in a pandas Series by passing it to a core function such as list(). But that is tricky to do with a DF. Furthermore, it's hard to read when it's not in a tabular format.

Similar to the solution for my first question, I imagine there's probably a way to write my own function (to be placed in a startup script), but I'm not sure the best way to write it.

5
  • 3
    I often find it easier to view large DataFrames by using Excel via df.to_clipboard() and then paste into excel. Depending on the complexity of the data frame, you may need to use df.to_excel.
    – Alexander
    Jun 16, 2015 at 19:22
  • I'd like to respectfully disagree with @Alexander. When the DF is too large for viewing in the Notebook, it's usually because I haven't thought of the right aggregation. A graph could help too.
    – Ami Tavory
    Jun 16, 2015 at 19:34
  • Well, I gave N=1000 to indicate something that is reasonably large but not ridiculously large, but maybe I should have used something more like 100. It's not necessarily that I want to view "all" rows, because obviously I don't want to do that if N is huge. But what I often want to do is see MORE than just 12 rows, to help me as I'm testing out different commands. In many cases I'd be happy with jumping quickly between 12 and 60 rows, but I'd also like the option to see ALL rows for cases where the DF isn't huge.
    – MMelnicki
    Jun 16, 2015 at 19:43
  • and sometimes it's not about proper aggregation, or it's impossible to do so. This question is mostly about needing some CONVENIENT way to assess the outcomes of different variations of commands (eg, to assess how different attempts to aggregate the data even work!!!)
    – MMelnicki
    Jun 16, 2015 at 19:45

10 Answers 10

39

This won't display anything because it does not return anything:

with pd.option_context("display.max_rows", 1000): myDF

Calling display inside the with block should work:

with pd.option_context("display.max_rows", 1000):
    display(myDF)
7

This seems to work as expected in pandas 0.22.0 (importing only pandas, without IPython):

import pandas as pd    
with pd.option_context("display.max_rows", 1000): myDF

Presumably that's because the default behaviour is to return the repr of myDF. IDEs may well override this.

If that's too much typing, then a straightforward print to the terminal also works when wrapped in a function:

from __future__ import print_statement  # for python2

def show_rows(df, nrows=1000):
    with pd.option_context("display.max_rows", nrows): print(df)

Edit: calling show_rows(df) will by default print the first 1000 rows of your dataframe df to standard output.

2
  • 1
    This doesn't print anything.
    – jtiscione
    Sep 27, 2019 at 12:42
  • You just need to call the function - edited to show above.
    – tgolubch
    Nov 1, 2019 at 19:38
4

One-liner to force display all rows (in jupyter):

import IPython.display

IPython.display.HTML(df.to_html())
1
  • easy and concise!
    – MoneyBall
    Jan 28, 2020 at 6:30
2

You can use following helper function to print full data frame and set max_rows to normal after printing.

def print_full(df):
    import pandas as pd
    pd.set_option('display.max_rows', len(df))
    print(df)
    pd.reset_option('display.max_rows')
1

Best way to display required number of rows in a resulting dataframe:

For rows:

pd.set_option('display.max_rows',100)

For columns:

Use display.max_columns option
pd.set_option('display.max_columns',100)
0

You could write a function that explicitly calls display

E.g., consider this function:

from IPython.display import display

def show_more(df, lines):
    foo = 1
    display(df)
    foo = 2

When I call the function (just tried it):

>> show_more(df, 1000)
... # <- Shows here the DF

then it displays the dataframe, even though the line foo = 2 is executed after. It follows that you can set the options instead of the line foo = 1 and unset it in the line foo = 2. In fact, you can just use the context manager from your question, probably.

6
  • this seems to work nicely! i'm gonna play around with it some more, but it looks promising! thanks!!
    – MMelnicki
    Jun 16, 2015 at 19:56
  • 1
    I was using this all day yesterday, it has been a tremendous help.
    – MMelnicki
    Jun 17, 2015 at 17:34
  • Note: I ended up tweaking the function a bit, in order to work on columns as well, to be able to accommodate different default settings, when resetting at the end, and to have its own default settings (for when i pass it no arguments). Plus I removed the underscore from the name because that slows down typing.
    – MMelnicki
    Jun 17, 2015 at 17:41
  • def showmore(df, lines=None, cols=None): if lines is None: lines = 50 if cols is None: cols = 20 oldrows = pd.options.display.max_rows oldcols = pd.options.display.max_columns pd.options.display.max_rows=lines pd.options.display.max_columns=cols display(df) pd.options.display.max_rows=oldrows pd.options.display.max_columns=oldcols
    – MMelnicki
    Jun 17, 2015 at 17:49
  • aagh, sorry, i'm new to S.O., and am just figuring things out, not sure where the best place is to put my tweaked version of your solution! Seems like the comments is not the best place, since i lose code-block formatting.
    – MMelnicki
    Jun 17, 2015 at 17:52
0

Just set the option to display max rows:

max_number_rows_to_display = 100
pd.set_option('display.max_rows', max_number_rows_to_display)
0

An alternative to setting a global option on pandas is calling one of the display methods of the dataframe with the optional max_rows parameters, e.g.

    print(myDF.to_string(max_rows=50))

or

    print(myDF.to_html(max_rows=50))

The output isn't exactly the same as print(myDF), but it avoids side-effects.

0

Quick and concise:

print(df.to_string())
-1

df.show(number of records)

Example:

df=spark.read.format('csv').load('hdfs://localhost:9000/user/test/region_location', header='false', nullValue='null', schema=transchema)

df.show(50)
3
  • With Pandas 1.2.0: AttributeError: 'DataFrame' object has no attribute 'show'
    – SirVer
    Jan 27, 2021 at 13:06
  • Check your columns and remove any white space if they have. df.columns
    – Anil
    Jan 28, 2021 at 7:07
  • Not sure what that has to do with anything, even when any([c.isspace() for d in df.columns for c in d]) == False, DataFrame still has no attribute show.
    – SirVer
    Jan 29, 2021 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.