104

Is it possible to use TQDM progress bar when importing and indexing large datasets using Pandas?

Here is an example of of some 5-minute data I am importing, indexing, and using to_datetime. It takes a while and it would be nice to see a progress bar.

#Import csv files into a Pandas dataframes and convert to Pandas datetime and set to index

eurusd_ask = pd.read_csv('EURUSD_Candlestick_5_m_ASK_01.01.2012-05.08.2017.csv')
eurusd_ask.index = pd.to_datetime(eurusd_ask.pop('Gmt time'))
1
  • 1
    No, it isn't possible.
    – cs95
    Nov 3, 2017 at 3:26

5 Answers 5

233

Find length by getting shape

for index, row in tqdm(df.iterrows(), total=df.shape[0]):
   print("index",index)
   print("row",row)
7
  • 4
    I think this is a better solution than from @sonance207, as this does not convert the iterator to a list, but accesses a given property.
    – guerda
    Nov 13, 2018 at 12:53
  • 1
    this solution doesn't seem to work with df.itertuples() ?
    – Giacomo
    Apr 26, 2019 at 11:27
  • 2
    doesn't iterrows slow down processing by a lot?
    – Matthew
    May 14, 2020 at 17:00
  • 3
    @Giacomo: At least for me and tqdm version '4.42.0' it also works for itertuples. for idx_row, col1, col2, ... in tqdm.tqdm(df.itertuples(), total=len(df)):
    – gebbissimo
    Sep 4, 2020 at 12:01
  • 1
    holds also for tqdm_notebook
    – Ofer Rahat
    Aug 31, 2022 at 12:25
22
with tqdm(total=Df.shape[0]) as pbar:    
    for index, row in Df.iterrows():
        pbar.update(1)
        ...
2
  • Add pbar.close(); at the end, else the loop will keep running (not exit), so will not move to the next code block. May 11, 2021 at 16:31
  • 2
    the with statement does close the object for you
    – meduz
    Jul 7, 2021 at 13:17
3

There is a workaround for tqdm > 4.24. As per https://github.com/tqdm/tqdm#pandas-integration:

from tqdm import tqdm
        
# Register `pandas.progress_apply` and `pandas.Series.map_apply` with `tqdm`
# (can use `tqdm_gui`, `tqdm_notebook`, optional kwargs, etc.)
tqdm.pandas(desc="my bar!")
eurusd_ask['t_stamp'] = eurusd_ask['Gmt time'].progress_apply(pd.Timestamp)
eurusd_ask.set_index(['t_stamp'], inplace=True)
1

You could fill a pandas data frame in line by line by reading the file normally and simply add each new line as a new row to the dataframe, though this would be a fair bit slower than just using Pandas own reading methods.

1

I find it very easy to implement. You only need to add the total argument.

import pandas as pd
df = pd.read_excel(PATH_TO_FILE)


for index, row in tqdm(df.iterrows(),  total=df.shape[0], desc=f'Reading DF'):
        print(row(['df_colum'])

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.