34

I have a simple question: How do I change the built-in Python logger's print function to tqdm.write such that logging messages do not interfere with tqdm's progress bars?

1
  • 4
    If you have to stick with tqdm, then @RolKau's answer is the way to go, but tqdm clears and redraws all of the progress bars on each write, so it's easy to overwhelm it if you have a lot of output. If you're not suck with tqdm, Enlighten does what you need out of the box and will hold up better under load since it doesn't rely on redrawing.
    – aviso
    Nov 12, 2018 at 20:35

6 Answers 6

55

tqdm now has a built-in contextmanager for redirecting the logger:

import logging
from tqdm import trange
from tqdm.contrib.logging import logging_redirect_tqdm

LOG = logging.getLogger(__name__)

if __name__ == '__main__':
    logging.basicConfig(level=logging.INFO)
    with logging_redirect_tqdm():
        for i in trange(9):
            if i == 4:
                LOG.info("console logging redirected to `tqdm.write()`")
    # logging restored

Copied from tqdm documentation

4
  • 3
    out of all the above answers. only this worked. Should be marked as accepted.
    – Ani
    Oct 12, 2021 at 19:46
  • 1
    If you've set your own formatter for LOG, then you should do logging_redirect_tqdm(loggers=[LOG]). This will cause tqdm to use LOG's formatter. Jun 16, 2022 at 14:08
  • this answer is more up to date, works and is much more straightforward than the accepted one. Jul 22, 2022 at 15:47
  • It does require a context manager wherever one wants to use tqdm.write though, whereas the other solution can be used to make the change globally.
    – Guillochon
    Jun 27 at 4:22
47

You need a custom logging handler:

import logging
import tqdm

class TqdmLoggingHandler(logging.Handler):
    def __init__(self, level=logging.NOTSET):
        super().__init__(level)

    def emit(self, record):
        try:
            msg = self.format(record)
            tqdm.tqdm.write(msg)
            self.flush()
        except Exception:
            self.handleError(record)  

and then add this to the logging chain:

import time

log = logging.getLogger(__name__)
log.setLevel(logging.INFO)
log.addHandler(TqdmLoggingHandler())
for i in tqdm.tqdm(range(100)):
    if i == 50:
        log.info("Half-way there!")
    time.sleep(0.1)

Edit: Fixed a bug in the call to super TqdmLoggingHandler's init method, that was pointed out by diligent reader @BlaineRogers in the comments. (If anyone wants to read further about this murky area of Python, I recommend https://fuhm.net/super-harmful/)

6
  • I needed something to use logging module for output, solved it like this: stackoverflow.com/a/41224909/638504. Output is like this: 2016-12-19 15:35:06 [INFO ] 16%|#####9 | 768/4928 [07:04<40:50, 1.70it/s]
    – ddofborg
    Dec 21, 2016 at 11:42
  • 1
    it is important to use import tqdm not from tqdm import tqdm else the IO will interrupt the progress bar Feb 13, 2017 at 15:28
  • super(self.__class__, self).__init__(level) will cause a RecursionError if you ever try to instantiate a subclass of TqdmLoggingHandler. super(cls, self).__init__() calls the __init__ method of the class after cls in the method resolution order of self.__class__. Let MyTqdmLoggingHandler be a subclass of TqdmLoggingHandler. Then MyTqdmLoggingHandler() calls TqdmLoggingHandler.__init__(self), which calls super(MyTqdmLoggingHandler, self).__init__(), which calls TqdmLoggingHandler.__init__(self). Feb 7, 2019 at 15:02
  • 1
    @BlaineRogers You are absolutely right! Thank you for pointing this out. I must apparently have written this before I became familiar with the MRO. I will correct the answer, in case anyone wants to use the code.
    – RolKau
    Feb 11, 2019 at 17:51
  • I don't think that reference to the ancient "Super considered harmful" does anyone any good. Please see Raymond Hettinger's talk "Super Considered Super!" for a thorough rebuttal. youtu.be/EiOglTERPEo
    – traal
    Dec 28, 2021 at 20:23
4

Based on RolKau's answer above, simplified:

import logging
from tqdm import tqdm


class TqdmLoggingHandler(logging.StreamHandler):
    """Avoid tqdm progress bar interruption by logger's output to console"""
    # see logging.StreamHandler.eval method:
    # https://github.com/python/cpython/blob/d2e2534751fd675c4d5d3adc208bf4fc984da7bf/Lib/logging/__init__.py#L1082-L1091
    # and tqdm.write method:
    # https://github.com/tqdm/tqdm/blob/f86104a1f30c38e6f80bfd8fb16d5fcde1e7749f/tqdm/std.py#L614-L620

    def emit(self, record):
        try:
            msg = self.format(record)
            tqdm.write(msg, end=self.terminator)
        except RecursionError:
            raise
        except Exception:
            self.handleError(record)

Testing:

import time

log = logging.getLogger(__name__)
log.setLevel(logging.INFO)
log.addHandler(TqdmLoggingHandler())
#   ^-- Assumes this will be the unique handler emitting messages to sys.stdout.
#       If other handlers output to sys.stdout (without tqdm.write),
#       progress bar will be interrupted by those outputs

for i in tqdm(range(20)):
    log.info(f"Looping {i}")
    time.sleep(0.1)

Caveat: if you're working in a Jupyter notebook, progress bar will be interrupted and AFAIK there's no way to avoid it.

2

One easy solution, not so elegant, is convert the tqdm object to string. After that yo can log the message (or handle it as you want). The "format_dict" attribute may be useful as well:

from tqdm import tqdm
import time

#loop with progressbar:
it=tqdm(total=10)
for i in range(10):
    time.sleep(0.1)
    it.update(1)

it.close()
print("\n","--"*10)

# Convert tdqm object last output to sting
str_bar_msg = str(it)
print(str_bar_msg)

# See attributes:
print(it.format_dict)

output:

100%|██████████| 10/10 [00:01<00:00,  8.99it/s]
 --------------------
100%|██████████| 10/10 [00:01<00:00,  8.98it/s]
{'n': 10, 'total': 10, 'elapsed': 1.1145293712615967, 'ncols': None, 'nrows': None, 'prefix': '', 'ascii': False, 'unit': 'it', 'unit_scale': False, 'rate': None, 'bar_format': None, 'postfix': None, 'unit_divisor': 1000, 'initial': 0, 'colour': None}

Best regards

2
  • Use from tqdm.notebook import tqdm if you are in Jupyter Notebook Jan 25, 2022 at 22:06
  • actually use from tqdm.auto import tqdm to auto detect if you are in a jupyter notebook and use the appropriate bar
    – tbrugere
    Sep 1 at 18:40
2

the easiest way is change stream of StreamHandler object e.g.:

import logging
from tqdm import tqdm, trange
import time

log = logging.getLogger(__name__)
log.setLevel(logging.INFO)
handler = logging.StreamHandler()
handler.setStream(tqdm) # <-- important
handler = log.addHandler(handler)

for i in trange(100):
    if i == 50:
        log.info("Half-way there!")
    time.sleep(0.1)
2
  • Not working, a warning appears in log.info when executed directly in COLAB.
    – krenerd
    Sep 15, 2021 at 12:34
  • replace log.info("Half-way there!") by tqdm.write("Half-way there!"). result is the same, wchich mean the problem is module tqdm itself, because .write(...) is official print method while using tqdm.
    – asiloisad
    Sep 16, 2021 at 3:21
1

The new io handler is useful!

class TqdmToLogger(io.StringIO):
    logger = None
    level = None
    buf = ""

    def __init__(self, logger, level=None, mininterval=5):
        super(TqdmToLogger, self).__init__()
        self.logger = logger
        self.level = level or logging.INFO
        self.mininterval = mininterval
        self.last_time = 0

    def write(self, buf):
        self.buf = buf.strip("\r\n\t ")

    def flush(self):
        if len(self.buf) > 0 and time.time() - self.last_time > self.mininterval:
            self.logger.log(self.level, self.buf)
            self.last_time = time.time()```

# before this line, you need to create logger with file handler
tqdm_out = TqdmToLogger(logger)
tbar = tqdm(sample, total=len(sample), file=tqdm_out)

logger.info("Model Inference.")
for it, batch_data in enumerate(tbar):
   pass

```
1
  • As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.
    – Community Bot
    Nov 4, 2022 at 15:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.