40

Is there an easy way with python's logging module to send messages with a DEBUG or INFO level and the one with a higher level to different streams?

Is it a good idea anyway?

1

9 Answers 9

42
import logging
import sys

class LessThanFilter(logging.Filter):
    def __init__(self, exclusive_maximum, name=""):
        super(LessThanFilter, self).__init__(name)
        self.max_level = exclusive_maximum

    def filter(self, record):
        #non-zero return means we log this message
        return 1 if record.levelno < self.max_level else 0

#Get the root logger
logger = logging.getLogger()
#Have to set the root logger level, it defaults to logging.WARNING
logger.setLevel(logging.NOTSET)

logging_handler_out = logging.StreamHandler(sys.stdout)
logging_handler_out.setLevel(logging.DEBUG)
logging_handler_out.addFilter(LessThanFilter(logging.WARNING))
logger.addHandler(logging_handler_out)

logging_handler_err = logging.StreamHandler(sys.stderr)
logging_handler_err.setLevel(logging.WARNING)
logger.addHandler(logging_handler_err)

#demonstrate the logging levels
logger.debug('DEBUG')
logger.info('INFO')
logger.warning('WARNING')
logger.error('ERROR')
logger.critical('CRITICAL')

Implementation aside, I do think it is a good idea to use the logging facilities in python to output to the terminal, in particular because you can add another handler to additionally log to a file. If you set stdout to be INFO instead of DEBUG, you can even include additional DEBUG information that the user wouldn't standardly see in the log file.

4
  • Where do you get the LessThanFilter class from. Can't find it in the Python3 libs.
    – buhtz
    Jul 31, 2015 at 8:16
  • 5
    I wrote it (and provided the implementation), it's not in the standard libraries.
    – Zoey Greer
    Aug 25, 2015 at 15:02
  • Thanks so much. This is the first place I've found anyone mention the default logging level of Warning. I lost a day to this. Some semblance to santiy is restored. THANK YOU!
    – penderi
    Dec 7, 2017 at 15:55
  • 2
    It's perhaps worth noting that you can simplify the filter method - i.e. directly write return record.levelno < self.max_level instead of return 1 if record.levelno < self.max_level else 0. Also, with Python 3.2 or newer you can simply supply that filter as lambda. Sep 26, 2020 at 16:23
6

Yes. You must define multiple handlers for your logging.

http://docs.python.org/library/logging.html#logging-to-multiple-destinations

http://docs.python.org/library/logging.handlers.html#module-logging.handlers

2
  • I would like only debug and info message in stdout and no warning or error messages.
    – Dinoboff
    Feb 21, 2010 at 21:50
  • documentation is updated, correct link is at docs.python.org/library/…
    – gcb
    Feb 17, 2012 at 6:13
6

Just for your convenience adding everything together with the formatter in one package:

# shared formatter, but you can use separate ones:
FORMAT = '%(asctime)s - %(name)s - %(levelname)s - %(threadName)s - %(message)s'
formatter = logging.Formatter(FORMAT)

# single app logger:
log = logging.getLogger(__name__)
log.setLevel(logging.INFO)

# 2 handlers for the same logger:
h1 = logging.StreamHandler(sys.stdout)
h1.setLevel(logging.DEBUG)
# filter out everything that is above INFO level (WARN, ERROR, ...)
h1.addFilter(lambda record: record.levelno <= logging.INFO)
h1.setFormatter(formatter)
log.addHandler(h1)

h2 = logging.StreamHandler(sys.stderr)
# take only warnings and error logs
h2.setLevel(logging.WARNING)
h2.setFormatter(formatter)
log.addHandler(h2)

# profit:
log.info(...)
log.debug(...)

My use case was to redirect stdout to a datafile while seeing errors on the screen during processing.

5

I had the same problem and wrote a custom logging handler called SplitStreamHandler:

import sys
import logging

class SplitStreamHandler(logging.Handler):
    def __init__(self):
        logging.Handler.__init__(self)

    def emit(self, record):
        # mostly copy-paste from logging.StreamHandler
        try:
            msg = self.format(record)
            if record.levelno < logging.WARNING:
                stream = sys.stdout
            else:
                stream = sys.stderr
            fs = "%s\n"

            try:
                if (isinstance(msg, unicode) and
                    getattr(stream, 'encoding', None)):
                    ufs = fs.decode(stream.encoding)
                    try:
                        stream.write(ufs % msg)
                    except UnicodeEncodeError:
                        stream.write((ufs % msg).encode(stream.encoding))
                else:
                    stream.write(fs % msg)
            except UnicodeError:
                stream.write(fs % msg.encode("UTF-8"))

            stream.flush()
        except (KeyboardInterrupt, SystemExit):
            raise
        except:
            self.handleError(record)
1
  • I know it is slightly off-topic, but can someone explain the encode/decode try/catch logic here in terms of character encoding?
    – Chris
    Aug 20, 2013 at 15:48
2

right from the updated docs, it cover this case pretty well now.

http://docs.python.org/howto/logging.html#logging-advanced-tutorial

import sys # Add this.
import logging

# create logger
logger = logging.getLogger('simple_example')
logger.setLevel(logging.DEBUG)

# create console handler and set level to debug
ch = logging.StreamHandler( sys.__stdout__ ) # Add this
ch.setLevel(logging.DEBUG)

# create formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# add formatter to ch
ch.setFormatter(formatter)

# add ch to logger
logger.addHandler(ch)

# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')

i've mentioned on comments the two changes required from the example to make the output go to stdout. you may also use filters to redirect depending on the level.

more information to understand the changes is at http://docs.python.org/library/logging.handlers.html#module-logging.handlers

3
  • 5
    This only redirects to stdout, but does not filter depending on the level to stderr.
    – guettli
    Mar 1, 2012 at 13:43
  • @guettli "you may also use filters to redirect depending on the level"
    – gcb
    Mar 9, 2012 at 4:03
  • 4
    @gcb please could you show (!) how to add a suitable filter which does the job? Your suggestion means that "logger" still outputs to stderr, but "ch" outputs the same thing to stdout. Given that ch is dependent on logger, how could you set a filter on logger without it then applying also to ch? Confused. Mar 8, 2014 at 11:31
2

You will have to associated a handler to each stream you want logged statements to be sent to. Python has 5 logging levels - CRITICAL, ERROR, WARNING, INFO, and DEBUG.

Level Numeric Value Stream
CRITICAL 50 STDERROR
ERROR 40 STDERROR
WARNING 30 STDOUT
INFO 20 STDOUT
DEBUG 10 STDOUT
NOTSET 0

Let's assume you want that Critical and Error logs should go to STDERROR, while others should go to STDOUT - as shown in the table above.

Here are the steps you should follow.

Step1 Initialize the logging module

import logging, sys
log = logging.getLogger()

Step 2 Create two streams, one attached to STDERROR, other to STDOUT

hStErr = logging.StreamHandler(sys.stderr)
hStOut = logging.StreamHandler(sys.stdout)

Step 3 Set log level for each handler. This tells handler to only log statements which have log level equal to or higher than the level we set. eg. if we set this to WARNING, the handler will only handle logging statements for WARNING, ERROR, and CRITICAL levels. Also make sure that the log level is set in handler only, and not with the log object, so that there is no conflict

hStErr.setLevel('ERROR')
hStOut.setLevel('DEBUG')
log.setLevel('NOTSET')

Now here comes a catch. While hStErr will only output logging for ERROR and CRITICAL, hStOut will output for all 5 levels. Remember that setLevel only tells the minimum logging level which should be handled, so all levels which are greater will also be handled. To limit hStOut to not handle ERROR and CRITICAL, we use a filter.

Step 4 Specify a filter so that ERROR, and CRITICAL aren't handled by hStOut

hStOut.addFilter(lambda x : x.levelno < logging.ERROR)

Step 5 Add these handlers to logger

log.addHandler(hStErr)
log.addHandler(hStOut)

Here are all the pieces together.

import logging, sys
log = logging.getLogger()
hStErr = logging.StreamHandler(sys.stderr)
hStOut = logging.StreamHandler(sys.stdout)

hStErr.setLevel('ERROR')
hStOut.setLevel('DEBUG')
log.setLevel('NOTSET')

hStOut.addFilter(lambda x : x.levelno < logging.ERROR)

log.addHandler(hStErr)
log.addHandler(hStOut)

log.error("error log")
log.info("info log")

Output when we run this script.

error log
info log

Pycharm IDE colors output from std error red. The following image shows that the error log statement above was sent to stderr.

output in pycharm IDE console

If we comment the addFilter line in above script, we will see the following output.

error log
error log
info log

output in pycharm IDE console

Note that without filter hStOut will output logging statements from both INFO and ERROR, while for INFO hStErr outputs nothing, and hStOut outputs a single statement - info log

1

Here's the compact solution I use (tested with Python 3.10):

import logging
import sys

root_logger = logging.getLogger()

# configure default StreamHandler to stderr
logging.basicConfig(
    format="%(asctime)s | %(levelname)-8s | %(filename)s:%(funcName)s(%(lineno)d) | %(message)s",
    level=logging.INFO,  # overall log level; DEBUG or INFO make most sense
)
stderr_handler = root_logger.handlers[0]  # get default handler
stderr_handler.setLevel(logging.WARNING)  # only print WARNING+

# add another StreamHandler to stdout which only emits below WARNING
stdout_handler = logging.StreamHandler(sys.stdout)
stdout_handler.addFilter(lambda rec: rec.levelno < logging.WARNING)
stdout_handler.setFormatter(stderr_handler.formatter)  # reuse the stderr formatter
root_logger.addHandler(stdout_handler)
1
  • 1
    Beautiful—I love how few wheels you're reinventing, including taking advantage of the basic configuration logging to stderr by default. We can even leave out the format argument and setFormatter, for it to output the messages without bells and whistles. For simple command line utilities, that's normally want when I've finished developing them. Nov 30 at 21:53
-4

Not necessarily a good idea (it could be confusing to see info and debug messages mixed in with normal output!), but feasible, since you can have multiple handler objects and a custom filter for each of them, in order to pick and choose which log records each handler gets to handle.

3
  • 1
    please could you show (!) how to add a suitable filter which does the job? gcb's suggestion below means that "logger" still outputs to stderr, but "ch" outputs the same thing to stdout. Given that ch is dependent on logger, how could you set a filter on logger without it then applying also to ch? Confused. Mar 8, 2014 at 11:33
  • There are platforms where the separation is required in order to improve operation tasks. Cloud Foundry is one such example. There, having all the DEBUG or INFO separated from the WARNING and ERROR logs is making troubleshooting via logs easier. Jun 19, 2020 at 15:52
  • Outputting errors to stderr and other messages to stdout has been common practice for about as long as I've been alive, and I've never seen anyone get confused by it. It makes it easier for devops to receive pages when programs produce errors. Nov 30 at 18:27
-7
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import logging
import sys

class LessThenFilter(logging.Filter):
    def __init__(self, level):
        self._level = level
        logging.Filter.__init__(self)

    def filter(self, rec):
        return rec.levelno < self._level

log = logging.getLogger()
log.setLevel(logging.NOTSET)

sh_out = logging.StreamHandler(stream=sys.stdout)
sh_out.setLevel(logging.DEBUG)
sh_out.setFormatter(logging.Formatter('%(levelname)s: %(message)s'))
sh_out.addFilter(LessThenFilter(logging.WARNING))
log.addHandler(sh_out)

sh_err = logging.StreamHandler(stream=sys.stderr)
sh_err.setLevel(logging.WARNING)
sh_err.setFormatter(logging.Formatter('%(levelname)s: %(message)s'))
log.addHandler(sh_err)

logging.critical('x')
logging.error('x')
logging.warning('x')
logging.info('x')
logging.debug('x')
1
  • 2
    Is this different from the answer above? What/why is the purpose/difference? Jun 26, 2018 at 19:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.