25

I'm using the logging module in python 3, and want a particular warning message to be displayed only once. The problem is that the check is inside a loop:

def canned_example():
    logger.warning("This function hasn't been created yet")


for i in range(10):
    canned_example()

Is there a flag to set inside the logging module that will denote that this particular warning is only to be displayed once? The alternative is to keep a record of different flags, but I'd like to keep it simple if possible.

Update: Amir Yazdanbakhsh has posted an answer in the comments that allows us to do this for all messages. Ideally, I'd like some per-message flag:

def canned_example():
    logger.warning("This function hasn't been created yet", norepeat=True)


for i in range(10):
    canned_example()
4

3 Answers 3

25

A bit of a hack but much easier is "misusing" functools lru_cache.

from functools import lru_cache
from logging import getLogger

# Keep track of 10 different messages and then warn again
@lru_cache(10)
def warn_once(logger: Logger, msg: str):
    logger.warning(msg)

You can increase the 10 to suppress more if required or set it to None to store suppress everything duplicated.

This answer also full fill the requirement to only suppress only specific duplicate warning, as logger.warning("message") is not filtered at all.

Also this solution is more memory efficient as duplicated logs don't remain in memory multiple times.

1
19

Define a filter which keeps track of what was logged, and attach it to your logger for the duration of the loop. This example will remember each message it sees, and only allow the first occurrence to be logged.

class DuplicateFilter(object):
    def __init__(self):
        self.msgs = set()

    def filter(self, record):
        rv = record.msg not in self.msgs
        self.msgs.add(record.msg)
        return rv

dup_filter = DuplicateFilter()
logger.addFilter(dup_filter)
for i in range(10):
    canned_example()
logger.removeFilter(dup_filter)
2
  • 3
    Just don't forget that any string you log will remain in memory. Aug 12, 2015 at 9:20
  • 1
    True; a quick fix would be to add del dup_filter after removing it from the logger. The garbage collector will free the memory on its next run.
    – chepner
    Aug 12, 2015 at 11:51
6

This is a modified variant of @chepner's answer. I had some issues getting it to work when the thing logged wasn't a string (i.e. an exception) and wanted to make use of context managers to more conveniently add and remove the filter:

class DuplicateFilter:
    """
    Filters away duplicate log messages.
    Modified version of: https://stackoverflow.com/a/31953563/965332
    """

    def __init__(self, logger):
        self.msgs = set()
        self.logger = logger

    def filter(self, record):
        msg = str(record.msg)
        is_duplicate = msg in self.msgs
        if not is_duplicate:
            self.msgs.add(msg)
        return not is_duplicate

    def __enter__(self):
        self.logger.addFilter(self)

    def __exit__(self, exc_type, exc_val, exc_tb):
        self.logger.removeFilter(self)

Usage:

logger = logging.getLogger(__name__)

with DuplicateFilter(logger):
    for _ in range(5):
        logger.info("This will only show up once!")
        logger.error(Exception("So will this!"))   # This didn't work in @chepner's example

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.