22

I have the logging module MemoryHandler set up to queue debug and error messages for the SMTPHandler target. What I want is for an email to be sent when the process errors that contains all debug statements up to that point (one per line). What I get instead is a separate email for every debug message.

This seems like it should be trivial, and part of the logging package, but I can't find anything about it, no examples, nothing on Google.

log = logging.getLogger()
log.setLevel(logging.DEBUG)
debug_format = logging.Formatter("%(levelname)s at %(asctime)s in %(filename)s (line %(lineno)d):: %(message)s")

# write errors to email
error_mail_subject = "ERROR: Script error in %s on %s" % (sys.argv[0], os.uname()[1])
error_mail_handler = logging.handlers.SMTPHandler(SMTP_HOST, 'errors@'+os.uname()[1], [LOG_EMAIL], error_mail_subject)
error_mail_handler.setLevel(logging.ERROR)
#error_mail_handler.setLevel(logging.DEBUG)
error_mail_handler.setFormatter(debug_format)

# buffer debug messages so they can be sent with error emails
memory_handler = logging.handlers.MemoryHandler(1024*10, logging.ERROR, error_mail_handler)
memory_handler.setLevel(logging.DEBUG)

# attach handlers
log.addHandler(memory_handler)
log.addHandler(error_mail_handler)

Related to this:

Do I need to add the error_mail_handler to the logger explicitly if it is a target of memory_handler anyway? Should error_mail_handler be set to DEBUG or ERROR target? Does it even need a target when it is being fed from memory_handler?

Would love to see some working code from anyone who has solved this problem.

6 Answers 6

35

You might want to use or adapt the BufferingSMTPHandler which is in this test script.

In general, you don't need to add a handler to a logger if it's the target of a MemoryHandler handler which has been added to a logger. If you set the level of a handler, that will affect what the handler actually processes - it won't process anything which is less severe than its level setting.

10
  • 2
    why isn't that class part of the standard library? it seems much more useful to send all messages at once, especially for batch jobs (as opposed to long-running servers)
    – anarcat
    Jan 6, 2016 at 23:33
  • 3
    @anarcat: Just because you have to draw the line somewhere. Jan 7, 2016 at 21:19
  • 4
    "why is the line drawn there?" "because you have to draw it somewhere" really, my point is that, exactly, i find the SMTPHandler basically useless and the BufferingSMTPHandler essential for SMTP support in logging. the line was drawn in the wrong place.
    – anarcat
    Jan 11, 2016 at 14:06
  • 1
    @anarcat: Actually the line may have been drawn too far already. SMTP can be notoriously laggy, so you are actually better off using something like a QueueHandler and doing the emailing in a separate thread or process. Certainly, in e.g. a web application, logging to SMTP could cause the response to be sluggish or even appear to hang while the SMTP connection times out. Jan 11, 2016 at 19:52
  • 1
    @Alf47 it should be, though I would recommend using the dictConfig() API rather than fileConfig() for better configuration coverage of the logging APIs. Apr 11, 2018 at 4:45
7

Instead of buffering for email, consider posting unbuffered to a message stream on a messaging app, e.g. on Matrix, Discord, Slack, etc. Having said that, I wrote my own beastly thread-safe implementation of BufferingSMTPHandler (backup link) which sends emails from a separate thread. The primary goal is to not block the main thread.

As written, it uses two queues - this seemed necessary in order to implement some useful class-level parameters that are defined in the "Configurable parameters" section of the code. Although you can use the code as-is, it's probably better if you study and use it to write your own class.

Issues:

  • Some class-level parameters can perhaps be instance-level instead.
  • Either threading.Timer or the signal module could perhaps be used to avoid loops that run forever.
3

If you are using django - here is simple buffering handler, which will use standard django email methods:

import logging

from django.conf import settings
from django.core.mail import EmailMessage


class DjangoBufferingSMTPHandler(logging.handlers.BufferingHandler):
    def __init__(self, capacity, toaddrs=None, subject=None):
        logging.handlers.BufferingHandler.__init__(self, capacity)

        if toaddrs:
            self.toaddrs = toaddrs
        else:
            # Send messages to site administrators by default
            self.toaddrs = zip(*settings.ADMINS)[-1]

        if subject:
            self.subject = subject
        else:
            self.subject = 'logging'

    def flush(self):
        if len(self.buffer) == 0:
            return

        try:
            msg = "\r\n".join(map(self.format, self.buffer))
            emsg = EmailMessage(self.subject, msg, to=self.toaddrs)
            emsg.send()
        except Exception:
            # handleError() will print exception info to stderr if logging.raiseExceptions is True
            self.handleError(record=None)
        self.buffer = []

In django settings.py you will need to configure email and logging like this:

EMAIL_USE_TLS = True
EMAIL_PORT = 25  
EMAIL_HOST = ''  # example: 'smtp.yandex.ru'
EMAIL_HOST_USER = ''  # example: '[email protected]'
EMAIL_HOST_PASSWORD = ''
DEFAULT_FROM_EMAIL = EMAIL_HOST_USER
SERVER_EMAIL = EMAIL_HOST_USER

LOGGING = {
    'handlers': {
        ...
        'mail_buffer': {
            'level': 'WARN',
            'capacity': 9999,
            'class': 'utils.logging.DjangoBufferingSMTPHandler',
            # optional: 
            # 'toaddrs': '[email protected]'
            # 'subject': 'log messages'
        }
    },
    ...
}
2

For this purpose I use the BufferingSMTPHandler suggested by Vinay Sajip with one minor tweak: I set the buffer length to something really big (say 5000 log records) and manualy call the flush method of the handler every some seconds and after checking for internet conectivity.

# init
log_handler1 = BufferingSMTPHandler(
    'smtp.host.lala', "[email protected]", ['[email protected]'], 'Log event(s)',5000)
...
logger.addHandler(log_handler1)
...

# main code
...
if internet_connection_ok and seconds_since_last_flush>60:
    log_handler1.flush() # send buffered log records (if any)
1
  • 3
    This only would work if the handler is still in scope when it is flushed. Otherwise, you would have to retrieve the handler from the logger object in order to call flush. May 17, 2012 at 15:25
2

Updated Vinay Sajip's answer for python3.

import logging
from logging.handlers import BufferingHandler

class BufferingSMTPHandler(BufferingHandler):
    def __init__(self, mailhost, fromaddr, toaddrs, subject, capacity):
        logging.handlers.BufferingHandler.__init__(self, capacity)
        self.mailhost = mailhost
        self.mailport = None
        self.fromaddr = fromaddr
        self.toaddrs = toaddrs
        self.subject = subject
        self.setFormatter(logging.Formatter("%(asctime)s %(levelname)-5s %(message)s"))

    def flush(self):
        if len(self.buffer) > 0:
            try:
                import smtplib
                port = self.mailport
                if not port:
                    port = smtplib.SMTP_PORT
                smtp = smtplib.SMTP(self.mailhost, port)
                msg = '''From: {}\r\nTo: {}\r\nSubject: {}\r\n\r\n'''.format(
                            self.fromaddr,
                            ",".join(self.toaddrs),
                            self.subject
                            )
                for record in self.buffer:
                    s = self.format(record)
                    print (s)
                    msg = msg + s + "\r\n"
                smtp.sendmail(self.fromaddr, self.toaddrs, msg)
                smtp.quit()
            except:
                self.handleError(None)  # no particular record
            self.buffer = []

#update for @Anant
if __name__ == '__main__'
    buff_smtp_handler=BufferingSMTPHandler(...your args)
    buff_smtp_handler.setLevel(logging.ERROR)
    handlers=[buff_smtp_handler]
    logging.basicConfig(handlers=handlers)
4
  • Hi @10mjg, could you help me to understand how to send an Email only for the logs with severity Error or higher.
    – Anant
    Feb 19, 2021 at 17:33
  • @Anant I am not sure, but I tried looking up a solution for you. If that is all you want, it might be as simple as just logger.setLevel(logging.ERROR) ... or using a MemoryHandler instead of a BufferingHandler , and setting a flush_level of logging.ERROR for your MemoryHandler ... docs.python.org/3/library/…
    – 10mjg
    Feb 19, 2021 at 21:12
  • @Anant (similar to the logger.setLevel solution, you could also call .setLevel(logging.ERROR) on just your BufferingSMTPHAndler handler instance, if you want to leave your other handlers alone. I added a little example to the code above.
    – 10mjg
    Feb 19, 2021 at 23:40
  • your suggestion really worked and helped me to configure as expected.
    – Anant
    Feb 20, 2021 at 22:16
1

I think the point about the SMTP logger is that it is meant to send out a significant log message functioning as some kind of alert if sent to a human recipient or else to be further processed by an automated recipient.

If a collection of log messages is to be sent by email then that constitutes a report being sent at the end of execution of a task and writing that log to a file and then emailing the file would seem to be a reasonable solution.

I took a look at the basic FileHandler log handler and how to build a mechanism to write to a temp file then attach that temp file when the script exits.

I found the "atexit" module that allows for a method to be registered that will be executed against an object when the script is exiting.

import logging
import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.mime.base import MIMEBase
import os
from email import encoders
import uuid
# atexit allows for a method to be set to handle an object when the script             exits
import atexit
filename = uuid.uuid4().hex
class MailLogger:

def __init__(self, filePath, smtpDict):
    self.filePath = filePath
    self.smtpDict = smtpDict
    # Generate random file name
    filename = '%s.txt' % ( uuid.uuid4().hex )
    # Create full filename
    filename = '%s/%s' % (filePath,filename)
    self.filename = filename
    self.fileLogger = logging.getLogger('mailedLog')
    self.fileLogger.setLevel(logging.INFO)
    self.fileHandler = logging.FileHandler(filename)
    self.fileHandler.setLevel(logging.INFO)
    formatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
    self.fileHandler.setFormatter(formatter)
    self.fileLogger.addHandler(self.fileHandler)
    atexit.register(self.mailOut)

def mailOut(self):
    '''
    Script is exiting so time to mail out the log file

    "emailSettings":    {
                        "smtpServer"    :     "smtp.dom.com",
                        "smtpPort"        :    25,
                        "sender"        :    "[email protected]>",
                        "recipients"    :    [
                                                "[email protected]"
                                            ],
                        "subject"        :    "Email Subject"
},
    '''
    # Close the file handler
    smtpDict = self.smtpDict
    self.fileHandler.close()
    msg = MIMEMultipart('alternative')
    s = smtplib.SMTP(smtpDict["smtpServer"], smtpDict["smtpPort"] )        
    msg['Subject'] = smtpDict["subject"]
    msg['From'] = smtpDict["sender"]
    msg['To'] = ','.join(smtpDict["recipients"])
    body = 'See attached report file'    
    content = MIMEText(body, 'plain')
    msg.attach(content)
    attachment = MIMEBase('application', 'octet-stream')
    attachment.set_payload(open(self.filename, 'rb').read())
    encoders.encode_base64(attachment)
    attachment.add_header('Content-Disposition', 'attachment; filename="%s"' % os.path.basename(self.filename))
    msg.attach(attachment)
    s.send_message(msg)
    s.quit()

My basic test script is:

from EmailLogRpt import MailLogger
import time
smtpDict = {
                        "smtpServer"    :     "smtp.dom.com",
                        "smtpPort"        :    25,
                        "sender"        :    "[email protected]",
                        "recipients"    :    [
                                                "[email protected]>"
                                            ],
                        "subject"        :    "Email Subject"
}
myMailLogger = MailLogger("/home/ed/tmp",smtpDict).fileLogger
myMailLogger.info("test msg 1")
time.sleep(5)
myMailLogger.info("test msg 2")

Hope this helps somebody.

0

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.