1

I am getting several of these errors from my logger in python.

UnicodeEncodeError: 'charmap' codec can't encode character X in position Y: character maps to undefined

Looking around on stackoverflow, I see that many people have been able to resolve the issue by telling their logger to use 'utf-8' encoding however for me the error still remains.

This is the function that I am using to create my loggers:

def makeLogger(name):
    logger = logging.getLogger(name)
    logger.setLevel(logging.INFO)

    formatter = logging.Formatter('%(asctime)s:%(name)s:%(message)s')

    file_handler = logging.FileHandler(r'C:\Users\lguarro\Documents\Work\SearchEngine_Pure\Log\custom.log', 'a', 'utf-8')
    file_handler.setFormatter(formatter)

    logger.addHandler(file_handler)
    return logger

As you can see I have explicitly told the FileHandler to use utf-8.

Here is an example of how I call my logger from which many errors originate:

self.logger.info("Starting url scrape for company, " + rec["Company"] + " using user agent: " + user_agent)

In particular the error typically comes from the characters inside rec["Company"] as there are a lot of weird company names in my database like lantmännen unibake.

So what exactly am I missing?

2
  • The name of the company is "lantmännen unibake", so the names in your database seem to be broken or at least they were loaded using the wrong encoding.
    – Wups
    Oct 22, 2020 at 8:46
  • Yeah actually I need to see if the source of my database is the cause for that or if it is something I did Oct 22, 2020 at 8:49

1 Answer 1

1

My issue was that I was using Scrapy and I was not configuring the root logger correctly and was only setting up the logger for each of my individual spiders (which behaves correctly)

In essence, I needed to add this to my execution script:

root_logger = logging.getLogger()
root_logger.setLevel(logging.WARNING)
handler = logging.FileHandler(r'C:\Users\lguarro\Documents\Work\SearchEngine_Pure\Log\scrapy.log', 'w', 'utf-8')
handler.setFormatter(logging.Formatter('%(asctime)s:%(levelname)s:%(message)s'))
root_logger.addHandler(handler)

For whatever reason, when I was trying to do the same thing using logging.basicConfig, it told me that "encoding" is not a valid attribute so I opted for this method instead.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.