3

I have small Python code executing in AWS lambda, but Lambda puts the logs to CloudWatch Logs in a very inconvenient format: enter image description here

I want to send those logs to ELK for visualization. Is there a way to put all Lambda iteration logs to one json file?

4 Answers 4

10

The preloaded LambdaLoggerHandler uses the standard class logging.Formatter. Source: https://www.denialof.services/lambda/

Replace the formatter by a custom class that outputs to JSON. Also the extra dict could contains additionnals values, extra['data'] avoids conflicts with others logrecord properties.

import logging
import json

class FormatterJSON(logging.Formatter):
    def format(self, record):
        record.message = record.getMessage()
        if self.usesTime():
            record.asctime = self.formatTime(record, self.datefmt)
        j = {
            'levelname': record.levelname,
            'time': '%(asctime)s.%(msecs)dZ' % dict(asctime=record.asctime, msecs=record.msecs),
            'aws_request_id': getattr(record, 'aws_request_id', '00000000-0000-0000-0000-000000000000'),
            'message': record.message,
            'module': record.module,
            'extra_data': record.__dict__.get('data', {}),
        }
        return json.dumps(j)


logger = logging.getLogger()
logger.setLevel('INFO')

formatter = FormatterJSON(
    '[%(levelname)s]\t%(asctime)s.%(msecs)dZ\t%(levelno)s\t%(message)s\n',
    '%Y-%m-%dT%H:%M:%S'
)
# Replace the LambdaLoggerHandler formatter :
logger.handlers[0].setFormatter(formatter)


def lambda_handler(event, context):
    my_input = {
        'key1': 'value1',
        'key2': 'value2'
    }
    logger.info('Process Info: %s', 'Hello', extra=dict(data=my_input))

However any stacktraces keeps render as usual.

Then CloudWatch gets this :

{
    "levelname": "INFO",
    "time": "2018-12-13T11:35:24.130Z",
    "aws_request_id": "2e0f7055-fecb-11e8-8376-b77695872964",
    "message": "Process Info: Hello",
    "module": "lambda_function",
    "extra_data": {
        "key1": "value1",
        "key2": "value2"
    }
}
0
2

Try to write your logs in the following way:

import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

def my_logging_handler(event, context):
    logger.error(
        json.dumps(
            {
                'key1': 'value1',
                'key2': 'value2'
            }
        )
    )
    return 'Hello from Lambda!'  
1

You should set your application to log as JSON. This way you can log all info as a single JSON line. This will also make it much easier to analyse downstream in ELK.

For example you can use Watchtower:

import watchtower, logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
logger.addHandler(watchtower.CloudWatchLogHandler())
logger.info("Hi")
logger.info(dict(foo="bar", details={}))

This won't include the START and END lines, but you can always have your application add that information if needed.

0

In short, there is not a way to force all logs to go to one log stream. They all should be in the same log group though. Lambda creates a new log stream whenever a new version is deployed or whenever a new instance of a lambda is spun up. You have a couple of options when it comes to getting this data into ELK.

  1. You could investegate the Cloudwatch Logstash Plugin. I've never used it so I can't vouch for it's ease and/or effectiveness.
  2. You can wire up Cloudwatch Logs as a trigger to another lambda that does processing.
  3. Cloudwatch provides a pretty powerful search. You could periodically query the data from Cloudwatch and post it to your Logstash.
  4. Would be to manually post this data to a Logstash server instead of logging it to Cloudwatch.
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.