112

I can view the log using the following command.

aws logs get-log-events --log-group-name groupName --log-stream-name streamName --limit 100

what is the command to get feature like tail -f so that i can see the log real time

4
  • You can use this command line utility for Python pypi.org/project/qaws
    – Jurass
    May 25, 2020 at 5:57
  • If you want the results within memory rather than console output, i.e. boto3 instead of awscli, gist.github.com/alanyee/601f995bfd6acfd4c3c16ee7e9115ab5
    – Flair
    Jan 7, 2021 at 23:19
  • 9
    TL/DR; Since mid-2019, newer versions of aws-cli include: aws logs tail $group_name (and it supports handy options like --since "1h" and --follow (like tail -f) and --filter-pattern "blah" (like grep).)
    – MarkHu
    Jan 27, 2021 at 1:16
  • @MarkHu please post such comments as answers. Feb 22 at 18:31

15 Answers 15

132

Note that tailing an aws log is now a supported feature of the official awscli, albeit only in awscli v2, which is not released yet. Tailing and following the logs (like tail -f) can now be accomplished by something like:

aws logs tail $group_name --follow

To install the v2 version, see the instructions on this page. It was implemented in this PR. To see it demonstrated at the last re:Invent conference, see this video.

In addition to tailing the logs, it allows viewing the logs back to a specified time using the --since parameter, which can take an absolute or relative time

aws logs tail $group_name --since 5d

To keep the v1 and v2 versions of awscli separate, I installed awscli v2 into a separate python virtual environment and activate it only when I need to use awscli v2.

7
  • 2
    Used this link for aws v2 : docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html , Try installing it using '-i' and '-b' option to provide custom install and binary file locations. Dec 5, 2019 at 7:52
  • 2
    Backported as a v1 plugin, for those unable to use v2 : pypi.org/project/awscli-plugin-logs-tail Oct 6, 2020 at 2:24
  • If you want to look at the current state of the code, it is present in the v2 branch under github.com/aws/aws-cli/tree/v2/awscli/customizations/logs
    – Flair
    Jan 7, 2021 at 0:48
  • saw and awslogs work out of the box but aws logs results in An error occurred (ThrottlingException) when calling the FilterLogEvents operation (reached max retries: 2): Rate exceeded... NICE!
    – luk2302
    Jun 22, 2021 at 12:39
  • 1
    If you're on Windows and you're using Git Bash, then you need to prefix the command with MSYS_NO_PATHCONV=1 For example: MSYS_NO_PATHCONV=1 aws logs tail /aws/lambda/my_group_name --since 1h --follow
    – Adam
    Sep 10, 2022 at 10:11
124

I was really disappointed with awslogs and cwtail so I made my own tool called Saw that efficiently streams CloudWatch logs to the console (and colorizes the JSON output):

You can install it on MacOS with:

brew tap TylerBrock/saw
brew install saw

It has a bunch of nice features like the ability to automatically expand (indent) the JSON output (try running the tool with --expand):

saw watch my_log_group --expand

Got a Lambda you want to see error logs for? No Problem:

saw watch /aws/lambda/my_func --filter error 

Saw is great because the output is easily readable and you can stream logs from entire log group, not just a single stream in the group. Filtering and watching streams with a certain prefix is also just as easy!

7
  • 2
    Amazing. I could not get the docker setup to work. I believe there is an issue with the way the aws configuration is mapped in. The release version worked like a charm and i am the biggest fan!
    – pandaadb
    Nov 1, 2018 at 17:09
  • What is the latest version of saw? I wanted to update my saw installation but it told me: brew install saw Updating Homebrew... ==> Installing saw from tylerbrock/saw Error: saw 1.2.1 is already installed To install 0.2.2, first run brew unlink saw. Hmm, do I have the latest or not?
    – LilMoke
    May 31, 2019 at 17:37
  • @Tyler Brock Do you have an option to filter out to exclude a string in the Saw tool?
    – DJo
    Aug 26, 2019 at 6:15
  • @AliGajani yeah there isn't much i can do about that, cloudwatch is slow Jun 7, 2020 at 16:58
  • Thanks Tyler, by the way, how do I filter strings with a colon in them like import:hu
    – Ali Gajani
    Jun 8, 2020 at 1:20
59

Have a look at awslogs.

If you happen to be working with Lambda/API Gateway specifically, have a look at apilogs.

8
  • 2
    awslogs package is amazing. Solved a problem I had this morning where a business team member just wanted to "grep the logs to find stuff". Definitely the way to go for simple solutions.
    – Adam Link
    Dec 8, 2017 at 18:57
  • 1
    Minus one point for shameless self promotion. Looks cool though!
    – RyanG
    Jun 20, 2018 at 19:08
  • 1
    @TylerBrock a lot of console apps are built on Python, including the official aws-cli, so your point is not 100% valid. Reasoning aside, great tool you've built!
    – Hnatt
    Sep 17, 2018 at 16:33
  • 2
    but... python comes pre-installed in a lot of unix based os :)
    – RicardoE
    Dec 18, 2018 at 21:47
  • Awslogs are not bad but this one looks better to me: pypi.org/project/qaws
    – Jurass
    May 25, 2020 at 5:57
13

I've just discovered cwtail and it works well (to watch a lambda function's CloudWatch logs).

To install:

npm install -g cwtail

To list log groups:

cwtail -l

Then, once you've picked which log group to 'tail':

cwtail -f /aws/lambda/ExampleFunction
2
  • 1
    And if you wanted to format the json coming back from cwtail, this might work. Running in a bash shell. $ echo '{"j":[' cwtail -e /aws/connect/carbonated | sed 's/$/,/' | sed '$ s/.$//' ']}' | python -m json.tool
    – Chai Ang
    Sep 5, 2019 at 23:37
  • 1
    Missed a very important detail -e, else it will show the log as one big paragraph, completely unreadable. Use cwtail -e group_name. Aug 6, 2022 at 8:49
10

AWS allows you to tail the logs now. Exactly like tail -f. use the following command

aws logs tail <log group name> --follow

E.g. if you are using ElasticBeanStalk with app name myapp-prd and want to tail web1.log it would be

aws logs tail /aws/elasticbeanstalk/myapp-prd/var/log/web-1.log --follow
8

Because CloudWatch logs can be delayed (i.e. not "realtime" by precise definition) you parse the previous events for the last timestamp and start the next iteration there. This script uses aws logs get-log-events for which you must specify a valid stream_name.

#!/bin/bash
    
group_name='<log-group-name>'
stream_name='<log-stream-name>'
start_seconds_ago=300

start_time=$(( ( $(date -u +"%s") - $start_seconds_ago ) * 1000 ))
while [[ -n "$start_time" ]]; do
    loglines=$(aws logs get-log-events --log-group-name "$group_name" --log-stream-name "$stream_name" --start-time $start_time --output text)
    [ $? -ne 0 ] && break
      next_start_time=$( sed -nE 's/^EVENTS.([[:digit:]]+).+$/\1/ p' <<< "$loglines" | tail -n1 )
    [ -n "$next_start_time" ] && start_time=$(( $next_start_time + 1 ))
    echo "$loglines"
    sleep 15
done

Or if you want to tail an entire log group, this script uses aws logs filter-log-events without a stream name:

#!/bin/bash

group_name='<log-group-name>'
start_seconds_ago=300
  
start_time=$(( ( $(date -u +"%s") - $start_seconds_ago ) * 1000 ))
while [[ -n "$start_time" ]]; do
    loglines=$(aws logs filter-log-events --log-group-name "$group_name" --interleaved --start-time $start_time --output text)
    [ $? -ne 0 ] && break
    next_start_time=$( sed -nE 's/^EVENTS.([^[:blank:]]+).([[:digit:]]+).+$/\2/ p' <<< "$loglines" | tail -n1 )
    [ -n "$next_start_time" ] && start_time=$(( $next_start_time + 1 ))
    echo "$loglines"
    sleep 15
done

I've also put up the scripts that I use as GitHub gists: https://gist.github.com/tekwiz/964a3a8d2d84ff4c8b5288d9a703fbce.

Warning: the above code & scripts are written for my macOS system which is customized (bastardized??) with Homebrew and GNU coreutils, so some command options may need to be tweaked for your system. Edits are welcome :)

1
  • Nice one. Can you show me how to get the log streams list . I want to build a solution to filter log stream by its name for k8s containers. But aws is not returning anything when I query the logstreams api ? Any idea why? Jun 29, 2021 at 18:14
8

After checking many options and testing some custom tools aws logs tail worked the best for me.

Here there is a simple example command:

aws logs tail <GROUP_NAME> --follow

and here is the official doc which was very useful:

https://awscli.amazonaws.com/v2/documentation/api/latest/reference/logs/tail.html

7

To tail CloudWatch Logs effectively I created a tool called cw.

It's super easy to install (it supports brew, snap and scoop), fast (it targets the specific hardware architecture, no intermediate runtime) and it has a set of features that make life easier.

Your example with cw would be:

cw tail -f groupName:streamName
4

I created a JetBrains plugin called awstail to do this :)

2
  • Why did you remove it from JB Marketplace?
    – Kirill
    Aug 11, 2021 at 6:34
  • 1
    @Kirill AWS has their own plugin for which has this exact feature. Therefore I am not planning to support and fix bugs on my tool. aws.amazon.com/intellij
    – godzsa
    Aug 16, 2021 at 13:02
3

You can use awslogs, a python package to tail aws logwatch logs.

Install it with

pip install awslogs

List all the groups with

awslogs groups        

Then select a stream and watch it with

awslogs get staging-cluster --watch

You can also filter logs with matching patterns.

# tail logs of a cluster
awslogs get staging-cluster --watch

# tail logs of a lambda function
awslogs get /aws/lambda/some-service --watch

# print all logs containg "error"
awslogs get staging-cluster --watch --filter-pattern="error"

# print all logs *not* containg "error"
awslogs get staging-cluster --watch --filter-pattern="-error"

See project readme for more information on using awslogs.

1
  • This Python utility uses better time ranges and provides CloudWatch Insights queries: pypi.org/project/qaws
    – Jurass
    May 25, 2020 at 5:59
2

This is not currently a feature of the CLI since it just exposes the HTTP API for CloudWatch Logs. You could fairly trivially emulate the functionality with a shell script:

#! /bin/sh

end_time=$(($(date +"%s") * 1000))
aws logs get-log-events --log-group-name groupName --log-stream-name streamName --end-time $end_time

while :
do
    start_time=$end_time
    end_time=$(($(date +"%s") * 1000))
    aws logs get-log-events --log-group-name groupName --log-stream-name streamName --start-time $start_time --end-time $end_time
    sleep 1
done

Disclaimer: this won't work on Windows, and there may be a better way to get the time in milliseconds.

2
  • Thank you for your answer. It helped but this work for me since server time and my local machine time is different. i tried changing my local time but still it wont sync properly.
    – LynAs
    Dec 2, 2015 at 6:26
  • CloudWatch logs are stored with the timezone, and the CloudWatch API uses of UTC for timestamps (UNIX epoch in milliseconds), so this will only get events in the past if you're system uses a timezone east of GMT (and nothing if you're west of GMT). Also, CloudWatch logs are almost always delayed by a couple seconds, so the likelihood this will return events even if you correct the time to UTC is pretty low (in my experience). Jun 27, 2016 at 12:32
2

AWS launched a console page specifically to support near real time tailing across multiple logGroups and across linked accounts checkout Cloud watch Live tail

2
  • Super useful feature. But can I use an exclude on the filter as opposed to an include? Aug 15 at 23:19
  • 1
    Yes you can! Place a hyphen/minus symbol ("-") before the terms that you want to exclude do -INFO to exclude INFO logs.. You can also use a combination of include and exclude, for example ERROR -ARGUMENTS to include ERROR but exclude ARGUMENTS. AWS docs have more examples.
    – Nishanth
    Aug 17 at 15:06
2

Amazon recently announced Live tail option to see the real time streaming experience to monitor logs as they occur. Live tail is focused on real-time monitoring and troubleshooting, while log groups are more about log storage, organization, and analysis over a longer period.

Navigate to AWS CloudWatch and select Live tail option under logs.

AWS CloudWatch Livetail

1
0

The aws cli does not provide a live tail -f option.

Those other tools mentioned above do provide a tailing feature, however, I tried all these tools, awslogs, cwtail and found them frustrating. They were slow to download events, often unreliable and not helpful in displaying JSON log data and were primitive with query options.

I wanted an extremely fast, simple log viewer that would allow me to instantly and easily see application errors and status. The CloudWatch logs viewer is slow and CloudWatch Insights can take > 1m for some pretty basic queries.

So I created SenseLogs, a free AWS CloudWatch Logs viewer that runs entirely in your browser. There is no server-side services required. SenseLogs transparently downloads log data and stores events in your browser application cache for immediate viewing, smooth infinite scrolling and full text queries. SenseLogs has live tail with infinite back scrolling. See https://github.com/sensedeep/senselogs/blob/master/README.md for details.

1
0

Here's a bash script that you can use. The script requires the AWS CLI and jq.

#!/bin/bash

# Bail out if anything fails, or if we do not have the required variables set
set -o errexit -o nounset

LOG_GROUP_NAME=$1
LOG_BEGIN=$(date --date "${2-now}" +%s)
LOG_END=$(date --date "${3-2 minutes}" +%s)
LOG_INTERVAL=5
LOG_EVENTIDS='[]'

while (( $(date +%s) < $LOG_END + $LOG_INTERVAL )); do
  sleep $LOG_INTERVAL
  LOG_EVENTS=$(aws logs filter-log-events --log-group-name $LOG_GROUP_NAME --start-time "${LOG_BEGIN}000" --end-time "${LOG_END}000" --output json)
  echo "$LOG_EVENTS" | jq -rM --argjson eventIds "$LOG_EVENTIDS" '.events[] as $event | select($eventIds | contains([$event.eventId]) | not) | $event | "\(.timestamp / 1000 | todateiso8601) \(.message)"'
  LOG_EVENTIDS=$(echo "$LOG_EVENTS" | jq -crM --argjson eventIds "$LOG_EVENTIDS" '$eventIds + [.events[].eventId] | unique')
done

Usage: save the file, chmod +x it, and then run it: ./cloudwatch-logs-tail.sh log-group-name. The script also takes parameters for begin and end times, which default to now and 2 minutes respectively. You can specify any strings which can be parsed by date --date for these parameters.

How it works: the script keeps a list of event IDs that have been displayed, which is empty to begin with. It queries CloudWatch Logs to get all log entries in the specified time interval, and displays those which do not match our list of event IDs. The it saves all of the event IDs for the next iteration.

The script polls every few seconds (set by LOG_INTERVAL in the script), and keeps polling for one more interval past the end time to account for the delay between log ingestion and availability.

Note that this script is not going to be great if you want to keep tailing the logs for more than a few minutes at a time, because the query results that it gets from AWS will keep getting bigger with every added log item. It's fine for quick runs though.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.