11

In bash I am trying to read a log file and will print only the lines that have a timestamp between two specific times. The time format is hh:mm:ss. For example, I would be searching for lines that would fall between 12:52:33 to 12:59:33.

I want to use regular expression because I can use it in grep function.

Each log line begins with some_nr 2014-05-15 21:58:00,000000 rest_of_line.

My solution gives me lines with 1 min margin. I cut out ss and take all lines with hh:mm:[0-9]{2}. $2 has format filename_hh:mm:; for example: "24249_16:05:;24249_16:05:;24249_16:07:;24249_16:07:;24249_16:08:"

My code:

B=$2  

for line in ${B//;/ } ;
do  
    TENT=`echo $line | awk '{split($0,numbers,"_"); print numbers[1]}'`"_logs.txt"
    TIME=`echo $line | awk '{split($0,numbers,"_"); print numbers[2]}'`"[0-9]{2}"

    grep -iE ${TIME} ${TENT} >> ${FILE1}
done

I need a solution with 15 sec margin for any time not 60. I want to have input in format filename_hh:mm:ss and take lines for hh:mm:ss +/- 15s or filename_hh:mm:ss(1)_hh:mm:ss(2) and take lines between hh:mm:ss(1) and hh:mm:ss(2). For sometime there is no lines so the solution should 'recognize' if sometimes match inputted interval or not.

Log files look like this:

1002143 1002143 2014/15/05 22:09:52.937004 bla 
1002130         2014/15/05 22:09:44.786002 bla bla
1001667         2014/15/05 22:09:44.592009 bl a bla
1001667 1001667 2014/15/05 22:09:44.592009 bl a bla
2
  • Can you post a sample of your log file?
    – anubhava
    May 16, 2014 at 15:48
  • yes, I added in in question above.
    – herder
    May 16, 2014 at 15:55

5 Answers 5

18

I believe sed is the best option:

sed -rne '/<timestamp>/,/<timestamp>/ p' <file>

ex:

tiago@dell:~$ sed -rne '/08:17:38/,/08:24:36/ p' /var/log/syslog 
May 16 08:17:38 dell AptDaemon.Worker: INFO: Processing transaction /org/debian/apt/transaction/08a244f7b8ce4fad9f6b304aca9eae7a
May 16 08:17:50 dell AptDaemon.Worker: INFO: Finished transaction /org/debian/apt/transaction/08a244f7b8ce4fad9f6b304aca9eae7a
May 16 08:18:50 dell AptDaemon.PackageKit: INFO: Initializing PackageKit transaction
May 16 08:18:50 dell AptDaemon.Worker: INFO: Simulating trans: /org/debian/apt/transaction/37c3ef54a6ba4933a561c49b3fac5f6e
May 16 08:18:50 dell AptDaemon.Worker: INFO: Processing transaction /org/debian/apt/transaction/37c3ef54a6ba4933a561c49b3fac5f6e
May 16 08:18:51 dell AptDaemon.PackageKit: INFO: Get updates()
May 16 08:18:52 dell AptDaemon.Worker: INFO: Finished transaction /org/debian/apt/transaction/37c3ef54a6ba4933a561c49b3fac5f6e
May 16 08:24:36 dell AptDaemon: INFO: Quitting due to inactivity

9
  • no sed is definitely not the "best" option. think about that the log output could have timestamp
    – Kent
    May 16, 2014 at 15:03
  • 1
    @kent in that case one could match the beggining of the line
    – Tiago Lopo
    May 16, 2014 at 15:10
  • my log lines are beginning with: 2014-05-15 21:58:00,000000 and becouse those miliseconds I wanted to use grep.
    – herder
    May 16, 2014 at 15:28
  • herder would you mind to share which grep command are you using?
    – Tiago Lopo
    May 16, 2014 at 15:36
  • I add my code in question above.
    – herder
    May 16, 2014 at 15:42
10

log file is usually sorted by timestamp, assume the timestamp is on the first column, you could:

awk -v from="12:52:33" -v to="12:59:33" '$1>=from && $1<=to' foo.log

in this way, you can change the from and to to get different set of log entries. regex is not a good tool to do number calculation/comparison.

5
  • This solution also doesn't work. In my log_file in second column sometimes is a number and sometimes not - and above code doesn't recognize properly columns. If in second column there is a number then it see time in 4th, if not - then in 3rd.
    – herder
    May 20, 2014 at 7:39
  • @herder the solution was for the 1st version of your question. You edited it and added new stuff, the solution will fail. but awk has date related function, also it can invoke external command. it could do your job. read some tutorials.
    – Kent
    May 20, 2014 at 7:54
  • I appreciate your help - I know it was for 1st version of question but I didn't predict those problems. I'll search in man awk ;)
    – herder
    May 20, 2014 at 8:55
  • awk -v from=$TIME1".000" -v to=$TIME2".000" '{for (i=3; i<=4; i++) if ($i~"([0-9]{2}:){2}[0-9]{2}.[0-9]{3}" && ($i<=to && $i>=from)) print $0}' $TENT
    – herder
    May 20, 2014 at 13:20
  • I used if to check in which column I have a time format. But actually using perl is more comfortable. Thanks everyone for help.
    – herder
    May 20, 2014 at 13:27
4

You can use this regex in egrep:

egrep '12:5[2-9]:33' file.log
3
  • I need general solution, not only for this example but any two times.
    – herder
    May 16, 2014 at 15:22
  • Sure that is also possible but 1. It will be better to clarify this in your question and 2. Show your own attempt.
    – anubhava
    May 16, 2014 at 15:24
  • I added my tries in question above.
    – herder
    May 16, 2014 at 15:40
2

You are using the wrong tool for this task. Once you have a regular expression like the one given by @anubhava, you can easily find a time interval that is not matched by it. grep and regexps might work for a few special cases, but they do not scale to the general case.

Can you use some tool that can actually "understand" the timestamps? Probably every scripting language out there (perl, python, ruby, lua) has builtin or library support for parsing time and date.

However, you might be able to employ the powers of GNU date:

% date --date="2014-05-15 21:58:00 15 sec ago" +'%Y-%m-%d %H:%M:%S'
2014-05-15 21:57:45
% date --date="2014-05-15 21:58:00 15 sec" +'%Y-%m-%d %H:%M:%S' 
2014-05-15 21:58:15

and plug that into Tiago's sed filter idea.

2
  • you are right, but this is not an answer buddy.
    – Kent
    May 16, 2014 at 14:44
  • sorry, just couldn't help myself ... May 16, 2014 at 15:06
1

You can try the following perl script:

#! /usr/bin/perl

use warnings;
use strict;
use Time::Piece;
use autodie;

my $arg=shift;
my @a =split("_",$arg);
my $fn=shift @a;

my $dfmt='%Y/%d/%m';
my $fmt=$dfmt.' %H:%M:%S';
my $t = localtime;
my $date=$t->strftime($dfmt);
my $t1; my $t2;
if (@a == 1) {
   my $d=$date.' '.$a[0];
   my $tt=Time::Piece->strptime($d, $fmt);
   $t1=$tt-15;
   $t2=$tt+15;
} elsif (@a == 2) {
   $t1=Time::Piece->strptime($date.' '.$a[0], $fmt);
   $t2=Time::Piece->strptime($date.' '.$a[1], $fmt);
} else {
   die "Unexpected input argument!";
}

$fn=$fn.'_logs.txt';
doGrep($fn,$t1,$t2,$fmt);

sub doGrep { 
   my ($fn,$t1,$t2,$fmt) = @_;

   open (my $fh, "<", $fn);
   while (my $line=<$fh>) {
      my ($d1,$d2) = $line=~/\S+\s+(\S+)\s+(\d\d:\d\d:\d\d)/;
      my $d=$d1.' '.$d2;
      my $t=Time::Piece->strptime($d, $fmt);
      print $line if ($t>$t1 && $t<$t2);
   }
   close ($fh);
}

Run it from command line using syntax : ./p.pl A_22:09:14.

8
  • I got an error: "Error parsing time at C:/Perl64/lib/Time/Piece.pm line 469, <$_[...]> line 1."
    – herder
    May 20, 2014 at 8:42
  • @herder Maybe you could paste an example of the logfile you are using on pastebin.com ? Then I could try to run it on my machine also.. to check May 20, 2014 at 8:53
  • @HH - thanks! pastebin.com/Us6BrbSF
    – herder
    May 20, 2014 at 9:18
  • 1
    @herder Thanks.. It seems that your file has a different date format than that you gave in your question.. Compare 04/16/14 and 2014/15/05 May 20, 2014 at 9:26
  • 1
    @herder In your question you only specified a time interval. There was no requirement regarding the date. So the script assumed it was the current date. However, it seems that the log file is not from the current date. What would you like to do? Specify a date on the command line also? May 20, 2014 at 9:33

Not the answer you're looking for? Browse other questions tagged or ask your own question.