188

I ended up writing a quick little script for this in Python, but I was wondering if there was a utility you could feed text into which would prepend each line with some text -- in my specific case, a timestamp. Ideally, the use would be something like:

cat somefile.txt | prepend-timestamp

(Before you answer sed, I tried this:

cat somefile.txt | sed "s/^/`date`/"

But that only evaluates the date command once when sed is executed, so the same timestamp is incorrectly prepended to each line.)

2
  • Is the cat somefile.txt a bit "misleading"? I'd expect it to happen "at once" and have a single timestamp. Wouldn't this be a better test program: (echo a; sleep 1; echo b; sleep 3; echo c; sleep 2)? Commented Apr 13, 2018 at 2:51
  • looks like none of the answers will easily give me milliseconds (I'm on Mac) Commented Aug 28, 2021 at 4:27

19 Answers 19

211

ts from moreutils will prepend a timestamp to every line of input you give it. You can format it using strftime too.

$ echo 'foo bar baz' | ts
Mar 21 18:07:28 foo bar baz
$ echo 'blah blah blah' | ts '%F %T'
2012-03-21 18:07:30 blah blah blah
$ 

To install it:

sudo apt-get install moreutils
8
  • 3
    to capture and stamp stderr at the same time: bad command 2>&1 | ts results in Jan 29 19:58:31 -bash: bad: command not found
    – rymo
    Commented Jan 30, 2014 at 1:59
  • I want to use the time format 2014 Mar 21 18:07:28. How do I get it?
    – a06e
    Commented Dec 6, 2014 at 17:38
  • @becko - according to the manual page, give a strftime format string as argument: [.. command ..] | ts '%Y %b %d %T', for example. Commented Jun 8, 2015 at 18:22
  • 2
    For OS X, brew install moreutils. To output UTC, you can set TZ locally, e.g. ls -l |TZ=UTC ts '%Y-%m-%dT%H:%M:%SZ'
    – karmakaze
    Commented Jan 25, 2016 at 19:52
  • 1
    @Dorian this is because ruby is buffering the output; if you flush the buffer before sleeping it works well enough: ruby -e "puts 1; STDOUT.flush; sleep 1; puts 2; STDOUT.flush; sleep 2; puts 3" | ts '%F %T'
    – umläute
    Commented Jan 12, 2017 at 15:04
195

Could try using awk:

<command> | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'

You may need to make sure that <command> produces line buffered output, i.e. it flushes its output stream after each line; the timestamp awk adds will be the time that the end of the line appeared on its input pipe.

If awk shows errors, then try gawk instead.

10
  • 33
    On my system 'awk' itself was buffering. (This can be problematic for log files). I fixed this by using: awk '{ print strftime("%Y-%m-%dT%H:%M:%S"), $0; fflush(); }'
    – user47741
    Commented Mar 11, 2010 at 13:28
  • 21
    strftime() appears to be a GNU awk extension, so if you're on Mac OS, for example, use gawk instead of awk.
    – Joe Shaw
    Commented Mar 8, 2012 at 14:24
  • It doesn't cease to amaze me how powerful bash is. I didn't think there would be such an easy solution for this complicated task.
    – recluze
    Commented Sep 11, 2012 at 10:58
  • 3
    For OS X users, you have to install gawk and use that instead of awk. If you have MacPorts: sudo port install gawk Commented Nov 26, 2013 at 20:40
  • 5
    @teh_senaus As far as I know, awk's strftime() does not have millisecond precision. However you can use the date command. Basically you just trim nanoseconds to three characters. It will look like this: COMMAND | while read -r line; do echo "$(date '+%Y-%m-%d %T.%3N') $line"; done. Note that you can abbreviate %H:%M:%S with %T. If you still want to use awk for some reason, you can do the following: COMMAND | awk '{ "date +%Y-%m-%d\\ %T.%3N" | getline timestamp; print timestamp, $0; fflush(); }'
    – Six
    Commented Sep 26, 2015 at 7:47
47

annotate, available via that link or as annotate-output in the Debian devscripts package.

$ echo -e "a\nb\nc" > lines
$ annotate-output cat lines
17:00:47 I: Started cat lines
17:00:47 O: a
17:00:47 O: b
17:00:47 O: c
17:00:47 I: Finished with exitcode 0
2
  • 1
    This is a great solution, but will not work with piped output or subshelled output. It will only reformat the output for a program or script that you give it as an argument.
    – slm
    Commented Sep 26, 2014 at 13:52
  • 1
    annotate-output works nicely and is easy to use, but is REALLY slow. Commented Aug 10, 2016 at 15:09
31

Distilling the given answers to the simplest one possible:

unbuffer $COMMAND | ts

On Ubuntu, they come from the expect-dev and moreutils packages.

sudo apt-get install expect-dev moreutils
3
  • 1
    It's expect, not expect-dev, but the latter pulls in the former so this works. (Ubuntu 14.10, I wonder if the binary was moved to a different package at some point.) Commented Apr 3, 2015 at 10:41
  • For Ubuntu 14.04 LTS, it's still in expect-dev
    – Willem
    Commented Apr 9, 2015 at 14:37
  • 3
    If you'd like to get elapsed time with millisecond accuracy, use unbuffer $COMMAND | ts -s %.s (-s if for elapsed time and %.s for time in seconds with fractional part) Commented Jun 2, 2016 at 21:15
26

How about this?

cat somefile.txt | perl -pne 'print scalar(localtime()), " ";'

Judging from your desire to get live timestamps, maybe you want to do live updating on a log file or something? Maybe

tail -f /path/to/log | perl -pne 'print scalar(localtime()), " ";' > /path/to/log-with-timestamps
5
  • 4
    This perl example was particularly useful for me as I was working as a user on an ubuntu deployment, which apparently uses 'mawk' instead of 'gawk' -- and doesn't have the strftime function. Thanks!
    – opello
    Commented Aug 19, 2010 at 5:17
  • 5
    +1 to this being handy on a default Ubuntu install. I used tail -f /path/to/log | perl -pne 'use POSIX qw(strftime); print strftime "%Y-%m-%dT%H:%M:%SZ ", gmtime();' to get an ISO8601/RFC3339 style date instead of the wacko one the simpler version yields.
    – natevw
    Commented May 2, 2013 at 19:32
  • 1
    If you want to be able look at timestamped log output as it arrives, it may be helpful to add BEGIN { $|++; } before the print statement in order to have Perl flush its output with each line.
    – user725091
    Commented May 29, 2014 at 11:25
  • 2
    You can shorten the perl script further to just ls | perl -pne 'print localtime." "'. The scalar conversion happens due to string concatenation. Commented Jun 10, 2015 at 15:01
  • Why do you use both -p and -n options? It seems like -n is redundant if you have specified -p. Commented Jan 31, 2020 at 19:54
19

Kieron's answer is the best one so far. If you have problems because the first program is buffering its out you can use the unbuffer program:

unbuffer <command> | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; }'

It's installed by default on most linux systems. If you need to build it yourself it is part of the expect package

http://expect.nist.gov

1
  • Note that I've found that 'unbuffer' sometimes hides the return result of the called program - so it's not useful in Jenkins or other things that rely on the return status Commented Aug 9, 2014 at 11:50
19

Just gonna throw this out there: there are a pair of utilities in daemontools called tai64n and tai64nlocal that are made for prepending timestamps to log messages.

Example:

cat file | tai64n | tai64nlocal
10

Use the read(1) command to read one line at a time from standard input, then output the line prepended with the date in the format of your choosing using date(1).

$ cat timestamp
#!/bin/sh
while read line
do
  echo `date` $line
done
$ cat somefile.txt | ./timestamp
1
  • 2
    A bit heavyweight as it forks a new process per line, but it works!
    – Joe Shaw
    Commented Mar 22, 2011 at 17:56
3

I'm not an Unix guy, but I think you can use

gawk '{print strftime("%d/%m/%y",systime()) $0 }' < somefile.txt
3
#! /bin/sh
unbuffer "$@" | perl -e '
use Time::HiRes (gettimeofday);
while(<>) {
        ($s,$ms) = gettimeofday();
        print $s . "." . $ms . " " . $_;
}'
3
$ cat somefile.txt | sed "s/^/`date`/"

you can do this (with gnu/sed):

$ some-command | sed "x;s/.*/date +%T/e;G;s/\n/ /g"

example:

$ { echo 'line1'; sleep 2; echo 'line2'; } | sed "x;s/.*/date +%T/e;G;s/\n/ /g"
20:24:22 line1
20:24:24 line2

of course, you can use other options of the program date. just replace date +%T with what you need.

2

Here's my awk solution (from a Windows/XP system with MKS Tools installed in the C:\bin directory). It is designed to add the current date and time in the form mm/dd hh:mm to the beginning of each line having fetched that timestamp from the system as each line is read. You could, of course, use the BEGIN pattern to fetch the timestamp once and add that timestamp to each record (all the same). I did this to tag a log file that was being generated to stdout with the timestamp at the time the log message was generated.

/"pattern"/ "C\:\\\\bin\\\\date '+%m/%d %R'" | getline timestamp;
print timestamp, $0;

where "pattern" is a string or regex (without the quotes) to be matched in the input line, and is optional if you wish to match all input lines.

This should work on Linux/UNIX systems as well, just get rid of the C\:\\bin\\ leaving the line

             "date '+%m/%d %R'" | getline timestamp;

This, of course, assumes that the command "date" gets you to the standard Linux/UNIX date display/set command without specific path information (that is, your environment PATH variable is correctly configured).

1
  • On OS X I needed a space before the date command. Sadly it doesn't seem to re-evaluate the command for each line. Like others have suggested, I'm trying to add timestamps to tail -f. Commented Jul 6, 2011 at 17:13
2

Mixing some answers above from natevw and Frank Ch. Eigler.

It has milliseconds, performs better than calling a external date command each time and perl can be found in most of the servers.

tail -f log | perl -pne '
  use Time::HiRes (gettimeofday);
  use POSIX qw(strftime);
  ($s,$ms) = gettimeofday();
  print strftime "%Y-%m-%dT%H:%M:%S+$ms ", gmtime($s);
  '

Alternative version with flush and read in a loop:

tail -f log | perl -pne '
  use Time::HiRes (gettimeofday); use POSIX qw(strftime);
  $|=1;
  while(<>) {
    ($s,$ms) = gettimeofday();
    print strftime "%Y-%m-%dT%H:%M:%S+$ms $_", gmtime($s);
  }'
1
  • 1
    Two flaws with the above: 1. $ms is not aligned and zero-padded. 2. Any % codes in the log will be mangled. Thus instead of the print line, I used: printf "%s+%06d %s", (strftime "%Y-%m-%dT%H:%M:%S", gmtime($s)), $ms, $_; Commented Apr 4, 2018 at 11:18
1

caerwyn's answer can be run as a subroutine, which would prevent the new processes per line:

timestamp(){
   while read line
      do
         echo `date` $line
      done
}

echo testing 123 |timestamp
4
  • 3
    how does that prevent date from being spawned at every line ?
    – Gyom
    Commented Nov 21, 2013 at 13:42
  • @Gyom It doesn't prevent date to be spawned every line. It prevents shell from spawning every line. I would add that in bash you coud do something like: timestamp() { while read line; do echo "$(date) $line"; done; }; export -f timestamp in a script that you source.
    – smbear
    Commented Nov 25, 2015 at 8:31
  • Calling date as a subprocess still means spawning it for every line, however you launch the script in the first place. Commented Jan 25, 2016 at 15:59
  • 3
    To prevent spawning command date for every line, try using bash printf builtin with %(datespec)T format. So it may look like timestamp() { while IFS= read -r line; do printf "%(%F %T)T %s\n" -1 "$line"; done; }. Shell builtins won't spawn. Datespec format is like strftime(3) i.e. date formats. This requires bash, not sure since which version it supports that printf specifier. Commented Nov 5, 2017 at 20:25
1

Disclaimer: the solution I am proposing is not a Unix built-in utility.

I faced a similar problem a few days ago. I did not like the syntax and limitations of the solutions above, so I quickly put together a program in Go to do the job for me.

You can check the tool here: preftime

There are prebuilt executables for Linux, MacOS, and Windows in the Releases section of the GitHub project.

The tool handles incomplete output lines and has (from my point of view) a more compact syntax.

<command> | preftime

It's not ideal, but I though I'd share it in case it helps someone.

1

The other answers mostly work, but have some drawbacks. In particular:

  1. Many require installing a command not commonly found on linux systems, which may not be possible or convenient.
  2. Since they use pipes, they don't put timestamps on stderr, and lose the exit status.
  3. If you use multiple pipes for stderr and stdout, then some do not have atomic printing, leading to intermingled lines of output like [timestamp] [timestamp] stdout line \nstderr line
  4. Buffering can cause problems, and unbuffer requires an extra dependency.

To solve (4), we can use stdbuf -i0 -o0 -e0 which is generally available on most linux systems (see How to make output of any shell command unbuffered?).

To solve (3), you just need to be careful to print the entire line at a time.

  • Bad: ruby -pe 'print Time.now.strftime(\"[%Y-%m-%d %H:%M:%S] \")' (Prints the timestamp, then prints the contents of $_.)
  • Good: ruby -pe '\$_ = Time.now.strftime(\"[%Y-%m-%d %H:%M:%S] \") + \$_' (Alters $_, then prints it.)

To solve (2), we need to use multiple pipes and save the exit status:

alias tslines-pipe="stdbuf -i0 -o0 ruby -pe '\$_ = Time.now.strftime(\"[%Y-%m-%d %H:%M:%S] \") + \$_'"
function tslines() (
  stdbuf -o0 -e0 "$@" 2> >(tslines-pipe) > >(tslines-pipe)
  status="$?"
  exit $status
)

Then you can run a command with tslines some command --options.

This almost works, except sometimes one of the pipes takes slightly longer to exit and the tslines function has exited, so the next prompt has printed. For example, this command seems to print all the output after the prompt for the next line has appeared, which can be a bit confusing:

tslines bash -c '(for (( i=1; i<=20; i++ )); do echo stderr 1>&2; echo stdout;  done)'

There needs to be some coordination method between the two pipe processes and the tslines function. There are presumably many ways to do this. One way I found is to have the pipes send some lines to a pipe that the main function can listen to, and only exit after it's received data from both pipe handlers. Putting that together:

alias tslines-pipe="stdbuf -i0 -o0 ruby -pe '\$_ = Time.now.strftime(\"[%Y-%m-%d %H:%M:%S] \") + \$_'"
function tslines() (
  # Pick a random name for the pipe to prevent collisions.
  pipe="/tmp/pipe-$RANDOM"
  
  # Ensure the pipe gets deleted when the method exits.
  trap "rm -f $pipe" EXIT

  # Create the pipe.  See https://www.linuxjournal.com/content/using-named-pipes-fifos-bash
  mkfifo "$pipe"
  # echo will block until the pipe is read.
  stdbuf -o0 -e0 "$@" 2> >(tslines-pipe; echo "done" >> $pipe) > >(tslines-pipe; echo "done" >> $pipe)
  status="$?"

  # Wait until we've received data from both pipe commands before exiting.
  linecount=0
  while [[ $linecount -lt 2 ]]; do
    read line
    if [[ "$line" == "done" ]]; then
      ((linecount++))
    fi
  done < "$pipe"
  exit $status
)

That synchronization mechanism feels a bit convoluted; hopefully there's a simpler way to do it.

0

doing it with date and tr and xargs on OSX:

alias predate="xargs -I{} sh -c 'date +\"%Y-%m-%d %H:%M:%S\" | tr \"\n\" \" \"; echo \"{}\"'"
<command> | predate

if you want milliseconds:

alias predate="xargs -I{} sh -c 'date +\"%Y-%m-%d %H:%M:%S.%3N\" | tr \"\n\" \" \"; echo \"{}\"'"

but note that on OSX, date doesn't give you the %N option, so you'll need to install gdate (brew install coreutils) and so finally arrive at this:

alias predate="xargs -I{} sh -c 'gdate +\"%Y-%m-%d %H:%M:%S.%3N\" | tr \"\n\" \" \"; echo \"{}\"'"
0

No need to specify all the parameters in strftime() unless you really want to customize the outputting format :

   echo "abc 123 xyz\njan 765 feb" \
    \
    | gawk -Sbe 'BEGIN {_=strftime()" "} sub("^",_)'

    Sat Apr  9 13:14:53 EDT 2022 abc 123 xyz
    Sat Apr  9 13:14:53 EDT 2022 jan 765 feb

works the same if you have mawk 1.3.4. Even on awk-variants without the time features, a quick getline could emulate it :

echo "abc 123 xyz\njan 765 feb" \
\
| mawk2 'BEGIN {     (__="date")|getline _;
                close(__)
                        _=_" " } sub("^",_)'
   
Sat Apr  9 13:19:38 EDT 2022 abc 123 xyz
Sat Apr  9 13:19:38 EDT 2022 jan 765 feb

If you wanna skip all that getline and BEGIN { }, then something like this :

mawk2 'sub("^",_" ")' \_="$(date)"
-1

If the value you are prepending is the same on every line, fire up emacs with the file, then:

Ctrl + <space>

at the beginning of the of the file (to mark that spot), then scroll down to the beginning of the last line (Alt + > will go to the end of file... which probably will involve the Shift key too, then Ctrl + a to go to the beginning of that line) and:

Ctrl + x r t

Which is the command to insert at the rectangle you just specified (a rectangle of 0 width).

2008-8-21 6:45PM <enter>

Or whatever you want to prepend... then you will see that text prepended to every line within the 0 width rectangle.

UPDATE: I just realized you don't want the SAME date, so this won't work... though you may be able to do this in emacs with a slightly more complicated custom macro, but still, this kind of rectangle editing is pretty nice to know about...

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.