I was wondering if there is a better way to split a log text file than doing the shell loop below, ideally with a single shell command.
The log file looks like that:
2016-11-20T16:19:21+00:00 Logging started
2016-11-20T16:20:41+00:00 System is up
2016-11-20T16:21:07+00:00 Unknown event 45
...
2016-11-25T08:40:00+00:00 Blah blah
2016-11-25T08:42:00+00:00 Blah blah
...
2016-11-27T11:32:00+00:00 System powering down
- All the lines start with an ISO8601 date marker (UTC)
- The lines are in chronological order, by construction, because lines are appended when there is an event to log.
- The file is growing
So, the task we want to accomplish is split the file at a given time. Let's say I keep only the last week entries to avoid the ever-growing syndrome.
So having a date of '2016-11-25T08:41:00+00:00', I wish to keep only those entries posterior to that date. Note that the date at which we want to cut our file does not necessary correspond to an existing entry (as in the example).
So, the best I could do is a piece of code like that:
WHEN='2016-11-25T08:41:00+00:00' # actually that is read as a parameter
while read line; do
if [ "${line}" \> "${WHEN}" ]; then
echo "${line}"
fi
done <"${LOGFILE}" >"${CUTFILE}"
That works, but as it is a shell loop, it could be slow if the file becomes really big.
So, any better suggestion with a standard command/utility?
awk
for this? – Inian Nov 27 '16 at 11:05logrotate
. It has a lot of interesting options such as compressing and moving files to a directory for "old" logs, mailing, and more. It evaluates the size of the log, the time (how frequently to rotate), but not the file contents. I don't think you even need the latter. – Ruslan Osmanov Nov 27 '16 at 14:03