3

I have a script that uses two separate grep statements:

grep -E "GET[^\"]*\.html" tmp.cleaned.log | grep -v "XMLHttpRequest" | wc -l

and,

grep -E "^[^\"]+\"[^\"]+\" \"[^\"]+\" \"[^\"]+\" \"[^\"-\\]+\"" tmp.cleaned.log | wc -l

It stores the resulting values in an output log file. When I run the script manually on the shell prompt, I get the correct results for both the statements : 680 and 10028.

However when I schedule the script using crontab, the first line returns the correct value of 680, but the second line results in 0.

I have redirected stderr and stdout, and there seems to be no error being logged. I have also added SHELL=/bin/bash in the crontab, in addition the shebang in the script itself. The crontab is for the user root and looks like this:

SHELL=/bin/bash
16      */1     *       *       *       /u02/sites/webstats/rundaily.sh

The script starts by changing directory to the correct location, so it is not path issue; besides both statements refer to the same file & executable.

Trying to solve this is really driving me crazy. Any help would be appreciated.

Thanks.

UPDATE:

I think I have figured out why I was getting 0. My expression had a bug in it. It should have been

grep -E "^[^\"]+\"[^\"]+\" \"[^\"]+\" \"[^\"]+\" \"[^\"-]+\"" tmp.cleaned.log | wc -l

instead of

grep -E "^[^\"]+\"[^\"]+\" \"[^\"]+\" \"[^\"]+\" \"[^\"-\\]+\"" tmp.cleaned.log | wc -l

This is the reason why it was returning 0 (no matches). However it still does not explain why I was seeing different results in cron vs in shell. I realize now that the value of 10028 was the total line count of tmp.cleaned.log.

So, when executing from shell, the grep expression returned ALL rows when it did not match any using the wrong regex. when executing the same wrong regex from cron, the grep correctly returned ZERO rows.

I am still interested in understanding this difference in behavior.

2
  • Was you grep directly in your crontab or was it done in a script called by the crontab ?
    – Pilou
    Jun 6, 2013 at 8:39
  • Always in a script called by the crontab. I had tried two different approaches, one where the script was directly invoked from crontab. The other where the crontab invoked script in turn invoked another script. In both cases, the results were the same.
    – sujitv
    Jun 8, 2013 at 0:14

1 Answer 1

3

I think you'll find that the difference was caused by the locale:

line='A"B" "C" "D" "E"'
regex="^[^\"]+\"[^\"]+\" \"[^\"]+\" \"[^\"]+\" \"[^\"-\\]+\""

LC_COLLATE=en_US.utf8 grep -E "$regex" <<< "$line"  # MATCH
LC_COLLATE=C grep -E "$regex" <<< "$line"           # NO MATCH

The character range from " to \ in plain ASCII values (34-92) includes the uppercase letters and a small set of other characters. A proper locale will group punctuation characters and letters separately, independent of their code points.

Most likely, one of your init files like .bashrc sets LANG, LC_ALL or LC_COLLATE to a proper locale. These files are not sourced by non-interactive shells like the ones cron starts, causing the difference you see.

3
  • You are correct, in that my environment in the shell hadLANG=en_US.UTF-8
    – sujitv
    Jun 8, 2013 at 0:23
  • ... whereas the environment obtained from crontab did not. The results are also consistent with what you have indicated. I tested my changes (removing the \\ from last exclusion class) also with your code, and the results were again consistent with what you suggested. But I am not sure I understand the reasoning. Why would attempting to exclude \ result in a no match
    – sujitv
    Jun 8, 2013 at 0:32
  • Ok. I understand now. I was trying to exclude ", -, and \. However, the way I constructed the last exclusion it was being interpreted as "-\ (ie, " to ). This combined with the A-Z being within or outside this range was resulting in the mis-match or match. Wow, a badly placed hyphen...
    – sujitv
    Jun 8, 2013 at 0:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.