5

I'm moving a website to Hostmonster and asked where the server log is located so I can automatically scan it for CGI errors. I was told, "We're sorry, but we do not have cgi errors go to any files that you have access to."

For organizational reasons I'm stuck with Hostmonster and this awful policy, so as a workaround I thought maybe I'd modify the CGI scripts to redirect STDERR to a custom log file.

I have a lot of scripts (269) so I need an easy way in both Python and Perl to redirect STDERR to a custom log file.

Something that accounts for file locking either explicitly or implicitly would be great, since a shared CGI error log file could theoretically be written to by more than one script at once if more than one script fails at the same time.

(I want to use a shared error log so I can email its contents to myself nightly and then archive or delete it.)

I know I may have to modify each file (grrr), that's why I'm looking for something elegant that will only be a few lines of code. Thanks.

5
  • The other problem is that they probably don't allow you to create files either. Commented Nov 23, 2009 at 8:28
  • I'm pretty sure you can create files in your home or web (public_html) tree. The other aspect I thought of is that if I have 269 scripts and I'm appending to my own log file, maybe I need to worry about file locking on the custom log file
    – Chirael
    Commented Nov 23, 2009 at 8:47
  • Here's another crazy idea I had, which would either be pretty cool or pretty insane. These scripts all start with #!/usr/bin/python or perl. What if I created my own wrapper such as /home/user/bin/mypython or myperl. Then that could have the redirection for stderr and pass the rest of the script to the "real" perl or python. If that's actually a good idea (not a horrible one), I just don't know the mechanics of "passing the rest of the [cgi] script to the real perl/python".
    – Chirael
    Commented Nov 23, 2009 at 9:06
  • (it occurred to me that maybe that wouldn't be portable because some hosts, possibly even Hostmonster, might not like CGI scripts calling something weird like /home/user/bin/myperl - they might have security issues with it, even though they really shouldn't per se)
    – Chirael
    Commented Nov 23, 2009 at 9:08
  • About the problem with many error logs / file locking: just append or zip the files before emailing them -> problem solved? If you want to have the log events in chronological order, just write a simple script that goes through the logs.
    – Makis
    Commented Nov 23, 2009 at 9:17

7 Answers 7

4

For Perl, just close and re-open STDERR to point to a file of your choice.

close STDERR;
open STDERR, '>>', '/path/to/your/log.txt' 
  or die "Couldn't redirect STDERR: $!";

warn "this will go to log.txt";

Alternatively, you could look into a filehandle multiplexer like File::Tee.

2
  • This is a basic, probably dumb question, but with the simple open/append writing to STDERR there, does Perl implicitly handle file locking? I ask because it occurred to me after posting the question, that theoretically with a shared log file I might have to consider the possibility of multiple scripts failing at once (and thus, file locking issues)
    – Chirael
    Commented Nov 23, 2009 at 8:53
  • 2
    On Unix, if you open a file in O_APPEND mode, the seek and write operations are performed in an atomic way, therefore no locking is required.
    – hillu
    Commented Nov 23, 2009 at 10:47
3

Python: cgitb. At the top of your script, before other imports:

import cgitb
cgitb.enable(False, '/home/me/www/myapp/logs/errors')

(‘errors’ being a directory the web server user has write-access to.)

1
  • This is pretty cool in that it uses a built-in module and doesn't require many more lines of code. The only thing is, it seems each traceback/error-dump would be written to its own file. While that gets around file locking issues nicely (it occurred to me after posting the question that with a custom error log I'd have to worry about scripts possibly failing at the same time, and thus file locking), I kind of wanted one shared error log file I could periodically scan (and email to myself, then archive).
    – Chirael
    Commented Nov 23, 2009 at 8:50
3

In Perl try CGI::Carp

BEGIN { 
use CGI::Carp qw(carpout); 
use diagnostics;
open(LOG, ">errors.txt"); 
carpout(LOG);
close(LOG);
}

use CGI::Carp qw(fatalsToBrowser);
3

The solution I finally went with was similar to the following, near the top of all my scripts:

Perl:

open(STDERR,">>","/path/to/my/cgi-error.log")
    or die "Could not redirect STDERR: $OS_ERROR";

Python:

sys.stderr = open("/path/to/my/cgi-error.log", "a")

Apparently in Perl you don't need to close the STDERR handle before reopening it.

Normally I would close it anyway as a best practice, but as I said in the question, I have 269 scripts and I'm trying to minimize the changes. (Plus it seems more Perlish to just re-open the open filehandle, as awful as that sounds.)

In case anyone else has something similar in the future, here's what I'm going to do for updating all my scripts at once:

Perl:

find . -type f -name "*.pl" -exec perl -pi.bak -e 's%/usr/bin/perl%/usr/bin/perl\nopen(STDERR,">>","/path/to/my/cgi-error.log")\n    or die "Could not redirect STDERR: \$OS_ERROR";%' {} \;

Python:

find . -type f -name "*.py" -exec perl -pi.bak -e 's%^(import os, sys.*)%$1\nsys.stderr = open("/path/to/my/cgi-error.log", "a")%' {} \;

The reason I'm posting these commands is that it took me quite a lot of syntactical massaging to get those commands to work (e.g., changing Couldn't to Could not, changing #!/usr/bin/perl to just /usr/bin/perl so the shell wouldn't interpret ! as a history character, using $OS_ERROR instead of $!, etc.)

Thanks to everyone who commented. Since no one answered for both Perl and Python I couldn't really "accept" any of the given answers, but I did give votes to the ones which led me in the right direction. Thanks again!

2

python:

import sys

sys.stderr = open('file_path_with_write_permission/filename', 'a')
1
  • 1
    You probably meant 'a' mode. Otherwise the file will be overwritten for each request. Commented Nov 24, 2009 at 9:22
0

Python has the sys.stderr module that you might want to look into.

>>>help(sys.__stderr__.read)
Help on built-in function read:

read(...)
    read([size]) -> read at most size bytes, returned as a string.

    If the size argument is negative or omitted, read until EOF is reached.
    Notice that when in non-blocking mode, less data than what was requested
    may be returned, even if no size parameter was given.

You can store the output of this in a string and write that string to a file.

Hope this helps

2
  • If I understand this correctly, you're saying sys.__stderr__.read() would let me retrieve the contents of the stderr stream after an error has occurred. But wouldn't I need to know the length of the error message to retrieve, in order to get that? Maybe I don't understand your suggestion correctly though.
    – Chirael
    Commented Nov 23, 2009 at 8:58
  • I think you are referring to the [size] parameter. The '[]' mean that the parameter is optional. As the help text says, if no size is specified, then it keeps reading until EOF is reached. My understanding of this is that if you don't specify a size, then you will get the whole error Commented Nov 24, 2009 at 0:12
0

In my Perl CGI programs, I usually have

BEGIN {
  open(STDERR,'>>','stderr.log');
}

right after shebang line and "use strict;use warnings;". If you want, you may append $0 to file name. But this will not solve multiple programs problem, as several copies of one programs may be run simultaneously. I usually just have several output files, for every program group.

1
  • It's not your fault because you couldn't have known this, but the reason I didn't go with this solution was because I want to use a variable in the construction of the error log path, and I learned a long time ago that Perl doesn't really allow variables in BEGIN blocks. (It's apparently possible using an external package, but seems too involved/brittle for my taste.)
    – Chirael
    Commented Nov 24, 2009 at 9:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.