I would like a "system" that monitors a process and would kill said process if:

  • the process exceeds some memory requirements
  • the process does not respond to a message from the "system" in some period of time

I assume this "system" could be something as simple as a monitoring process? A code example of how this could be done would be useful. I am of course not averse to a completely different solution to this problem.

share|improve this question
up vote 9 down vote accepted

For the first requirement, you might want to look into either using ulimit, or tweaking the kernel OOM-killer settings on your system.

Monitoring daemons exist for this sort of thing as well. God is a recent example.

share|improve this answer

Try Process Resource Monitor for a classic, easy-to-use process monitor. Code available under the GPL.

There's a few other monitoring scripts there you might find interesting too.

share|improve this answer

I wrote a script that runs as a cron job and can be customized to kill problem processes:

#!/usr/local/bin/perl

use strict;
use warnings;
use Proc::ProcessTable;

my $table = Proc::ProcessTable->new;

for my $process (@{$table->table}) {
    # skip root processes
    next if $process->uid == 0 or $process->gid == 0;

    # skip anything other than Passenger application processes
    #next unless $process->fname eq 'ruby' and $process->cmndline =~ /\bRails\b/;

    # skip any using less than 1 GiB
    next if $process->rss < 1_073_741_824;

    # document the slaughter
    (my $cmd = $process->cmndline) =~ s/\s+\z//;
    print "Killing process: pid=", $process->pid, " uid=", $process->uid, " rss=", $process->rss, " fname=", $process->fname, " cmndline=", $cmd, "\n";

    # try first to terminate process politely
    kill 15, $process->pid;

    # wait a little, then kill ruthlessly if it's still around
    sleep 5;
    kill 9, $process->pid;
}

http://blog.endpoint.com/2012/08/automatically-kill-process-using-too.html

share|improve this answer
    
Lone link is considered a poor answer since it is meaningless by itself and target resource is not guaranteed to be alive in the future. Please try to include at least summary of information you are linking to. – j0k Aug 31 '12 at 7:51
    
this is great, I just made it infinitely loop every second. – over_optimistic May 17 '14 at 0:27

If you want to set up a fairly comprehensive monitoring system, check out monit. It can be very (VERY VERY VERY VERY) chatty at times, but it will do a lot of monitoring, restart services, alert you, etc.

That said, don't be surprised if you're getting dozens of e-mails a day until you get used to configuring it and telling it what not to bug you about.

share|improve this answer

To limit memory usage of processes, check /etc/security/limits.conf

share|improve this answer
    
Or ulimit (shell) or setrlimit (C). – ephemient Oct 9 '08 at 19:03

Are the monitored processes ones you're writing, or just any process?

If they're arbitrary processes then it might be hard to monitor for responsiveness. Unless the process is already set up to handle and respond to events that you can send it, then I doubt you'll be able to monitor them. If they're processes that you're writing, you'd need to add some kind of message handling that you can use the check against.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.