79

I am working on a websocket oriented node.js server using Socket.IO. I noticed a bug where certain browsers aren't following the correct connect procedure to the server, and the code isn't written to gracefully handle it, and in short, it calls a method to an object that was never set up, thus killing the server due to an error.

My concern isn't with the bug in particular, but the fact that when such errors occur, the entire server goes down. Is there anything I can do on a global level in node to make it so if an error occurs it will simply log a message, perhaps kill the event, but the server process will keep on running?

I don't want other users' connections to go down due to one clever user exploiting an uncaught error in a large included codebase.

1
  • 11
    This is the part of Node that really bothers me. I can't really stomach PHP, but at least with PHP having an error doesn't mean that the whole site is down...
    – Merc
    Nov 23, 2017 at 15:08

7 Answers 7

93

You can attach a listener to the uncaughtException event of the process object.

Code taken from the actual Node.js API reference (it's the second item under "process"):

process.on('uncaughtException', function (err) {
  console.log('Caught exception: ', err);
});

setTimeout(function () {
  console.log('This will still run.');
}, 500);

// Intentionally cause an exception, but don't catch it.
nonexistentFunc();
console.log('This will not run.');

All you've got to do now is to log it or do something with it, in case you know under what circumstances the bug occurs, you should file a bug over at Socket.IO's GitHub page:
https://github.com/LearnBoost/Socket.IO-node/issues

5
  • Ok, almost perfect. Too bad it doesn't give the line numbers that the error occurred on.
    – RobKohr
    Nov 20, 2010 at 0:14
  • 7
    You can print out err.stack, that will give you a stack trace which also happens to include the line numbers.
    – Ivo Wetzel
    Nov 20, 2010 at 11:36
  • 404 status codes are for HTTP, this question was about a socket server so while you can handle the error however you want, a 404 code wouldn't make sense here. However, if you were running an HTTP server through node.js you could certainly return an error 404 to the client if you built that logic into the error handler.
    – StapleGun
    Apr 20, 2012 at 2:32
  • 5
    Beware that this is bad practice and can lead to unexpected behaviour. See my answer below. Nov 8, 2012 at 10:37
  • var process = require('node:process');
    – arcsin
    Sep 10, 2022 at 7:26
36

Using uncaughtException is a very bad idea.

The best alternative is to use domains in Node.js 0.8. If you're on an earlier version of Node.js rather use forever to restart your processes or even better use node cluster to spawn multiple worker processes and restart a worker on the event of an uncaughtException.

From: http://nodejs.org/api/process.html#process_event_uncaughtexception

Warning: Using 'uncaughtException' correctly

Note that 'uncaughtException' is a crude mechanism for exception handling intended to be used only as a last resort. The event should not be used as an equivalent to On Error Resume Next. Unhandled exceptions inherently mean that an application is in an undefined state. Attempting to resume application code without properly recovering from the exception can cause additional unforeseen and unpredictable issues.

Exceptions thrown from within the event handler will not be caught. Instead the process will exit with a non-zero exit code and the stack trace will be printed. This is to avoid infinite recursion.

Attempting to resume normally after an uncaught exception can be similar to pulling out of the power cord when upgrading a computer -- nine out of ten times nothing happens - but the 10th time, the system becomes corrupted.

The correct use of 'uncaughtException' is to perform synchronous cleanup of allocated resources (e.g. file descriptors, handles, etc) before shutting down the process. It is not safe to resume normal operation after 'uncaughtException'.

To restart a crashed application in a more reliable way, whether uncaughtException is emitted or not, an external monitor should be employed in a separate process to detect application failures and recover or restart as needed.

6
  • This is a much better answer than the top one (killing the processes on error (slide 28))
    – maxdec
    Jun 28, 2013 at 15:01
  • 24
    I think this answer requires some kind of example on how to use domains and how do they resolve this problem. Apr 27, 2014 at 11:40
  • it has its own risks... ;-)
    – inf3rno
    Mar 9, 2015 at 0:02
  • 3
    Sorry, still don't agree with your answer as to why.. you do not explain why domain use is better.. and the reason is that - it's not... domain usage simply allows you to wrap smaller sections of code (rather than entire process) to catch uncaught exception. However, if your entire code is in a domain, it is just the same as uncaught exception. So according to the question, your answer "this is bad, this is good" seems to me irrelevant. nodejs.org/api/domain.html#domain_warning_don_t_ignore_errors Mar 13, 2015 at 21:38
  • 2
    An undefined exception places your application in an undefined state, the only good way to "resolve" this is to restart it. The role of domains is to better contain this undefined state. After an uncaught exception, the particular domain will have to be "restarted" by returning an error and stopping further code execution. So uncaught exceptions are still bad, but domains limit their effect and possibly allows you to discard less data. E.g. you can discard the incoming request that generated the uncaught exception and return an error, but the other 100 pending requests can still continue. Mar 15, 2015 at 18:21
8

I just did a bunch of research on this (see here, here, here, and here) and the answer to your question is that Node will not allow you to write one error handler that will catch every error scenario that could possibly occur in your system.

Some frameworks like express will allow you to catch certain types of errors (when an async method returns an error object), but there are other conditions that you cannot catch with a global error handler. This is a limitation (in my opinion) of Node and possibly inherent to async programming in general.

For example, say you have the following express handler:

app.get("/test", function(req, res, next) {
    require("fs").readFile("/some/file", function(err, data) {
        if(err)
            next(err);
        else
            res.send("yay");
    });
});

Let's say that the file "some/file" does not actually exist. In this case fs.readFile will return an error as the first argument to the callback method. If you check for that and do next(err) when it happens, the default express error handler will take over and do whatever you make it do (e.g. return a 500 to the user). That's a graceful way to handle an error. Of course, if you forget to call next(err), it doesn't work.

So that's the error condition that a global handler can deal with, however consider another case:

app.get("/test", function(req, res, next) {
    require("fs").readFile("/some/file", function(err, data) {
        if(err)
            next(err);
        else {
            nullObject.someMethod(); //throws a null reference exception
            res.send("yay");
        }
    });
});

In this case, there is a bug if your code that results in you calling a method on a null object. Here an exception will be thrown, it will not be caught by the global error handler, and your node app will terminate. All clients currently executing requests on that service will get suddenly disconnected with no explanation as to why. Ungraceful.

There is currently no global error handler functionality in Node to handle this case. You cannot put a giant try/catch around all your express handlers because by the time your asyn callback executes, those try/catch blocks are no longer in scope. That's just the nature of async code, it breaks the try/catch error handling paradigm.

AFAIK, your only recourse here is to put try/catch blocks around the synchronous parts of your code inside each one of your async callbacks, something like this:

app.get("/test", function(req, res, next) {
    require("fs").readFile("/some/file", function(err, data) {
        if(err) {
            next(err);
        }
        else {
            try {
                nullObject.someMethod(); //throws a null reference exception
                res.send("yay");
            }
            catch(e) {
                res.send(500);
            }
        }
    });
});

That's going to make for some nasty code, especially once you start getting into nested async calls.

Some people think that what Node does in these cases (that is, die) is the proper thing to do because your system is in an inconsistent state and you have no other option. I disagree with that reasoning but I won't get into a philosophical debate about it. The point is that with Node, your options are lots of little try/catch blocks or hope that your test coverage is good enough so that this doesn't happen. You can put something like upstart or supervisor in place to restart your app when it goes down but that's simply mitigation of the problem, not a solution.

Node.js has a currently unstable feature called domains that appears to address this issue, though I don't know much about it.

1
  • Are there appropriate means to handle such errors, for example, in Python? Or Golang? Or because those languages are synchronous, they do not have these problems?
    – Green
    Aug 18, 2015 at 3:40
2

I've just put together a class which listens for unhandled exceptions, and when it see's one it:

  • prints the stack trace to the console
  • logs it in it's own logfile
  • emails you the stack trace
  • restarts the server (or kills it, up to you)

It will require a little tweaking for your application as I haven't made it generic as yet, but it's only a few lines and it might be what you're looking for!

Check it out!

Note: this is over 4 years old at this point, unfinished, and there may now be a better way - I don't know!)

process.on
(
    'uncaughtException',
    function (err)
    {
        var stack = err.stack;
        var timeout = 1;

        // print note to logger
        logger.log("SERVER CRASHED!");
        // logger.printLastLogs();
        logger.log(err, stack);


        // save log to timestamped logfile
        // var filename = "crash_" + _2.formatDate(new Date()) + ".log";
        // logger.log("LOGGING ERROR TO "+filename);
        // var fs = require('fs');
        // fs.writeFile('logs/'+filename, log);


        // email log to developer
        if(helper.Config.get('email_on_error') == 'true')
        {
            logger.log("EMAILING ERROR");
            require('./Mailer'); // this is a simple wrapper around nodemailer http://documentup.com/andris9/nodemailer/
            helper.Mailer.sendMail("GAMEHUB NODE SERVER CRASHED", stack);
            timeout = 10;
        }

        // Send signal to clients
//      logger.log("EMITTING SERVER DOWN CODE");
//      helper.IO.emit(SIGNALS.SERVER.DOWN, "The server has crashed unexpectedly. Restarting in 10s..");


        // If we exit straight away, the write log and send email operations wont have time to run
        setTimeout
        (
            function()
            {
                logger.log("KILLING PROCESS");
                process.exit();
            },
            // timeout * 1000
            timeout * 100000 // extra time. pm2 auto-restarts on crash...
        );
    }
);
3
  • What happened to your class on Github? It is giving Page not found error.
    – Raf
    Jan 24, 2016 at 14:12
  • 1
    @Raf sorry about that champ! Must have been deleted in some gist purge I've done over the last 4 years... Updated w/ what I hope is the correct version of the file :) Feb 11, 2016 at 7:20
  • Thanks for the update in your answer. Interesting to know that it can be done this way too. What I have done is, let winston take care of capturing the uncaughtExceptions and using its mail transport to send emails and used modules like pm2 or forever to do the restart of node instance.
    – Raf
    Feb 11, 2016 at 12:06
1

Had a similar problem. Ivo's answer is good. But how can you catch an error in a loop and continue?

var folder='/anyFolder';
fs.readdir(folder, function(err,files){
    for(var i=0; i<files.length; i++){
        var stats = fs.statSync(folder+'/'+files[i]);
    }
});

Here, fs.statSynch throws an error (against a hidden file in Windows that barfs I don't know why). The error can be caught by the process.on(...) trick, but the loop stops.

I tried adding a handler directly:

var stats = fs.statSync(folder+'/'+files[i]).on('error',function(err){console.log(err);});

This did not work either.

Adding a try/catch around the questionable fs.statSynch() was the best solution for me:

var stats;
try{
    stats = fs.statSync(path);
}catch(err){console.log(err);}

This then led to the code fix (making a clean path var from folder and file).

1
  • You should use the async version of stat, and then just deal with the error as you would do - you'd have to use some recursive loop rather than a for though. It has the same interface as readdir, which gives you back an error as an arg (incidentally, you're ignoring that in the snippet).
    – whitfin
    Oct 21, 2015 at 17:48
1

I found PM2 as the best solution for handling node servers, single and multiple instances

1
  • This is the best way. I think when you throw an exception, pm2 will catch it and track the exception to log. So your process will not exit unless you write code like process.exit(1);.
    – Lane
    Jan 9 at 9:08
0

One way of doing this would be spinning the child process and communicate with the parent process via 'message' event.

In the child process where the error occurs, catch that with 'uncaughtException' to avoid crashing the application. Mind that Exceptions thrown from within the event handler will not be caught. Once the error is caught safely, send a message like: {finish: false}.

Parent Process would listen to the message event and send the message again to the child process to re-run the function.

Child Process:

// In child.js
// function causing an exception
  const errorComputation = function() {

        for (let i = 0; i < 50; i ++) {
            console.log('i is.......', i);
            if (i === 25) {
                throw new Error('i = 25');
            }
        }
        process.send({finish: true});
}

// Instead the process will exit with a non-zero exit code and the stack trace will be printed. This is to avoid infinite recursion.
process.on('uncaughtException', err => {
   console.log('uncaught exception..',err.message);
   process.send({finish: false});
});

// listen to the parent process and run the errorComputation again
process.on('message', () => {
    console.log('starting process ...');
    errorComputation();
})

Parent Process:

// In parent.js
    const { fork } = require('child_process');

    const compute = fork('child.js');

    // listen onto the child process
    compute.on('message', (data) => {
        if (!data.finish) {
            compute.send('start');
        } else {
            console.log('Child process finish successfully!')
        }
    });

    // send initial message to start the child process. 
    compute.send('start'); 
1
  • Please explain your code in order to have a better understanding of your approach.
    – sirandy
    Oct 5, 2017 at 23:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.