19

We're using Django + Gunicorn + Nginx in our server. The problem is that after a while we see lot's of gunicorn worker processes that have became orphan, and a lot other ones that have became zombie. Also we can see that some of Gunicorn worker processes spawn some other Gunicorn workers. Our best guess is that these workers become orphans after their parent workers have died.
Why Gunicorn workers spawn child workers? Why do they die?! And how can we prevent this?
I should also mention that we've set Gunicorn log level to debug and still we don't see any thing significant, other than periodical log of workers number, which reports count of workers we wanted from it.

UPDATE This is the line we used to run gunicorn:

gunicorn --env DJANGO_SETTINGS_MODULE=proj.settings proj.wsgi --name proj --workers 10 --user proj --group proj --bind 127.0.0.1:7003 --log-level=debug --pid gunicorn.pid --timeout 600 --access-logfile /home/proj/access.log --error-logfile /home/proj/error.log

4
  • Can you post your Gunicorn config? It's hard to understand what might be going on without knowing how it's set up.
    – jnovinger
    Oct 31, 2015 at 23:41
  • 1
    any update on your question? did you come up with a solution or find out what was wrong? Nov 16, 2015 at 6:00
  • this was for long time ago, we changed from gunicorn to uWSGI a day or two after I asked this question and we didn't find a solution.
    – sazary
    Nov 16, 2015 at 12:35
  • you need a monitor, for that you might need to integrate supervisord to restart in case it fails. and pass all this gunicorn line to a .sh file.
    – Eddwin Paz
    Jan 9, 2020 at 3:25

3 Answers 3

1

In my case I deploy in Ubuntu servers (LTS releases, now almost are 14.04 LTS servers) and I never did have problems with gunicorn daemons, I create a gunicorn.conf.py and launch gunicorn with this config from upstart with an script like this in /etc/init/djangoapp.conf

description "djangoapp website"
start on startup
stop on shutdown
respawn
respawn limit 10 5

script
  cd /home/web/djangoapp
  exec /home/web/djangoapp/bin/gunicorn -c gunicorn.conf.py -u web -g web djangoapp.wsgi
end script

I configure gunicorn with a .py file config and i setup some options (details below) and deploy my app (with virtualenv) in /home/web/djangoapp and no problems with zombie and orphans gunicorn processes.

i verified your options, timeout can be a problem but another one is that you don't setup max-requests in your config, by default is 0, so, no automatic worker restart in your daemon and can generate memory leaks (http://gunicorn-docs.readthedocs.org/en/latest/settings.html#max-requests)

6
  • We were using supervisor, then when we saw the problem we thought it's happening because of supervisor. We removed it, and the problem remained.
    – sazary
    Dec 28, 2015 at 13:21
  • ok, so now my answers, are you using uwsgi and the problem remains? which one you using now instead of supervisor? Dec 29, 2015 at 1:10
  • we're using uwsgi's own daemonizer mechanism (without supervisor) and no, the problem doesn't remain.
    – sazary
    Dec 30, 2015 at 11:14
  • well, uwsgi is C-based, gunicorn is Python based, the only difference that i see with my case is the timeout, i use gunicorn default timeout (30 seconds) and i see that in your case is 10 minutes (600 seconds) maybe too much time for you applications (60/120 seconds maybe if you need more). uwsgi is more aggresive, --harakiri for timeout that force the kill python processes. Dec 30, 2015 at 14:19
  • this was an application that we used for internal uses, and it was heavily cpu intensive. so 10 minutes is something we needed (in some cases). and we set timeout in both cases to 10 minutes. if gunicorn breaks when we set it's timeout to 10 minutes, I call it a bug.
    – sazary
    Dec 30, 2015 at 15:33
1

We will use a .sh file to start the gunicorn process. Later you will use a supervisord configuration file. what is supervisord? some external know how information link about how to install supervisord with Django,Nginx,Gunicorn Here

gunicorn_start.sh remember to give chmod +x to the file.

#!/bin/sh
NAME="myDjango"
DJANGODIR="/var/www/html/myDjango"
NUM_WORKERS=3
echo "Starting myDjango -- Django Application"
cd $DJANGODIR
exec gunicorn -w $NUM_WORKERS $NAME.wsgi:application --bind 127.0.0.1:8001

mydjango_django.conf : Remember to install supervisord on your OS. and Copy this on the configuration folder.

[program:myDjango]
command=/var/www/html/myDjango/gunicorn_start.sh
user=root
autorestart=true
redirect_sderr=true

Later on use the command:

Reload the daemon’s configuration files, without add/remove (no restarts)

supervisordctl reread

Restart all processes Note: restart does not reread config files. For that, see reread and update.

supervisordctl start all

Get all process status info.

supervisordctl status 
1
  • This didn't help. Still could see those orphan processes. May 20, 2020 at 8:29
0

This sounds like a timeout issue.

You have multiple timeouts going on and they all need to be in a descending order. It seems they may not be.

For example:

  • Nginx has a default timeout of 60 seconds
  • Gunicorn has a default timeout of 30 seconds
  • Django has a default timeout of 300 seconds
  • Postgres default timeout is complicated but let's pose 60 seconds for this example.

In this example, when 30 seconds has passed and Django is still waiting for Postgres to respond. Gunicorn tells Django to stop, which in turn should tell Postgres to stop. Gunicorn will wait a certain amount of time for this to happen before it kills django, leaving the postgres process as an orphan query. The user will re-initiate their query and this time the query will take longer because the old one is still running.

I see that you have set your Gunicorn tiemeout to 300 seconds.

This would probably mean that Nginx tells Gunicorn to stop after 60 seconds, Gunicorn may wait for Django who waits for Postgres or any other underlying processes, and when Nginx gets tired of waiting, it kills Gunicorn, leaving Django hanging.

This is still just a theory, but it is a very common problem and hopefully leads you and any others experiencing similar problems, to the right place.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.