14

Running nginx 1.9.* / PHP 7.0.* (but exact same behavior in 5.6.* also)

Attempting to gracefully stop a PHP-FPM / nginx combo for node shutdown during maintenance. To do this, I'm sending the SIGQUIT to php-fpm, which should provide a graceful shutdown.

To test this, I made a dumb script

<?php sleep(5); echo 'done';

Tested locally with the following curl

curl -I x.x.x.x:8080

Which normally produces the output:

HTTP/1.1 200 OK
Server: nginx
Date: Tue, 12 Apr 2016 04:48:00 GMT
Content-Type: text/html; charset=UTF-8
Connection: close

Desired: in the middle of any in-flight request, when a graceful shutdown is requested, the current requests should finish, but any additional requests should fail.

Unfortunately, when I try to trigger this behavior, by sending a SIGQUIT (http://manpages.ubuntu.com/manpages/precise/man8/php5-fpm.8.html) to the PHP-FPM master process:

kill -s SIGQUIT $FPMPID

The connection immediately drops, resulting in an ngnix 502

HTTP/1.1 502 Bad Gateway
Server: nginx
Date: Tue, 12 Apr 2016 04:48:07 GMT
Content-Type: text/html
Content-Length: 166
Connection: close

Any advice? I would love to make this piece of the system as seamless as possible. Thanks!

3
  • Attached link is fairly dead (503), process signals summarized here: forum.nginx.org/read.php?3,3485,template=head%3F%3F. Commented Apr 12, 2016 at 20:08
  • 3
    Yep. bugs.php.net/bug.php?id=41593 and, for me even important-er, bugs.php.net/bug.php?id=60961. I don't know how exactly (and if) they're related, but PHP-FPM unfortunately has its little annoyances.
    – Smuuf
    Commented Apr 13, 2016 at 21:38
  • I can suggest little workaround to you. It's not fix php-fpm, but help to answer 200 :) All you need is to set up upstream in nginx config with 2-3 backends (it can be same php-fpm pool few times). So, if one of requests fail, nginx will try to request backend again. Commented Sep 7, 2016 at 22:53

2 Answers 2

4

After struggling with this same situation for a while, I believe I've found the magical config setting to make child processes finish handling requests before dying.

http://php.net/manual/en/install.fpm.configuration.php#process-control-timeout

process_control_timeout

Time limit for child processes to wait for a reaction on signals from master

Basically, by setting this to something like 10s, the child process will wait that long, while handling existing requests before quitting.

Unfortunately, it seems that the php-fpm master process exits immediately, so, inspired by the code here, I wrote a wrapper script:

#!/bin/bash

PHP_FPM_PID='/php-fpm.pid'

wait_for_pid () {
    try=0

    while test $try -lt 35 ; do
        if [ ! -f "$1" ] ; then
            try=''
            break
        fi

        echo -n .
        try=`expr $try + 1`
        sleep 1
    done
}

function clean_up {

    echo "Killing $(cat $PHP_FPM_PID)"

    kill -QUIT `cat $PHP_FPM_PID`
    wait_for_pid $PHP_FPM_PID

    echo "Done!"

    exit 0
}

trap clean_up EXIT

nohup php-fpm --daemonize --pid $PHP_FPM_PID 2>&1 &

while true; do sleep 1; done
# ^ do nothing forever

which waits 35 seconds or until that pid file has been removed (presumably by one of the child processes? I'm still unclear on how it's removed).

Regardless, this wrapper script works well as the CMD for our php-fpm docker container that we're running with Kubernetes.

3
  • 3
    Seems like a useful setting, but does this mean that it always waits X seconds before responding? I've sort of reversed this process for my docker container (github.com/bryanlatten/docker-php), where instead of immediately signaling fpm to gracefully shut down, I gracefully shut down its forward nginx proxy - which will be graceful and wait for connections to complete. In theory, with no connections coming in, and outgoing connections waiting for close, PHP can then terminate like an elephant and nothing in flight should be affected. Does this make sense? Commented Dec 22, 2016 at 19:03
  • Absolutely, I think using nginx as a graceful-shutdown-proxy here is the right solution. It won't work for us unfortunately, but I think that's the way to go in the future. Commented Dec 23, 2016 at 15:51
  • "Unfortunately, it seems that the php-fpm master process exits immediately" What nonsense is this. Master process cannot be killed wihout children processes with it
    – gadelat
    Commented Aug 28, 2019 at 12:46
0

Nginx also reacts on SIGWINCH to gracefully shutdown workers. So this could be used to stop nginx before php-fpm. In my case during deployment I got 502 errors because php-fpm instantly stopped processing request, but nginx had some in request to process queue. It was brief period of time but enough to generate errors under the load. Just setting stop_signal: SIGWINCH in docker-compose.yml help to solve this problem. Of course it is possible to create custom script that traps SIGWINCH and after NGINX stops gracefully kills other processes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.