1

Recently, I've started to experience an issue on my Django-based site where logins will fail with no errors reported to the user after a few days of site uptime. Already logged in sessions continue to work fine, but no new logins can happen.

The relevant info:

  • I'm using the normal django.contrib.auth authentication stuff

  • I am using PostgreSQL for the DB via the django.db.backends.postgresql_psycopg2 backend

  • I am running on OSX 10.6.7 with Python 2.6.1 and Django 1.3

  • Django is running in FastCGI mode behind nginx

My gut feeling is that there is something breaking down in the connection / socket to the DB at some point, because if I kill Django and restart it, everything works just fine again (i.e. the DB itself is definitely not overloaded and can be accessed just fine using the psql commandline tool).

Unfortunately, there is nothing in the logs about the error (well, at least nothing is being emitted via the normal Python logging module which is how I trap all my logs) and there are no errors reported to the web browser. All the client sees is that they get sent back to the login page again, as if they had just refreshed their browser.

Any help much appreciated.

Not sure if it's relevant, but my middleware classes are:

MIDDLEWARE_CLASSES = (
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.transaction.TransactionMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
)

UPDATE

After looking into the nginx access logs, I can see that the login actually works briefly then suddenly doesn't work:

"POST /accounts/login/ HTTP/1.1" 302 5 "https://myapp.com/accounts/login/?next=/orders"
"GET /orders HTTP/1.1" 301 185 "-"
"GET /orders HTTP/1.1" 302 5 "-"
"GET /accounts/login/?next=/orders HTTP/1.1" 301 185 "-"
"GET /accounts/login/?next=/orders HTTP/1.1" 200 1297 "-"

As you can see, the login works and the client is redirected to the 'next' URL (/orders), but then the third line redirects (302) the client back out to the login page, presumably because the @login_required decorator (which is applied to the /orders controller) determined that they weren't really logged in.

For comparison, this is a successful login sequence:

"POST /accounts/login/ HTTP/1.1" 302 5 "https://myapp.com/accounts/login/?next=/orders"
"GET /orders HTTP/1.1" 301 185 "-"
"GET /orders HTTP/1.1" 200 59364 "-"

And a login with the wrong password (the POST comes back with a 200 instead of a 302):

"POST /accounts/login/ HTTP/1.1" 200 1426 "https://myapp.com/accounts/login/?next=/orders"

The difference between a normal login and the broken one is that the client gets a 200 OK for /orders instead of 302 back to the login page. I have no idea how the auth middleware can allow the login and then kick the user back out immediately after. Is there a possible race condition here where the login controller hasn't been able to persist the logged-in state into the DB in time for the /orders controller to see it and allow the user to stay logged in?

Also - I have noticed that a Django restart isn't necessarily required to fix the issue - sometimes the server just miraculously starts letting clients log in again.

5
  • What web server are you using? Have you checked the logs for the web server? Jun 3, 2011 at 14:35
  • I'm running Django in FastCGI mode with nginx sitting in front of it. Since existing sessions still work and other webapps that are sitting behind the same nginx instance allow logins at the same time that this site is silently failing logins, I don't think it has anything to do with the web proxy that sits in front of the site.
    – glenc
    Jun 3, 2011 at 15:50
  • Well, the fact that killing Django and restarting it fixes the issue seems to suggest otherwise. If it were a DB issue as you believe, you'd have to restart or otherwise mess with the DB to make it work again. Check the logs for the web server, and in the future try to piece meal the solution, so you can narrow it down more. For instance, if you're running memcached, try just restarting that and leave everything else alone. You also might want to try just restarting the DB to see if that's enough to correct the issue before rebooting the whole server. Jun 3, 2011 at 17:17
  • Hey Chris - thanks for the response. Yeah, I mean that I don't think it's an issue with the DB server itself, I thought maybe Django's connection to the server was dropping or something. Anyway, I just determined that that's not the case through putting in a debug method that tries to touch the DB. The server just went into "broken logins" mode but the debug controller could still pull stuff from the DB. Just went through the nginx access and error logs and not seeing anything there, and I'm not running memcached either ... :(
    – glenc
    Jun 3, 2011 at 17:35
  • I guess the semi-consolation is that it is happening in less than "days" now :/
    – glenc
    Jun 3, 2011 at 17:40

1 Answer 1

1

Sounds like your web server is using persistent connections and running out. What do the pg logs say when you can't log in?

1
  • Hi scott - thanks for the tip. You were right. Django was blowing the connection limit to postgresql but not showing connection errors in the logs for some reason. I spent a couple hours looking into pgpool-II and getting it set up and now the issue has gone away. Thank god!
    – glenc
    Jun 4, 2011 at 19:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.