0

I've graphite server and intermittently its daemon fails with this error,

HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'datapoints' not found on 'http://192.168.12.15:8000/render/?target=sys.example_com.snmp.if_octets-Gi0_0_0.rx&format=json&from=-5min' - 723 bytes in 0.002 second response time |time=0.002067s;;;0.000000 size=723B;;;0

So I restarted apache which fixed the issue. But I would like to fix the root cause of this issue, I can't restart apache every time. I want to fix this permanently. This is what I found in error logs,

[Tue Aug 01 20:16:01] [wsgi:error] RuntimeError: populate() isn't reentrant
[Tue Aug 01 20:16:21] [wsgi:error] Target WSGI script '/usr/lib/python2.7/dist-packages/graphite/wsgi.py' cannot be loaded as Python module.
[Tue Aug 01 20:16:21] [wsgi:error] Exception occurred processing WSGI script '/usr/lib/python2.7/dist-packages/graphite/wsgi.py'.
[Tue Aug 01 20:16:21] [wsgi:error] Traceback (most recent call last):
[Tue Aug 01 20:16:21] [wsgi:error] File "/usr/lib/python2.7/dist-packages/graphite/wsgi.py", line 14, in <module>
[Tue Aug 01 20:16:21] [wsgi:error] application = get_wsgi_application()
[Tue Aug 01 20:16:21] [wsgi:error] File "/usr/lib/python2.7/dist-packages/django/core/wsgi.py", line 14, in get_wsgi_application
[Tue Aug 01 20:16:21] [wsgi:error] django.setup()
[Tue Aug 01 20:16:21] [wsgi:error] File "/usr/lib/python2.7/dist-packages/django/__init__.py", line 21, in setup
[Tue Aug 01 20:16:21] [wsgi:error] apps.populate(settings.INSTALLED_APPS)
[Tue Aug 01 20:16:21] [wsgi:error] File "/usr/lib/python2.7/dist-packages/django/apps/registry.py", line 78, in populate

This is my /usr/lib/python2.7/dist-packages/graphite/wsgi.py file,

import os
import sys

try:
    from importlib import import_module
except ImportError:
    from django.utils.importlib import import_module

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'graphite.settings')  # noqa

from django.conf import settings
from django.core.wsgi import get_wsgi_application

application = get_wsgi_application()

try:
    import whitenoise
except ImportError:
    whitenoise = False
else:
    # WhiteNoise < 2.0.1 does not support Python 2.6
    if sys.version_info[:2] < (2, 7):
        whitenoise_version = tuple(map(
                int, getattr(whitenoise, '__version__', '0').split('.')))
        if whitenoise_version < (2, 0, 1):
            whitenoise = False

if whitenoise:
    from whitenoise.django import DjangoWhiteNoise
    application = DjangoWhiteNoise(application)
    prefix = "/".join((settings.URL_PREFIX.strip('/'), 'static'))
    for directory in settings.STATICFILES_DIRS:
        application.add_files(directory, prefix=prefix)
    for app_path in settings.INSTALLED_APPS:
        module = import_module(app_path)
        directory = os.path.join(os.path.dirname(module.__file__), 'static')
        if os.path.isdir(directory):
            application.add_files(directory, prefix=prefix)

Update

Here is my config file for graphite which I'm using,

<VirtualHost *:8000>
  ServerName graphite-web

  ## Vhost docroot
  DocumentRoot "/var/www"
  ## Alias declarations for resources outside the DocumentRoot
  Alias /static "/var/lib/graphite/webapp/content"

  ## Directories, there should at least be a declaration for /var/www

  <Directory "/var/www">
    Options Indexes FollowSymLinks MultiViews
    AllowOverride None
    Require all granted
  </Directory>

  ## Logging
  ErrorLog "/var/log/apache2/graphite-web_error.log"
  ServerSignature Off
  CustomLog "/var/log/apache2/graphite-web_access.log" combined
  WSGIScriptAlias / "/usr/lib/python2.7/dist-packages/graphite/wsgi.py"
</VirtualHost>

What is causing this problem ? Any help on this please ?

1 Answer 1

3

You need to go back and find the error in the logs before that. That is, the one for the very first request against the process, or when WSGI script was loaded if pre-loading has been enabled. The error you give isn't the root cause, but the result of subsequent requests failing after the first and is because Django initialisation is no longer re-entrant when it fails the first time.

If it is a transient issue, you should ensure you are using mod_wsgi daemon mode and set startup-timeout=15 on WSGIDaemonProcess as way to recover automatically from the transient issue. One cause of a transient failure is a database not being available when Apache first starts.

You will need to make sure you have a recent mod_wsgi version though and not the ancient versions that Linux distros ship to get that startup-timeout option.

I would also suggest you add to the question the mod_wsgi configuration you are using so can comment on whether anything else you should do differently.

8
  • thanks for the info. Added config. Hope that is more clear.
    – Rahul
    Commented Aug 1, 2017 at 12:35
  • and to answer this "You need to go back and find the error in the logs before that", I've double-checked the logs. I found these logs repetitive.
    – Rahul
    Commented Aug 1, 2017 at 12:36
  • 1
    That error can only occur if Django initialisation is called more than once. If there is no prior error, the only other possibility can think of is that it is caused by the WSGI script file being updated all the time, causing it to be reloaded. You should really use daemon mode of mod_wsgi. Embedded mode as you are using is not recommended, and could cause it. If you have some system which is always causing WSGI script file to be updated, then add WSGIScriptReloading Off and resort to manual restarts on code changes. Commented Aug 1, 2017 at 12:47
  • I would be very grateful if you could share some info on how to "use daemon mode of mod_wsgi" ?
    – Rahul
    Commented Aug 1, 2017 at 12:52
  • Have you checked any documentation at all. Even Django documentation has example of using daemon mode. modwsgi.readthedocs.io/en/develop/user-guides/… docs.djangoproject.com/en/1.11/howto/deployment/wsgi/modwsgi/… modwsgi.readthedocs.io/en/develop/user-guides/… Commented Aug 1, 2017 at 21:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.