14

Running Python on a Django project which communicates with various web-services, we have an issue that occasionally requests are taking around 5 seconds instead of their usual < 100 ms.

I've narrowed this down to time taken in the socket.getaddrinfo function - this is being called by requests when we connect to external services, but it also appears to effect the default Django connection to the Postgres database box in the cluster. When we restart uwsgi after a deployment the first requests that come in will take 5 seconds to send a response. I also believe that our celery tasks are taking 5 seconds on a regular basis, but I've not added statsd timer tracking to them yet.

I've written some code to reproduce the issue:

import socket
import timeit

def single_dns_lookup():
    start = timeit.default_timer()
    socket.getaddrinfo('stackoverflow.com', 443)
    end = timeit.default_timer()
    return int(end - start)

timings = {}

for _ in range(0, 10000):
    time = single_dns_lookup()
    try:
        timings[time] += 1
    except KeyError:
        timings[time] = 1

print timings

Typical results are {0: 9921, 5: 79}

My colleague has already pointed to potential issues around ipv6 lookup times and has added this to the /etc/gai.conf:

precedence ::ffff:0:0/96  100

This has definitely improved lookups from non-Python programs such as curl which we use, but not from Python itself. The server boxes are running Ubuntu 16.04.3 LTS and I'm able to reproduce this on a vanilla VM with Python 2.

What steps can I take to improve the performance of all Python lookups so that they can take < 1s?

12
  • 1
    it sounds like your dns resolver is slow, give ncsd a try?
    – georgexsh
    Oct 19, 2017 at 18:47
  • 1
    @georgexsh I'm not sure how nscd would help - the first request that is not cached would still take 5s. And after the cached value expires then the first request that goes through the resolver would take 5s again. This would just reduce the percentage of slow requests and not remove them entirely right?
    – jamesc
    Oct 23, 2017 at 11:28
  • 1
    Have you tried to disable the IPv6 stack altogether with sysctl net.ipv6.conf.all.disable_ipv6=1, and check if that fixes your issue ? If yes, then most probably your python is linked to a version of glibc (or statically compiled against) which does not respect gai.conf.
    – zeppelin
    Oct 26, 2017 at 14:23
  • 1
    can't reproduce it locally in vagrant ubuntu/xenial64 Oct 26, 2017 at 19:26
  • 1
    Consider using collections.Counter instead of rolling your own with KeyError
    – claymation
    Oct 30, 2017 at 14:46

3 Answers 3

10
+75

5s is a default timeout to DNS lookup.

You can lower that.

Your real problem is probably (silent) UDP packets drop on the network though.

Edit: Experiment with resolution over TCP. Never done that. Might help you.

4

There are two things that can be done. One is that you don't query the IPV6 address, this can be done by monkey patching getaddrinfo

orig_getaddrinfo = socket.getaddrinfo

def _getaddrinfo(host, port, family=0, type=0, proto=0, flags=0):
    return orig_getaddrinfo(host, port, socket.AF_INET, type, proto, flags)

socket.getaddrinfo = _getaddrinfo

Next you can also use a ttl based cache to cache the result. You can use cachepy package for the same.

from cachetools import cached
import socket
import timeit
from cachepy import *
# or from cachepy import Cache

cache_with_ttl = Cache(ttl=600) # ttl given in seconds

orig_getaddrinfo = socket.getaddrinfo

# @cached(cache={})
@cache_with_ttl
def _getaddrinfo(host, port, family=0, type=0, proto=0, flags=0):
    return orig_getaddrinfo(host, port, socket.AF_INET, type, proto, flags)

socket.getaddrinfo = _getaddrinfo

def single_dns_lookup():
    start = timeit.default_timer()
    socket.getaddrinfo('stackoverflow.com', 443)
    end = timeit.default_timer()
    return int(end - start)

timings = {}

for _ in range(0, 10000):
    time = single_dns_lookup()
    try:
        timings[time] += 1
    except KeyError:
        timings[time] = 1

print (timings)
2

I would first try to understand the root cause of the slowness before building a cache or monkeypatching socket.getaddrinfo. Are your nameservers configured correctly in /etc/resolv.conf? Are you seeing packet loss on the network?

If you're experiencing loss that's outside your control, running a caching server (nscd) will mask but not completely eliminate the problem.

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.