14

I am building a test crawler and wanted to know if Go (golang) caches DNS queries. I don't see anything about caching in the dnsclient. This seems like an important thing to add to any crawler to prevent lots of extra DNS queries.

Does Go (1.4+) cache DNS lookups?

If not, does debian/ubuntu/linux, windows, or darwin/OSX do any caching at the network level Go benefits from?

3
  • No, but it's quite common to use a local dns server to cache responses.
    – JimB
    Oct 26, 2016 at 13:14
  • 2
    You don't want Go to do this. Go the runtime / stdlib / language absolutely should not, and your app probably shouldn't either. Java does this and it's a pain and leads to all sorts of unexpected situations. How do you clear the cache without modifying or restarting the program? How do you know what's in the cache? How long is it there for? How do you troubleshoot it? It's opaque.
    – cbednarski
    May 5, 2019 at 4:37
  • There was a feature request github.com/golang/go/issues/24796 Oct 11, 2021 at 15:42

2 Answers 2

21

The answer to your question is no. There is no built-in dns caching in the std lib resolver. Would it be helpful? Maybe in some cases. Our org runs a local dns cache on each server and points resolv.conf there. So it wouldn't necessarily help us much to have caching in the language.

There are some solutions that could help you. This package seems to have a pretty good solution. From the snippet in their readme you could even do:

http.DefaultClient.Transport = &http.Transport {
  MaxIdleConnsPerHost: 64,
  Dial: func(network string, address string) (net.Conn, error) {
    separator := strings.LastIndex(address, ":")
    ip, _ := dnscache.FetchString(address[:separator])
    return net.Dial("tcp", ip + address[separator:])
  },
}

To enable it for all http requests from http.Get and friends.

3
  • 3
    In addition, many Linuxes now include local DNS caching so there is not much benefit to be had from extra caching within apps. Ubuntu is the obvious example. You can see this in action by e.g. dig www.google.com, which shows some comments that might include ;; SERVER: 127.0.1.1, this being your local DNS cache in operation.
    – Rick-777
    Nov 4, 2016 at 22:26
  • 1
    It's better to use net.SplitHostPort to split address. May 19, 2020 at 4:37
  • I believe that in-process cache will work more effectively. Also now I have a problem with leaking connections and on some point dns resolve stopped to work because can’t connect to localhost udp 53 Oct 11, 2021 at 14:19
5

The Go resolver does not do any in-process caching. While it would be possible to roll your own, your best bet is probably to run a system-wide DNS cache on each machine. (My favourite being dnsmasq.)

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.