24

There are lots of ways in which Docker containers can get confused about DNS settings (just search SO or the wider internet for "Docker DNS" to see what I mean), and one of the common workarounds suggested is to:

  1. Set up dnsmasq as a local DNS resolver on the host system
  2. Bind it to the docker0 network interface
  3. Configure Docker to use the docker0 IP address for DNS resolution

However, attempting to apply this workaround naively on many modern Linux systems will send you down a rabbithole of Linux networking and process management complexity, as systemd assures you that dnsmasq isn't running, but netstat tells you that it is, and actually attempting to start dnsmasq fails with the complaint that port 53 is already in use.

So, how do you reliably give your containers access to a local resolver running on the host, even if the system already has one running by default?

2
  • This reads like a statement, or even a complaint, rather than like a question. What are you looking for people to tell you?
    – mc0e
    Jun 30, 2016 at 13:28
  • 1
    I wasn't looking for them to tell me anything - I'd already answered my own question, and wanted to share the results of that as described in blog.stackoverflow.com/2011/07/… I didn't give the title the Jeopardy treatment though, so I'll do that now :)
    – ncoghlan
    Jul 8, 2016 at 4:35

2 Answers 2

27

The problem here is that many modern Linux systems run dnsmasq implicitly, so what you're now aiming to do is to set up a second instance specifically for Docker to use. There are actually 3 settings needed to do that correctly:

  • --interface=docker0 to listen on the default Docker network interface
  • --except-interface=lo to skip the implicit addition of the loopback interface
  • --bind-interfaces to turn off a dnsmasq feature where it still listens on all interfaces by default, even when its only processing traffic for one of them

Setting up a dedicated dnsmasq instance

Rather than changing the settings of the default system wide dnsmasq instance, these instructions show setting up a dedicated dnsmasq instance with systemd, on a system which already defines a default dnsmasq service:

$ sudo cp /usr/lib/systemd/system/dnsmasq.service /etc/systemd/system/dnsmasq-docker.service
$ sudoedit /etc/systemd/system/dnsmasq-docker.service

First, we copy the default service settings to a dedicated service file. We then edit that service file, and look for the service definition section, which should be something like this:

[Service]
ExecStart=/usr/sbin/dnsmasq -k

We edit that section to define our additional options:

[Service]
ExecStart=/usr/sbin/dnsmasq -k --interface=docker0 --except-interface=lo --bind-interfaces

The entire file is actually pretty short:

[Unit]
Description=DNS caching server.
After=network.target
After=docker.service
Wants=docker.service

[Service]
ExecStart=/usr/sbin/dnsmasq -k --interface=docker0 --except-interface=lo --bind-interfaces

[Install]
WantedBy=multi-user.target

The [Unit] section tells systemd to wait until after both the network stack and the main docker daemon are available to start this service, while [Install] indicates which system state target to add the service to when enabling it.

We then configure our new service to start on system boot, and also start it explicitly for immediate use:

$ sudo systemctl enable dnsmasq-docker
$ sudo systemctl start dnsmasq-docker

As the final step in getting the service running, we check it has actually started as expected:

$ sudo systemctl status dnsmasq-docker

The two key lines we're looking for in that output are:

Loaded: loaded (/etc/systemd/system/dnsmasq-docker.service; enabled; vendor preset: disabled)
Active: active (running) since <date & time>

On the first line, note the "enabled" status, while on the second, the "active (running)" status. If the service hasn't started correctly, then the additional diagnostic information will hopefully explain why (although it can be unfortunately cryptic at times, hence this post).

Note: This configuration may fail to start dnsmasq-docker on system restart with an error about the docker0 interface not being defined. While waiting for docker.service seems to be pretty reliable in avoiding that problem, if name resolution from docker containers isn't working after a system restart, then try running:

$ sudo systemctl start dnsmasq-docker

Configuring the host firewall

To be able to use the resolver from local Docker containers, we also need to drop the network firewall between the host and systems running in containers:

sudo firewall-cmd --permanent --zone=trusted --change-interface=docker0
sudo firewall-cmd --reload

(This would be an absolutely terrible idea on a production container host, but can be a helpful risk-vs-convenience trade-off on a developer workstation)

Configuring Docker using a systemd environment file

Now that we have our local resolver running, we need to configure Docker to use it by default. Docker needs the IP address of the docker0 interface rather than the interface name, so we use ifconfig to retrieve that:

$ ifconfig docker0 | grep inet
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0

So, for my system, the host's interface on the default docker0 bridge is accessible as 172.17.0.1 (Appending | cut -f 10 -d ' ' to that command should filter the output to just the IP address)

Since I'm assuming a systemd-based Linux with a system provided Docker package, we'll query the system package's service file to find out how the service is being started:

$ cat /usr/lib/systemd/system/docker.service

The first thing we're looking for is the exact command used to start the daemon, which should look something like this:

ExecStart=/usr/bin/docker daemon \
          $OPTIONS \
          $DOCKER_STORAGE_OPTIONS \
          $DOCKER_NETWORK_OPTIONS \
          $INSECURE_REGISTRY

The second part we're looking for is whether or not the service is configured to use an environment file, as indicated by one of more lines like this:

EnvironmentFile=-/etc/sysconfig/docker

When an environment file is in use (as it is on Fedora 23), then the way to change the Docker daemon settings is to edit that file and update the relevant environment variable:

$ sudoedit /etc/sysconfig/docker

The existing OPTIONS entry on Fedora 23 looks like this:

OPTIONS='--selinux-enabled --log-driver=journald'

To change the default DNS resolution settings, we amend it to look like this:

OPTIONS='--selinux-enabled --log-driver=journald --dns=172.17.0.1'

And then restart the Docker daemon:

$ sudo systemctl restart docker

With this change implemented, Docker containers should now be reliably able to access any systems your host system can access (including via VPN tunnels, which was my own reason for needing to figure this out)

You can run curl inside a container to check name resolution is working correctly:

docker run -it centos curl google.com

Replace google.com with whichever hostname was giving you problems (as you should have only ended up finding this answer if you had a name resolution problem when running a process inside a Docker container)

Configuring Docker using a systemd drop-in file

(Caveat: since my system uses an environment file, I haven't been able to test the drop-in file based approach below, but it should work - I've included it since the Docker documentation seems to indicate they now prefer the use of systemd drop-in files to the use of environment files)

If the system service file doesn't use EnvironmentFile, then the entire ExecStart entry can be replaced by using a drop-in configuration file:

$ sudo mkdir -p /etc/systemd/system/docker.service.d
$ sudoedit /etc/systemd/system/docker.service.d/daemon.conf

We then tell Docker to clear the existing ExecStart entry and replace it with our new one with the additional settings:

[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon \
          $OPTIONS \
          --dns 172.17.0.1 \
          $DOCKER_STORAGE_OPTIONS \
          $DOCKER_NETWORK_OPTIONS \
          $INSECURE_REGISTRY

We then tell systemd to load that configuration change and restart Docker:

$ sudo systemctl daemon-reload
$ sudo systemctl restart docker

References:

8
  • I'd like to avoid putting the docker0 interface in the trusted firewalld zone, but my initial attempt to instead add it to the internal zone and add the dns service to that zone resulted in a configuration where ping worked inside a container, but curl didn't.
    – ncoghlan
    Feb 29, 2016 at 6:35
  • It also looks like there may be a potential conflict between the use of firewalld here and the way docker-compose expects iptables to be configured. Updating/reinstalling the docker packages fixed that for me without breaking the local resolver setup.
    – ncoghlan
    Feb 29, 2016 at 7:39
  • Curiouser & curiouser - this setup is working fine with fedora & centos based containers, but debian & ubuntu containers aren't able to see the outside world at all.
    – ncoghlan
    Mar 1, 2016 at 7:34
  • Ah, it looks like there's a bug in the current instructions where dnsmasq-docker may fail to come up after a system reboot (complaining about "docker0" being missing). It just so happened that the images I was running today were debian based, so I misdiagnosed the source of the problem.
    – ncoghlan
    Mar 1, 2016 at 7:49
  • 1
    ifconfig is deprecated by now, if anyone else is coming across this and wonders what the corresponding command using ip is, here you go: ip -c addr show docker0 (-c for coloring).
    – swenzel
    Jun 18, 2020 at 7:44
8

You can use the host's local DNS resolver (e.g. dnsmasq) from your Docker containers if they are on a custom network. In that case a container's /etc/resolv.conf will have the nameserver 127.0.0.11 (a.k.a. the Docker's embedded DNS server), which can forward DNS requests to the host's loopback address properly.

$ cat /etc/resolv.conf
nameserver 127.0.0.1
$ docker run --rm alpine cat /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4
$ docker network create demo
557079c79ddf6be7d6def935fa0c1c3c8290a0db4649c4679b84f6363e3dd9a0
$ docker run --rm --net demo alpine cat /etc/resolv.conf
nameserver 127.0.0.11
options ndots:0    

If you use docker-compose, it will set up a custom network for your services automatically (with a file format v2+). Note, however, that while docker-compose runs containers in a user-defined network, it still builds them in the default bridge network. To use a custom network for builds you can specify the network parameter in the build configuration (requires file format v3.4+).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.