0

General context

Docker daemon comes with an embedded DNS server. It resolves local Docker swarm and network records and forwards queries for external records to an upstream nameserver configured with --dns 1.

Docs say you can set an IP address for this upstream nameserver with --dns=[IP_ADDRESS...]. The default port used is 53.

My question

Can I configure the port used as well?

My host's /etc/docker/daemon.json shows "dns": ["10.99.0.1"],. Is there a way for me to specify something like "dns": ["10.99.0.1:53"], so that dockerd always knows to forward DNS queries to port 53?

My use case

In my case, 10.99.0.1 is the IP of a localhost bridge interface. I run a local DNS caching server on this host. So DNS queries sent to 10.99.0.1:53 work. But dockerd forwards queries originating from containers connected to user-defined bridge networks (created with docker network create) to non-standard ports it picks. See terminal output below.

Detailed terminal output and debugging info

"toogle" is a Docker container connected to a Docker network I created with docker network create. 127.0.0.11 is another loopback address. DNS queries originating from within Docker containers connected to user-defined Docker networks are destined for this IP.

Is Docker's embedded DNS server actually running?

DNS queries are routed by toogle's firewall rules this way.

$ sudo nsenter -n -t $(docker inspect --format {{.State.Pid}} toogle) iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 1 packets, 60 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 1 packets, 60 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER_OUTPUT  all  --  *      *       0.0.0.0/0            127.0.0.11

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER_POSTROUTING  all  --  *      *       0.0.0.0/0            127.0.0.11

Chain DOCKER_OUTPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            127.0.0.11           tcp dpt:53 to:127.0.0.11:37619   <-- look at this rule
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            127.0.0.11           udp dpt:53 to:127.0.0.11:58552   <-- look at this rule

Chain DOCKER_POSTROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 SNAT       tcp  --  *      *       127.0.0.11           0.0.0.0/0            tcp spt:37619 to::53
    0     0 SNAT       udp  --  *      *       127.0.0.11           0.0.0.0/0            udp spt:58552 to::53

Whatever's listening on those ports is accepting TCP and UDP connections

$ sudo nsenter -n -t $(docker inspect --format {{.State.Pid}} toogle) nc 127.0.0.11 37619 -vz

127.0.0.11: inverse host lookup failed: Host name lookup failure
(UNKNOWN) [127.0.0.11] 37619 (?) open

$ sudo nsenter -n -t $(docker inspect --format {{.State.Pid}} toogle) nc 127.0.0.11 58552 -vzu

127.0.0.11: inverse host lookup failed: Host name lookup failure
(UNKNOWN) [127.0.0.11] 58552 (?) open

But there's no DNS reply from either

$ sudo nsenter -n -t $(docker inspect --format {{.State.Pid}} toogle) dig @127.0.0.11 -p 58552 accounts.google.com

; <<>> DiG 9.11.3-1ubuntu1.14-Ubuntu <<>> @127.0.0.11 -p 58552 accounts.google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached


$ sudo nsenter -n -t $(docker inspect --format {{.State.Pid}} toogle) dig @127.0.0.11 -p 37619 accounts.google.com +tcp

; <<>> DiG 9.11.3-1ubuntu1.14-Ubuntu <<>> @127.0.0.11 -p 37619 accounts.google.com +tcp
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

dockerd is listening for DNS queries at that IP and port from within toogle.

$ sudo nsenter -n -p -t $(docker inspect --format {{.State.Pid}} toogle) ss -utnlp
Netid        State          Recv-Q         Send-Q                    Local Address:Port                    Peer Address:Port
udp          UNCONN         0              0                            127.0.0.11:58552                        0.0.0.0:*             users:(("dockerd",pid=10984,fd=38))
tcp          LISTEN         0              128                          127.0.0.11:37619                        0.0.0.0:*             users:(("dockerd",pid=10984,fd=40))
tcp          LISTEN         0              128                                   *:80                                 *:*             users:(("toogle",pid=12150,fd=3))

But dockerd is trying to forward the DNS query to 10.99.0.1, which is my docker0 bridge network interface.

$ sudo journalctl --follow -u docker
-- Logs begin at Tue 2019-11-05 18:17:27 UTC. --
Apr 22 15:43:12 my-host dockerd[10984]: time="2021-04-22T15:43:12.496979903Z" level=debug msg="[resolver] read from DNS server failed, read udp 172.20.0.127:37928->10.99.0.1:53: i/o timeout"
Apr 22 15:43:13 my-host dockerd[10984]: time="2021-04-22T15:43:13.496539033Z" level=debug msg="Name To resolve: accounts.google.com."
Apr 22 15:43:13 my-host dockerd[10984]: time="2021-04-22T15:43:13.496958664Z" level=debug msg="[resolver] query accounts.google.com. (A) from 172.20.0.127:51642, forwarding to udp:10.99.0.1"

dockerd forwards the DNS query that asks for nameserver 127.0.0.11:58552 to 10.99.0.1 but only changes the IP and not the port. So the DNS query is forwarded to 10.99.0.1:58552 and nothing is listening at that port.

$ dig @10.99.0.1 -p 58552 accounts.google.com
[NO RESPONSE]

$ nc 10.99.0.1 58552  -vz
10.99.0.1: inverse host lookup failed: Unknown host
(UNKNOWN) [10.99.0.1] 58552 (?) : Connection refused

A DNS query to 10.99.0.1:53 works as expected.

dig @10.99.0.1 -p 53 accounts.google.com

; <<>> DiG 9.11.3-1ubuntu1.14-Ubuntu <<>> @10.99.0.1 -p 53 accounts.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53674
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;accounts.google.com.       IN  A
;; ANSWER SECTION:
accounts.google.com.    235 IN  A   142.250.1.84
;; Query time: 0 msec
;; SERVER: 10.99.0.1#53(10.99.0.1)
;; WHEN: Thu Apr 22 17:20:09 UTC 2021
;; MSG SIZE  rcvd: 64

1 Answer 1

-1

I don't think there's a way to do this. I also misread the output. Docker daemon was forwarding to port 53.

read udp 172.20.0.127:37928->10.99.0.1:53: i/o timeout

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.