General context
Docker daemon comes with an embedded DNS server. It resolves local Docker swarm and network records and forwards queries for external records to an upstream nameserver configured with --dns 1.
Docs say you can set an IP address for this upstream nameserver with --dns=[IP_ADDRESS...]
. The default port used is 53.
My question
Can I configure the port used as well?
My host's /etc/docker/daemon.json
shows "dns": ["10.99.0.1"],
. Is there a way for me to specify something like "dns": ["10.99.0.1:53"]
, so that dockerd always knows to forward DNS queries to port 53?
My use case
In my case, 10.99.0.1 is the IP of a localhost bridge interface. I run a local DNS caching server on this host. So DNS queries sent to 10.99.0.1:53 work. But dockerd forwards queries originating from containers connected to user-defined bridge networks (created with docker network create
) to non-standard ports it picks. See terminal output below.
Detailed terminal output and debugging info
"toogle" is a Docker container connected to a Docker network I created with docker network create
. 127.0.0.11 is another loopback address. DNS queries originating from within Docker containers connected to user-defined Docker networks are destined for this IP.
Is Docker's embedded DNS server actually running?
DNS queries are routed by toogle's firewall rules this way.
$ sudo nsenter -n -t $(docker inspect --format {{.State.Pid}} toogle) iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 1 packets, 60 bytes)
pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 1 packets, 60 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER_OUTPUT all -- * * 0.0.0.0/0 127.0.0.11
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER_POSTROUTING all -- * * 0.0.0.0/0 127.0.0.11
Chain DOCKER_OUTPUT (1 references)
pkts bytes target prot opt in out source destination
0 0 DNAT tcp -- * * 0.0.0.0/0 127.0.0.11 tcp dpt:53 to:127.0.0.11:37619 <-- look at this rule
0 0 DNAT udp -- * * 0.0.0.0/0 127.0.0.11 udp dpt:53 to:127.0.0.11:58552 <-- look at this rule
Chain DOCKER_POSTROUTING (1 references)
pkts bytes target prot opt in out source destination
0 0 SNAT tcp -- * * 127.0.0.11 0.0.0.0/0 tcp spt:37619 to::53
0 0 SNAT udp -- * * 127.0.0.11 0.0.0.0/0 udp spt:58552 to::53
Whatever's listening on those ports is accepting TCP and UDP connections
$ sudo nsenter -n -t $(docker inspect --format {{.State.Pid}} toogle) nc 127.0.0.11 37619 -vz
127.0.0.11: inverse host lookup failed: Host name lookup failure
(UNKNOWN) [127.0.0.11] 37619 (?) open
$ sudo nsenter -n -t $(docker inspect --format {{.State.Pid}} toogle) nc 127.0.0.11 58552 -vzu
127.0.0.11: inverse host lookup failed: Host name lookup failure
(UNKNOWN) [127.0.0.11] 58552 (?) open
But there's no DNS reply from either
$ sudo nsenter -n -t $(docker inspect --format {{.State.Pid}} toogle) dig @127.0.0.11 -p 58552 accounts.google.com
; <<>> DiG 9.11.3-1ubuntu1.14-Ubuntu <<>> @127.0.0.11 -p 58552 accounts.google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
$ sudo nsenter -n -t $(docker inspect --format {{.State.Pid}} toogle) dig @127.0.0.11 -p 37619 accounts.google.com +tcp
; <<>> DiG 9.11.3-1ubuntu1.14-Ubuntu <<>> @127.0.0.11 -p 37619 accounts.google.com +tcp
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
dockerd is listening for DNS queries at that IP and port from within toogle.
$ sudo nsenter -n -p -t $(docker inspect --format {{.State.Pid}} toogle) ss -utnlp
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
udp UNCONN 0 0 127.0.0.11:58552 0.0.0.0:* users:(("dockerd",pid=10984,fd=38))
tcp LISTEN 0 128 127.0.0.11:37619 0.0.0.0:* users:(("dockerd",pid=10984,fd=40))
tcp LISTEN 0 128 *:80 *:* users:(("toogle",pid=12150,fd=3))
But dockerd is trying to forward the DNS query to 10.99.0.1, which is my docker0 bridge network interface.
$ sudo journalctl --follow -u docker
-- Logs begin at Tue 2019-11-05 18:17:27 UTC. --
Apr 22 15:43:12 my-host dockerd[10984]: time="2021-04-22T15:43:12.496979903Z" level=debug msg="[resolver] read from DNS server failed, read udp 172.20.0.127:37928->10.99.0.1:53: i/o timeout"
Apr 22 15:43:13 my-host dockerd[10984]: time="2021-04-22T15:43:13.496539033Z" level=debug msg="Name To resolve: accounts.google.com."
Apr 22 15:43:13 my-host dockerd[10984]: time="2021-04-22T15:43:13.496958664Z" level=debug msg="[resolver] query accounts.google.com. (A) from 172.20.0.127:51642, forwarding to udp:10.99.0.1"
dockerd forwards the DNS query that asks for nameserver 127.0.0.11:58552 to 10.99.0.1 but only changes the IP and not the port. So the DNS query is forwarded to 10.99.0.1:58552 and nothing is listening at that port.
$ dig @10.99.0.1 -p 58552 accounts.google.com
[NO RESPONSE]
$ nc 10.99.0.1 58552 -vz
10.99.0.1: inverse host lookup failed: Unknown host
(UNKNOWN) [10.99.0.1] 58552 (?) : Connection refused
A DNS query to 10.99.0.1:53 works as expected.
dig @10.99.0.1 -p 53 accounts.google.com
; <<>> DiG 9.11.3-1ubuntu1.14-Ubuntu <<>> @10.99.0.1 -p 53 accounts.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53674
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;accounts.google.com. IN A
;; ANSWER SECTION:
accounts.google.com. 235 IN A 142.250.1.84
;; Query time: 0 msec
;; SERVER: 10.99.0.1#53(10.99.0.1)
;; WHEN: Thu Apr 22 17:20:09 UTC 2021
;; MSG SIZE rcvd: 64