4

I have a container with 2 subnets:

  • one is the reverse proxy subnet
  • the second one is the internal subnet for the different containers of that project

The container needs to access an external SMTP server (on mailgun.com), but it looks like, with docker-compose, you can put a container on both one or more subnets and give it access to the host network at the same time.

Is there a way to allow this container to initiate connections to the outside world?

and, if no, what common workarounds are used? (for example, adding an extra IP to the container to be on the host network, etc.)

This is the docker compose file:

version: '2.3'

services:

  keycloak:
    container_name: keycloak
    image: jboss/keycloak
    restart: unless-stopped
    volumes:
      - '/appdata/keycloak:/opt/jboss/keycloak/standalone/data'
    expose:
      - 8080
    external_links:
      - auth
    networks:
      - default
      - nginx
    environment:
      KEYCLOAK_USER: XXXX
      KEYCLOAK_PASSWORD: XXXX
      PROXY_ADDRESS_FORWARDING: 'true'
      ES_JAVA_OPTS: '-Xms512m -Xmx512m'
      VIRTUAL_HOST: auth.XXXX.com
      VIRTUAL_PORT: 80
      LETSENCRYPT_HOST: auth.XXXX.com
      LETSENTRYPT_EMAIL: [email protected]


networks:
  default:
    external:
      name: app-network
  nginx:
    external:
      name: nginx-proxy

The networks are as follows:

$ dk network ls
NETWORK ID          NAME                DRIVER              SCOPE
caba49ae8b1c        bridge              bridge              local
2b311986a6f6        app-network         bridge              local
67f70f82aea2        host                host                local
9e0e2fe50385        nginx-proxy         bridge              local
dab9f171e37f        none                null                local

and nginx-proxy network info is:

$ dk network inspect nginx-proxy
[
{
    "Name": "nginx-proxy",
    "Id": "9e0e2fe503857c5bc532032afb6646598ee0a08e834f4bd89b87b35db1739dae",
    "Created": "2019-02-18T10:16:38.949628821Z",
    "Scope": "local",
    "Driver": "bridge",
    "EnableIPv6": false,
    "IPAM": {
        "Driver": "default",
        "Options": {},
        "Config": [
            {
                "Subnet": "172.18.0.0/16",
                "Gateway": "172.18.0.1"
            }
        ]
    },
    "Internal": false,
    "Attachable": false,
    "Ingress": false,
    "ConfigFrom": {
        "Network": ""
    },
    "ConfigOnly": false,
    "Containers": {
        "360b49ab066853a25cd739a4c1464a9ac25fe56132c596ce48a5f01465d07d12": {
            "Name": "keycloak",
            "EndpointID": "271ed86cac77db76f69f6e76686abddefa871b92bb60a007eb131de4e6a8cb53",
            "MacAddress": "02:42:ac:12:00:04",
            "IPv4Address": "172.18.0.4/16",
            "IPv6Address": ""
        },
        "379dfe83d6739612c82e99f3e8ad9fcdfe5ebb8cdc5d780e37a3212a3bf6c11b": {
            "Name": "nginx-proxy",
            "EndpointID": "0fcf186c6785dd585b677ccc98fa68cc9bc66c4ae02d086155afd82c7c465fef",
            "MacAddress": "02:42:ac:12:00:03",
            "IPv4Address": "172.18.0.3/16",
            "IPv6Address": ""
        },
        "4c944078bcb1cca2647be30c516b8fa70b45293203b355f5d5e00b800ad9a0d4": {
            "Name": "adminmongo",
            "EndpointID": "65f1a7a0f0bcef37ba02b98be8fa1f29a8d7868162482ac0b957f73764f73ccf",
            "MacAddress": "02:42:ac:12:00:06",
            "IPv4Address": "172.18.0.6/16",
            "IPv6Address": ""
        },
        "671cc99775e09077edc72617836fa563932675800cb938397597e17d521c53fe": {
            "Name": "portainer",
            "EndpointID": "950e4b5dcd5ba2a13acba37f50e315483123d7da673c8feac9a0f8d6f8b9eb2b",
            "MacAddress": "02:42:ac:12:00:02",
            "IPv4Address": "172.18.0.2/16",
            "IPv6Address": ""
        },
        "90a98111cbdebe76920ac2ebc50dafa5ea77eba9f42197216fcd57bad9e0516e": {
            "Name": "kibana",
            "EndpointID": "fe1768274eec9c02c28c74be0104326052b9b9a9c98d475015cd80fba82ec45d",
            "MacAddress": "02:42:ac:12:00:05",
            "IPv4Address": "172.18.0.5/16",
            "IPv6Address": ""
        }
    },
    "Options": {},
    "Labels": {}
}
]

Update:

The following test was done to test the solution proposed by lbndev:

a test network was created:

# docker network create \
       -o "com.docker.network.bridge.enable_icc"="true" \
       -o "com.docker.network.bridge.enable_ip_masquerade"="true" \
       -o "com.docker.network.bridge.host_binding_ipv4"="0.0.0.0" \
       -o"com.docker.network.driver.mtu"="1500" \
       test_network
e21057cf83eec70e9cfeed459d79521fb57e9f08477b729a8c8880ea83891ed9

we can display the contents:

# docker inspect test_network
[
{
    "Name": "test_network",
    "Id": "e21057cf83eec70e9cfeed459d79521fb57e9f08477b729a8c8880ea83891ed9",
    "Created": "2019-02-24T21:52:44.678870135+01:00",
    "Scope": "local",
    "Driver": "bridge",
    "EnableIPv6": false,
    "IPAM": {
        "Driver": "default",
        "Options": {},
        "Config": [
            {
                "Subnet": "172.22.0.0/16",
                "Gateway": "172.22.0.1"
            }
        ]
    },
    "Internal": false,
    "Attachable": false,
    "Ingress": false,
    "ConfigFrom": {
        "Network": ""
    },
    "ConfigOnly": false,
    "Containers": {},
    "Options": {
        "com.docker.network.bridge.enable_icc": "true",
        "com.docker.network.bridge.enable_ip_masquerade": "true",
        "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
        "com.docker.network.driver.mtu": "1500"
    },
    "Labels": {}
}
]

Then we can inspect the container:

I put the contents on pastebin: https://pastebin.com/5bJ7A9Yp since it's quite large and would make this post unreadable.

and testing:

# docker exec -it 5d09230158dd sh
sh-4.2$ ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
^C
--- 1.1.1.1 ping statistics ---
11 packets transmitted, 0 received, 100% packet loss, time 10006ms

So, we couldn't get this solution to work.

10
  • We could assist you better if you posted your docker-compose.yml file.
    – Steve Boyd
    Feb 24, 2019 at 21:58
  • I just added the (simplified) contents. As you can see, the container is on 2 subnets: the one of the reverse proxy, that routes external connections to the container, and deals with ssl, and the second one that allow this service to talk to other services on the same subnet. What I need is this service to talk to an external site (smtp.mailgun.com). But if I put the service on the host network, it can't be in the subnets, which is essential for the operation, especially since we'll hever several of these running behind the reverse proxy.
    – Thomas
    Feb 24, 2019 at 22:13
  • Can you please add the output of docker network ls and docker network inspect nginx-proxy (I assume the network you want to use for access to/from the "outside world" is nginx-proxy) ? My guess is that something is wrong in the way you created those networks mentioned in your compoose file.
    – lbndev
    Feb 27, 2019 at 11:54
  • @lbndev, the nginx-proxy network is the reverse proxy network; external requests come to a nginx reverse proxy container and this container routes the requests to appropriate containers that are on the nginx-proxy network. Then each 'project', has its own subnet where containers connect to each other. What I posted is a simplified version on my dev machine, but there are a few subnets like app-network, one per project.
    – Thomas
    Feb 27, 2019 at 12:13
  • but, in one case, I have a container that is both on its own project subnet (app-network) and can be reached through the reverse proxy (so it is on the nginx-network subnet) and this container needs to send emails out, so it needs to talk to a 3rd part smtp gateway (we're using mailgun, so it needs to talk to smtp.mailgun.,com). I can't open each project's subnet since they're passing database data, etc and there is no security inside the subnet. So it needs to either go out through the nginx-proxy network, but it's not designed for it, or we need to add another network to outgoing connections
    – Thomas
    Feb 27, 2019 at 12:15

2 Answers 2

1

Looks like your bridge network is missing a few options, to allow it to reach the outside world.

Try executing docker network inspect bridge (the default bridge network). You'll see this in the options :

...
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
...

On your nginx-proxy network, these are missing.

You should delete your network and re-create it with these additional options. From the documentation on user-defined bridged networks and docker network create command :

docker network create \
      -o "com.docker.network.bridge.enable_icc"="true" \
      -o "com.docker.network.bridge.enable_ip_masquerade"="true" \
      -o "com.docker.network.bridge.host_binding_ipv4"="0.0.0.0" \
      -o"com.docker.network.driver.mtu"="1500" \
      nginx-proxy

Enabling ICC or not is up to you.

What will enable you to reach your mail server is ip_masquerade to be enabled. Without this setup, your physical infrastructure (= network routers) would need to properly route the IPs of the docker network subnet (which I assume is not the case).

Alternatively, you could configure your docker network's subnet, ip range and gateway, to match those of your physical network.

4
  • I have edited the main post showing a test doing what you recommend, and I've also attached all the output. So far, no connection possible to the outside. Did we miss something?
    – Thomas
    Feb 27, 2019 at 21:38
  • I did the exact same test, using your dockerfile, and I am able to ping google.com from inside the keycloak container. My guess is that something is wrong with your network. Do you have firewall rules active on your docker host ? In your docker-compose.yml, try replacing the keycloak image by busybox (add "stdin_open: true, tty: true" so that the container keeps running), and run docker exec -it <containerId> traceroute www.google.com to see if the packets manage to leave your docker host.
    – lbndev
    Feb 28, 2019 at 14:09
  • It turns out that your instructions were right, but we found the problem to be somewhere else. Will update the thread.
    – Thomas
    Mar 1, 2019 at 18:35
  • 1
    Glad to have helped. :)
    – lbndev
    Mar 2, 2019 at 8:47
1

In the end, the problem turned out to be very simple:

In the daemon.json file, in the docker config, there was the following line:

{"iptables": false, "dns": ["1.1.1.1", "1.0.0.1"]}

It comes from the setup scripts we’ve been using and we didn’t know about iptables:false

It prevents docker from updating the host’s iptables; while the bridge networks were set up correctly, there was no communication possible with the outside.

While simple in nature, it proved very long to find, so I’m posting it as an answer with the hope it might help someone.

Thanks to everyone involved for trying to solve this issue!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.