I am trying to scrape docker metrics from inside a container. I have seen the creator of Prometheus doing it this way see video here. I am not able to reproduce it.
Configuration
I have exposed docker metrics page via json config.
{
"metrics-addr" : "172.17.0.1:4999",
"experimental" : true
}
I have also tried with 0.0.0.0. In any case I am able to curl from the host machine itself with no issues.
$ curl 172.17.0.1:4999/metrics | more
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0# HELP builder_builds_failed_total Number of failed image builds
# TYPE builder_builds_failed_total counter
builder_builds_failed_total{reason="build_canceled"} 0
builder_builds_failed_total{reason="build_target_not_reachable_error"} 0
...
From inside container
However, I am not able to get the same metrics page from inside a container. I have tried to basi/socat image, the same one from the video. I have also tried a handful of other ways like using curl from inside the container and so on.
The image does essentially just one thing
ENV IN="172.17.0.1:4999" \
OUT="4999"
ENTRYPOINT socat -d -d TCP-L:$OUT,fork TCP:$IN
Start Socat Container
docker run --rm -p 4998:4999 basi/socat
2021/04/02 16:27:47 socat[8] N listening on AF=2 0.0.0.0:4999
curl the Container published port
curl localhost:4998/metrics
Connection just hangs
The socat
images recognizes the connection, but it hangs for 1-2 minutes and times out.
2021/04/02 16:29:53 socat[8] N accepting connection from AF=2 172.17.0.1:57678 on AF=2 172.17.0.2:4999
2021/04/02 16:29:53 socat[8] N forked off child process 9
2021/04/02 16:29:53 socat[8] N listening on AF=2 0.0.0.0:4999
2021/04/02 16:29:53 socat[9] N opening connection to AF=2 172.17.0.1:4999
2021/04/02 16:32:04 socat[9] E connect(5, AF=2 172.17.0.1:4999, 16): Operation timed out
2021/04/02 16:32:04 socat[9] N exit(1)
2021/04/02 16:32:04 socat[8] N childdied(): handling signal 17
2021/04/02 16:32:04 socat[8] W waitpid(): child 9 exited with status 1
curl: (52) Empty reply from server
Firewall
I am slowly wondering if it could be something in my iptables
rules. I have posted below INPUT
and DOCKER-USER
chain. Those are the only ones I have changed myself.
sudo iptables -L -n -v
Chain INPUT (policy DROP 32060 packets, 1810K bytes)
pkts bytes target prot opt in out source destination
716K 1484M ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
599K 1069M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
16 3138 ACCEPT all -- eth0 * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
12 624 DROP all -- eth0 * 0.0.0.0/0 0.0.0.0/0
74921 139M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
0.0.0.0
docker_gwbridge
and all containers have access to this. Socat is also not saying connection refused as it happens with completely invalid addresses. Again, in the image from the video its not using hostip.