Background:
I'm using Drone to test an application. Drone is deployed to Kubernetes, with with a docker (dind / docker-in-docker) container side-carred.
After the test completes, I use drone again to build & push several docker images of about ~40mb each to us.gcr.io
When Drone creates the docker container to test my application, and the separate container to build my application and images, it creates a docker network to link the containers to build services, like a temporary test database (pretty standard in a CI pipeline).
However, the combination of Kubernetes pod networking, and Docker-in-Docker results in the following when trying to push to gcr:
time="2018-03-19T03:31:12.037507241Z" level=error msg="Upload failed, retrying: net/http: HTTP/1.x transport connection broken: write tcp w.x.y.z:39662->z.y.x.w:443: write: broken pipe"
time="2018-03-19T03:31:17.208009069Z" level=error msg="Upload failed, retrying: net/http: HTTP/1.x transport connection broken: write tcp w.x.y.z:39662->z.y.x.w:443: write: broken pipe"
time="2018-03-19T03:31:17.216232506Z" level=error msg="Upload failed, retrying: net/http: HTTP/1.x transport connection broken: write tcp w.x.y.z:39662->z.y.x.w:443: write: broken pipe"
time="2018-03-19T03:31:17.407608372Z" level=error msg="Upload failed, retrying: net/http: HTTP/1.x transport connection broken: write tcp w.x.y.z:39662->z.y.x.w:443: write: broken pipe"
time="2018-03-19T03:31:17.410403394Z" level=error msg="Upload failed, retrying: net/http: HTTP/1.x transport connection broken: write tcp w.x.y.z:39662->z.y.x.w:443: write: broken pipe"
time="2018-03-19T03:31:23.432621075Z" level=error msg="Upload failed, retrying: unexpected EOF"
However, when pushing to (what I assume is) an older registry version, then it works perfectly.
When pushing to gcr while there is no docker container networking enabled, then it also works perfectly.
Here are the docker commands being ran. Obviously the sensitive data has been omitted.
docker network create test-network && \
docker run --network=test-network -d cockroachdb/cockroach:v1.1.2 -c /cockroach sql --insecure && \
docker run --rm -it -e GKE_CLUSTER_NAME=my-cluster-1 -e GKE_CLUSTER_ZONE=us-east1-b -e GCP_PROJECT=my-gcp-project -e DOCKER_USE_GCP=true -v /var/run/docker.sock:/var/run/docker.sock --network=test-network us.gcr.io/my-project/runner /bin/sh -c 'mkdir -p src/git.example.com/project && git clone https://user:pass!@git.example.com/project/project $GOPATH/src/git.example.com/project/project && cd $GOPATH/src/git.example.com/project/project && git checkout gcr && jules -stage deploy_docker'
The jules -stage deploy_docker command runs a go build, docker build, and then gcloud docker -- push... on 8 different directories simultaneously.
So, summary:
Kubernetes pod + docker-in-docker + gcloud docker push results in a consistently interrupted connection.
Is there something I could do with docker daemon or kubernetes network settings or something to mitigate this? At the very least I want to understand why this is happening.
Thanks!
Update:
This doesn't even require Kubernetes to happen!
I just tried it with a fresh GCE instance running Ubuntu and it happens there, too.