3

I have been trying to use some custom, local built image on my kind cluster by following the instructions on https://kind.sigs.k8s.io/docs/user/local-registry - i.e. applying the following containerdConfigPatches to my cluster.cfg:

containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.83.82:5000"]
    endpoint = ["http://192.168.83.82:5000"]

192.168.83.82:5000 is the IP of the VM where the local (insecure) registry is running alongside the kind cluster and its exposed port.

Once the cluster is created, I can verify the settings where applied to all nodes:

docker exec kind-worker3 cat /etc/containerd/config.toml
# [...]
# Relevant sections:
# [...]
  [plugins."io.containerd.grpc.v1.cri"]
    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.83.82:5000"]
          endpoint = ["http://192.168.83.82:5000"]

However, pods keep failing with to ErrImagePull and these Event log:

Events:
  Type     Reason     Age                From                             Message
  ----     ------     ----               ----                             -------
  Normal   Scheduled  32s                default-scheduler                Successfully assigned default/test-6bc95ff8c5-g6g86 to kind-worker3
  Normal   Pulled     31s                kubelet, kind-worker3  Container image "docker.elastic.co/beats/filebeat-oss:6.4.2" already present on machine
  Normal   Created    31s                kubelet, kind-worker3  Created container test-log-agent
  Normal   Started    31s                kubelet, kind-worker3  Started container test-log-agent
  Normal   Pulling    16s (x2 over 31s)  kubelet, kind-worker3  Pulling image "192.168.83.82:5000/test/image:2.2.1"
  Warning  Failed     16s (x2 over 31s)  kubelet, kind-worker3  Failed to pull image "192.168.83.82:5000/test/image:2.2.1": rpc error: code = Unknown desc = failed to resolve image "192.168.83.82:5000/test/image:2.2.1": no available registry endpoint: failed to do request: Head https://192.168.83.82:5000/v2/test/image/manifests/2.2.1: http: server gave HTTP response to HTTPS client
  Warning  Failed     16s (x2 over 31s)  kubelet, kind-worker3  Error: ErrImagePull
  Normal   BackOff    3s (x3 over 30s)   kubelet, kind-worker3  Back-off pulling image "192.168.83.82:5000/test/image:2.2.1"
  Warning  Failed     3s (x3 over 30s)   kubelet, kind-worker3  Error: ImagePullBackOff

TL;DR: "http: server gave HTTP response to HTTPS client" - which I thought would be solved with the ConfigPatch above (As it happens when you tweak docker's daemon.json).

Also, as an alternative, tried loading the image from the host to the cluster nodes:

kind load docker-image 192.168.83.82:5000/test/image:2.2.1 --name="kind-cluster"

And verified the image was loaded to all nodes by listing them:

sysadmin@ubuntu:~/kind$ docker exec kind-worker3 crictl images
IMAGE                                TAG                 IMAGE ID            SIZE
192.168.83.82:5000/test/image        2.2.1               ba1601dfa9c48       822MB
docker.io/kindest/kindnetd           0.5.0               ef97cccdfdb50       83.6MB
k8s.gcr.io/coredns                   1.3.1               eb516548c180f       40.5MB
k8s.gcr.io/etcd                      3.3.10              2c4adeb21b4ff       258MB
k8s.gcr.io/kube-apiserver            v1.15.3             be321f2ded3f3       249MB
k8s.gcr.io/kube-controller-manager   v1.15.3             ac7d3fe5b34b7       200MB
k8s.gcr.io/kube-proxy                v1.15.3             d428039608992       97.3MB
k8s.gcr.io/kube-scheduler            v1.15.3             a44f53b10fee0       96.5MB
k8s.gcr.io/pause                     3.1                 da86e6ba6ca19       746kB

Hoping that the image now would be read from the image cache. However, the result was exactly the same.

Any clue on how to approach this one? Is there something I might have overlooked?

2
  • 1
    Have you seen this? try setting insecure_skip_verify = true as can be found in docs.
    – Matt
    Commented Mar 17, 2020 at 8:47
  • Have you tried pinging the registry VM from the control plane or worker nodes? I run my local registry as a container along side the kind cluster node containers and not a VM. Commented Apr 28, 2020 at 22:53

1 Answer 1

3

As Matt alluded to above, try with the insecure_skip_verify flag in your config.toml file:

  [plugins."io.containerd.grpc.v1.cri"]
    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.83.82:5000"]
          endpoint = ["http://192.168.83.82:5000"]
      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."192.168.83.82:5000".tls]
          insecure_skip_verify = true
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.