51

I have two applications - app1 and app2, where app1 is a config server that holds configs for app2. I have defined /readiness endpoint in app1 and need to wait till it returns OK status to start up pods of app2.

It's crucial that deployment of app2 wait till kubernetes receives Http Status OK from /readiness endpoint in app1 as it's a configuration server and holds crucial configs for app2.

Is it possible to do this kind of deployment dependency?

6 Answers 6

37

You can use initContainers. Following is an example of how you can do in your YAML file

initContainers:
- name: wait-for-other-pod
  image: docker.some.image
  args:
  - /bin/sh
  - -c
  - >
    set -x;
    while [ $(curl -sw '%{http_code}' "http://www.<your_pod_health_check_end_point>.com" -o /dev/null) -ne 200 ]; do
      sleep 15;
    done

I have used curl to hit the health check endpoint, you can use any other UNIX command to check if the other pod is ready.

If you have a dependency on k8s resources, you can make use of stackanetes/kubernetes-entrypoint example:

initContainers:
- command:
  - kubernetes-entrypoint
  name: init-dependency-check
  env:
  - name: POD_NAME
    valueFrom:
      fieldRef:
        apiVersion: v1
        fieldPath: metadata.name
  - name: NAMESPACE
    valueFrom:
      fieldRef:
        apiVersion: v1
        fieldPath: metadata.namespace
  - name: DEPENDENCY_SERVICE
  - name: DEPENDENCY_DAEMONSET
  - name: DEPENDENCY_CONTAINER
  - name: DEPENDENCY_POD_JSON
    value: '[{"labels":{"app.kubernetes.io/name":"postgres"}}]'
  - name: COMMAND
    value: echo done
  image: projects.registry.vmware.com/tcx/snapshot/stackanetes/kubernetes-entrypoint:latest
  securityContext:
    privileged: true
    runAsUser: 0

In the above example, the pod with initContainer init-dependency-check will wait until pod with label "app.kubernetes.io/name":"postgres" is in the Running state. Likewise you can make use of DEPENDENCY_SERVICE, DEPENDENCY_DAEMONSET, DEPENDENCY_CONTAINER

6
  • 1
    200 ] there need to be a space between "200" and "]" please fix it for newbies like me :) Jul 12, 2020 at 3:52
  • 2
    For docker.some.image, one can use k8s.gcr.io/busybox or busybox from Docker Hub.
    – Lam Le
    Jan 28, 2021 at 10:48
  • 1
    busybox from docker hub does not contain curl command. Sep 15, 2021 at 8:58
  • Entrypoint WARNING: 2022/09/03 21:16:30 entrypoint.go:72: Resolving dependency Pod on same host with labels map[app:development-bitcoin] in namespace default failed: Getting POD: development-lightning-8469b65998-pqp6z failed : pods "development-lightning-8469b65998-pqp6z" is forbidden: User "system:serviceaccount:default:default" cannot get resource "pods" in API group "" in the namespace "default" . Getting this permission error when the init container runs. Am I missing something obvious? Sep 3, 2022 at 21:20
  • @PatrickGeyer you have not configured the RBAC correctly.
    – Vishrant
    Sep 5, 2022 at 23:37
27

Yes, it's possible using Init Containers (also, see this blog post for some background re timing) but a better, more Kubernetes-native pattern is to use retries and timeouts rather than hard-coding dependencies in this way.

1
  • 21
    I understand that "use retries and timeouts" is the approach in Kubernetes, but it clogs my logging so much with failures. It creates a new problem; filtering 'normal' failures due to this design pattern. A 'simple' dependency management like "do not schedule before <pod/deployment/replicaset/...> is ready" would be SO MUCH helpful to reduce this problem and more obvious in monitoring the cluster than looking at a "initContainer" to be run and you don't know it's actually init'ing or just waiting.
    – gertvdijk
    Jul 19, 2019 at 10:56
13
initContainers:
    - name: wait-for-dependent-service
      image: stefanevinance/wait-for-200
      env:
        - name: URL
          value: http://dependent-service.{{.Release.Namespace}}.svc.cluster.local:3000

Using https://hub.docker.com/r/stefanevinance/wait-for-200/

5

Using wait-for-it.sh is actually quite easy:

  initContainers:
  - name: wait-for-app1
    image: image-docker-containing-sh
    args:
    - /bin/sh
    - -c
    - /usr/app/wait-for-it.sh app1:<portapp1> -t 0

Of course retries and timeouts are the way to go, but this works great as a workaround.

5

In my case the application don't have a /readiness endpoint. We have master pods and worker pods and we want the worker pods to start up only after master pod is up and running. The master pod runs an application listening on TCP port 80. I ended up using netcat in an init container in the worker pod.

nc -z <host> <port>

nc will check if the port is open, exiting with 0 on success, 1 on failure.

initContainers:
- name: wait-for-master-before-starup
  image: busybox
  command:  ["sh", "-c", "until nc -z master-service 80 > /dev/null; do echo Waiting for master.; sleep 2; done;"]

If you already have a health/readiness endpoint, its much better to use it like mentioned in other answers here.

0

same as @Vishrant Answer... just other way to write bash commands in single line in YAML file

  initContainers:
  - name: wait-for-some-pod
    image: yourDockerImage
    command: ["/bin/sh","-c"]
    args: ['while [ $(curl -ksw "%{http_code}" "https://<pod_health_check_end_point>" -o /dev/null) -ne 200 ]; do sleep 5; echo "health check failed . Waiting for the service..."; done']
2
  • @foobarbecue its a way to get http code from curl response. Jul 28, 2021 at 7:20
  • 1
    Ah thanks, I see now. With so many layers of tools it's hard to know if this is k8s, bash, or curl. Ref: curl.se/docs/…. Jul 28, 2021 at 20:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.