348

I have the following replication controller in Kubernetes on GKE:

apiVersion: v1
kind: ReplicationController
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  replicas: 2
  selector:
    app: myapp
    deployment: initial
  template:
    metadata:
      labels:
        app: myapp
        deployment: initial
    spec:
      containers:
      - name: myapp
        image: myregistry.com/myapp:5c3dda6b
        ports:
        - containerPort: 80
      imagePullPolicy: Always
      imagePullSecrets:
        - name: myregistry.com-registry-key

Now, if I say

kubectl rolling-update myapp --image=us.gcr.io/project-107012/myapp:5c3dda6b

the rolling update is performed, but no re-pull. Why?

14
  • 31
    I gave a different image, just with the same tag. If it is necessary to give a different tag, well, I see no point in the imagePullPolicy field. Commented Oct 14, 2015 at 7:00
  • 7
    I want to use a specific tag, but its newest version. Commented Jul 3, 2018 at 22:26
  • 6
    @TorstenBronger I think this is a breaking change in Kubernetes/Docker theory. The idea that you could pull image:tag (other than latest) at two different times and get two different images would be problematic. A tag is akin to a version number. It would be better practice to always change the tag when the image changes. Commented Mar 12, 2019 at 18:41
  • 5
    It depends. There is software with a very stable API but security updates. Then, I want the latest version without having to say so explicitly. Commented Mar 13, 2019 at 6:40
  • 5
    @TorstenBronger Regarding using latest, dont do it. Latest will pull the, well, more recently image with the latest tag. What you want is a SemVer range. ~1.2.3 for example. this will pull images with tags between the range of >= 1.2.3 and < 1.3.0. As long as the image vendor follows SemVer your know (and this is the important part) no backwards breaking change were added (on purpose) and that no new features were added (possible security concern). Please, please never use latest in production systems. Commented Jul 12, 2019 at 12:30

21 Answers 21

300

Kubernetes will pull upon Pod creation if either (see updating-images doc):

  • Using images tagged :latest
  • imagePullPolicy: Always is specified

This is great if you want to always pull. But what if you want to do it on demand: For example, if you want to use some-public-image:latest but only want to pull a newer version manually when you ask for it. You can currently:

  • Set imagePullPolicy to IfNotPresent or Never and pre-pull: Pull manually images on each cluster node so the latest is cached, then do a kubectl rolling-update or similar to restart Pods (ugly easily broken hack!)
  • Temporarily change imagePullPolicy, do a kubectl apply, restart the pod (e.g. kubectl rolling-update), revert imagePullPolicy, redo a kubectl apply (ugly!)
  • Pull and push some-public-image:latest to your private repository and do a kubectl rolling-update (heavy!)

No good solution for on-demand pull. If that changes, please comment; I'll update this answer.

8
  • 4
    You say kubernetes will pull on Pod creation when using :latest - what about patching? does it also always pull the newest/latest image? Seems not to work for me :( Commented Oct 26, 2016 at 11:57
  • It depends if your patch forces the re-creation of a Pod or not. Most likely not, then it'll not pull again. You may kill the Pod manually, or tag with something unique and patch with that updated tag.
    – Wernight
    Commented Nov 14, 2016 at 14:19
  • 4
    This is an answer to a different question. I asked for forcing a re-pull. Commented Dec 2, 2016 at 7:11
  • 2
    Thanks. Went for the Pull & Push approach. Automated as much of it as possible with bash scripts but agreed, it's heavy :)
    – arcseldon
    Commented Mar 31, 2020 at 8:03
  • 2
    How about having for each environment a label like "prod", "stage", "test", leave the imagePullPolicy to "always" and push the label, whenever you want to deploy, to the image that shall be deployed? Commented Jan 17, 2022 at 8:59
165

One has to group imagePullPolicy inside the container data instead of inside the spec data. However, I filed an issue about this because I find it odd. Besides, there is no error message.

So, this spec snippet works:

spec:
  containers:
  - name: myapp
    image: myregistry.com/myapp:5c3dda6b
    ports:
    - containerPort: 80
    imagePullPolicy: Always
  imagePullSecrets:
    - name: myregistry.com-registry-key

Please be aware, that if the image is already cached on the local node, it will not be re-pulled (you will have to change the remote image). This applies even when the local image is actually different, e.g. corrupted by misguided attempts to free some space manually, because image checksums are not re-computed.

8
  • 14
    imagePullPolicy (or tagging :latest) is good if you want to always pull, but doesn't solve the question of pulling on demande.
    – Wernight
    Commented Mar 11, 2016 at 13:13
  • 4
    Yes, I want to always pull, as stated in the question. Commented Dec 2, 2016 at 7:12
  • 3
    Using imagePullPolicy: Always inside the container definition will have kubernetes fetch images tagged with :latest whenever a newer version of them is pushed to the registry?
    – pkaramol
    Commented Jan 15, 2018 at 13:25
  • 3
    @pkaramol No. imagePullPolicy: Always simply tells Kubernetes to always pull image from the registry. What image it will is configured by image attribute. If you configure it to image: your-image:latest, then it will always pull the your-image image with the latest tag.
    – Gajus
    Commented Dec 17, 2018 at 6:52
  • 1
    I just had the same issue here with a cronjob. The "latest" tag was ignored and only setting the job spec to the always pull policy made k8s reload the image for the next execution (=container creation) something seems to be different between these two options, despite every documentation treating them as equal. Commented Oct 30, 2020 at 18:14
123

There is a comand to directly do that:

Create a new kubectl rollout restart command that does a rolling restart of a deployment.

The pull request got merged. It is part of the version 1.15 (changelog) or higher.

7
  • Yes part of Issue: github.com/kubernetes/kubernetes/issues/13488
    – Tilo
    Commented Jul 5, 2019 at 16:28
  • 5
    Yes,this is the best way to trigger update in new kubernetes verion of 1.15.
    – Dolphin
    Commented Feb 15, 2020 at 5:01
  • 11
    No there isn't a command to directly do that. This only work with imagePullPolicy: Always set.
    – spinkus
    Commented Sep 10, 2021 at 23:57
  • 10
    @spinkus together with kubectl rollout restart deploy <name> Commented Oct 25, 2021 at 13:51
  • 1
    @melroy-van-den-berg This is easy and perfect. Thanks.
    – ozeray
    Commented Sep 11, 2023 at 12:47
45

My hack during development is to change my Deployment manifest to add the latest tag and always pull like so

image: etoews/my-image:latest
imagePullPolicy: Always

Then I delete the pod manually

kubectl delete pod my-app-3498980157-2zxhd

Because it's a Deployment, Kubernetes will automatically recreate the pod and pull the latest image.

4
  • I like taking advantage of the "desired state" premises of the "deployment" object... thanks for the suggestion! Commented Apr 10, 2018 at 23:08
  • 10
    It's worth noting that strategy is viable only if failures in the service and downtime are tolerable. For development it seems reasonable, but I would never carry this strategy over for a production deploy. Commented Jun 28, 2018 at 14:52
  • Edit the deployment, changing the imagePullPolicy to always and deleting the pod was enough for me, as Everett suggested. This is a development environment though. kubernetes.io/docs/concepts/containers/images Commented Feb 5, 2019 at 0:19
  • The "Always" imagePullPolicy is the default for tags named "latest" or no tag. Therefore you don't need to specify it in this example
    – hookenz
    Commented Nov 2, 2022 at 2:25
33

A popular workaround is to patch the deployment with a dummy annotation (or label):

kubectl patch deployment <name> -p \
  "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}"

Assuming your deployment meets these requirements, this will cause K8s to pull any new image and redeploy.

3
  • 3
    Yes, I use an annotation for this. Commented Mar 21, 2019 at 10:15
  • what annotation?
    – Jeryl Cook
    Commented Apr 5, 2019 at 15:18
  • 3
    Another sophisticated solution would be a combination of both ie. adding an annotation and setting ImagePullPolicy as Always. annotations like deployment.kubernetes.io/revision: "v-someversion" and kubernetes.io/change-cause: the reason can be quite helpful and heads towards immutable deployments.
    – chandan
    Commented May 22, 2019 at 18:14
18

Now, the command kubectl rollout restart deploy YOUR-DEPLOYMENT combined with a imagePullPolicy: Always policy will allow you to restart all your pods with a latest version of your image.

18
  1. Specify strategy as:
  strategy: 
    type: Recreate
    rollingUpdate: null
  1. Make sure you have different annotation for each deployment. Helm does it like:
  template:
    metadata:
      labels:
        app.kubernetes.io/name: AppName
        app.kubernetes.io/instance: ReleaseName
      annotations:
        rollme: {{ randAlphaNum 5 | quote }}
  1. Specify image pull policy - Always
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: Always
1
  • Warning: changing the annotation value leads to the pod recreation even if the docker image has not been changed! Commented May 11, 2023 at 5:49
10
# Linux

kubectl patch deployment <name> -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}"

# windows

kubectl patch deployment <name> -p (-join("{\""spec\"":{\""template\"":{\""metadata\"":{\""annotations\"":{\""date\"":\""" , $(Get-Date -Format o).replace(':','-').replace('+','_') , "\""}}}}}"))
10

This answer aims to force an image pull in a situation where your node has already downloaded an image with the same name, therefore even though you push a new image to container registry, when you spin up some pods, your pod says "image already present".

For a case in Azure Container Registry (probably AWS and GCP also provides this):

  1. You can look to your Azure Container Registry and by checking the manifest creation date you can identify what image is the most recent one.

  2. Then, copy its digest hash (which has a format of sha256:xxx...xxx).

  3. You can scale down your current replica by running command below. Note that this will obviously stop your container and cause downtime.

kubectl scale --replicas=0 deployment <deployment-name> -n <namespace-name>
  1. Then you can get the copy of the deployment.yaml by running:
kubectl get deployments.apps <deployment-name> -o yaml > deployment.yaml
  1. Then change the line with image field from <image-name>:<tag> to <image-name>@sha256:xxx...xxx, save the file.

  2. Now you can scale up your replicas again. New image will be pulled with its unique digest.

Note: It is assumed that, imagePullPolicy: Always field is present in the container.

10

Having gone through all the other answers and not being satisfied, I found much better solution here: https://cloud.google.com/kubernetes-engine/docs/how-to/updating-apps

It works without using latest tag or imagePullPolicy: Always. It also works if you push new image to the same tag by specifying image sha256 digest.

Steps:

  1. get image SHA256 from docker hub (see image below)
  2. find your deployment using kubectl get deployments
  3. kubectl set image deployment/<your-deployment> <your_container_name>=<some/image>@sha256:<your sha>
  4. kubectl scale deployment <your-deployment>--replicas=0
  5. kubectl scale deployment <your-deployment>--replicas=original replicas count

Note: Rollout might also work instead of scale but in my case we don't have enough hardware resources to create another instance and k8s gets stuck.

docker hub sha256 location

2
  • 2
    Small correction. In the second point, it should be <your_container_name> instead of <your-pod-name>
    – Vikram
    Commented Oct 25, 2021 at 12:24
  • 1
    Setting replicas to 0 will cause a service outage. Is there a way to do this without taking the app down?
    – Josiah
    Commented Mar 30, 2023 at 18:21
9

Apparently now when you run a rolling-update with the --image argument the same as the existing container image, you must also specify an --image-pull-policy. The following command should force a pull of the image when it is the same as the container image:

kubectl rolling-update myapp --image=us.gcr.io/project-107012/myapp:5c3dda6b --image-pull-policy Always

1
7

You can define imagePullPolicy: Always in your deployment file.

1
  • works for dev environment, but for prod use rolligupdate strategy
    – Slok
    Commented Nov 24, 2020 at 5:36
7

The rolling update command, when given an image argument, assumes that the image is different than what currently exists in the replication controller.

7
  • Does this mean the image tag (aka name) must be different? Commented Oct 14, 2015 at 7:01
  • Yes, the image name must be different if you pass the --image flag. Commented Oct 14, 2015 at 20:49
  • 2
    As my own answer says, it works also if the image name is the same. It was simply that the imagePullPolicy was in the wrong place. To my defence, the k8s 1.0 docs are erroneous in this aspect. Commented Oct 14, 2015 at 21:04
  • Gotta love when the docs are out of sync with the behavior. :/ Commented Oct 14, 2015 at 23:44
  • 2
    That url is outdated too. Commented Jan 4, 2020 at 17:32
5

I have used kubectl rollout restart for my springboot api and it works.

kubectl rollout restart -f pod-staging.yml --namespace test

Yaml for the Deployment:

apiVersion: "apps/v1"
kind: "Deployment"
metadata:
    name: "my-api"
    labels:
      app: "my-api"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "my-api"
  template:
    metadata:
      labels:
        app: "my-api"
    spec:
      containers:
        - name: my-api
          image: harbor.url.com/mycompany/my-api:staging
          ports:
            - containerPort: 8099
              protocol: TCP
          imagePullPolicy: Always
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8099
            initialDelaySeconds: 90
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8099
            initialDelaySeconds: 90
            periodSeconds: 5
          envFrom:
            - configMapRef:
                name: "my-api-configmap"
          env:
            - name: "TOKEN_VALUE"
              valueFrom:
                secretKeyRef:
                  name: "my-api-secret"
                  key: "TOKEN_VALUE"
          resources:
            requests:
              memory: "512Mi"
              cpu: "500m"
            limits:
              memory: "2048Mi"
              cpu: "1000m"
      imagePullSecrets:
        - name: "my-ci-user"
4

Defining imagePullPolicy: Always in the deployment would do.

1
  • 1
    Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
    – Community Bot
    Commented Feb 7 at 8:08
3

The Image pull policy will always actually help to pull the image every single time a new pod is created (this can be in any case like scaling the replicas, or pod dies and new pod is created)

But if you want to update the image of the current running pod, deployment is the best way. It leaves you flawless update without any problem (mainly when you have a persistent volume attached to the pod) :)

1

The below solved my problem:

kubectl rollout restart
0

if you want to perform a direct image update on a specific pod, you can use kubectl set image also.

https://kubernetes.io/docs/concepts/workloads/controllers/deployment/

1
  • Please add further details to expand on your answer, such as working code or documentation citations.
    – Community Bot
    Commented Sep 5, 2021 at 13:22
0

either you have deleted all the pods manually to get it recreated with pulling the image again.

or

run this below command kubectl rollout restart deployment/deployment_name kubectl rollout restart deployment/nginx

this command should recreate all the pods.

for both scenarios imagepullPolicy should be set as Always.

0

A one-liner solution based on invalidating Deployment hash by adding some new unique data, here: a timestamp-based environment variable (just like adding a "volatile" ENV to bust docker cache during image builds):

kubectl set env deployment/nginx REDEPLOY_TIME="$(date)"

or when using oc Client Tools under OCP/OKD:

oc set env dc/nginx REDEPLOY_TIME="$(date)"

It will trigger an automatic rolling re-deployment/re-pull even in older installations of k8s (not just in v1.15 or above, where kubectl rollout restart is the correct solution as described in this answer). In fact I verified this workaround even in the archaic Openshift 3.11 based on k8s 1.11 from mid-2018!

Note we need the usual prerequisites of imagePullPolicy: Always and a "rolling" container image tag such as latest.

Note: kudos and the original idea (using a YAML Deployment manifest file and sed) go back to this comment in the rather long-running k8s issue devoted to this now thankfully gone opinionated choice made initially by k8s devs.

0

I developed a lightweight Kubernetes tool URunner in order to automatically restart deployment resources while maintaining the same tag (es. :latest)

URunner can also be installed using Helm (Artifacthub link)

It uses Docker V2 API standard in order to continuously check if a Tag Digest is changed and eventually perform the needed restarts.
So basically it is compatible with ALL available container registries (ex. AWS ECR, GCP, Harbor, DigitalOcean registry..)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.