121

Yesterday, I stopped a Helm upgrade when it was running on a release pipeline in Azure DevOps and the followings deployments failed.

I tried to see the chart that has failed with the aim of delete it but the chart of the microservice ("auth") doesn't appear. I used the command «helm list -n [namespace_of_AKS]» and it doesn't appear.

What can I do to solve this problem?

Error in Azure Release Pipeline

2022-03-24T08:01:39.2649230Z Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
2022-03-24T08:01:39.2701686Z ##[error]Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

Helm List Helm List

4
  • 20
    helm ls -a -n {namespace} will list all releases within a namespace, regardless of status. You can also use helm ls -aA instead to list all releases in all namespaces -- in case you actually deployed the release to a different namespace (I've done that before) Commented Mar 24, 2022 at 12:50
  • @BlenderFox thank you, your response solved my problem
    – fjalcaraz
    Commented Apr 11, 2022 at 7:55
  • @kavyasaraboju-MT thank you for your response. I didn't prove it because I coul fix with the previous response
    – fjalcaraz
    Commented Apr 11, 2022 at 7:56
  • @fjalcaraz no problem. I will add as an answer for others to find Commented Apr 11, 2022 at 8:08

6 Answers 6

184
Answer recommended by Microsoft Azure Collective

This error can happen for few reasons, but it most commonly occurs when there is an interruption during the upgrade/install process as you already mentioned.

To fix this one may need to, first rollback to another version, then reinstall or helm upgrade again.

Try below command to list

helm ls --namespace <namespace>

but you may note that when running that command ,it may not show any columns with information

Try to check the history of the previous deployment

helm history <release> --namespace <namespace>

This provides with information mostly like the original installation was never completed successfully and is pending state something like STATUS: pending-upgrade state.

To escape from this state, use the rollback command:

helm rollback <release> <revision> --namespace <namespace>

revision is optional, but you should try to provide it.

You may then try to issue your original command again to upgrade or reinstall.

7
  • 31
    To rollback to the previous revision and you don't know the number of the revision, you can use helm rollback <release> 0 and it will do it automatically.
    – saranicole
    Commented Aug 4, 2022 at 17:37
  • Do you know how can I reproduce this behavior, in oder to test a automation?
    – Felipe Hsu
    Commented Dec 22, 2022 at 19:14
  • I use TrueNAS and rolled back to a prior version. After trying the upgrade again, it worked this time. Commented Jan 8, 2023 at 8:44
  • 6
    To clarify something that confused me for a second : <release> is something from the NAME column returned by helm ls
    – Nick
    Commented Oct 11, 2023 at 22:57
  • I have a similar problem here. My helm upgrade fails with this same error the first time it runs, but the retry is successful. When I do a helm list my last deployment shows as successful.
    – Sumitk
    Commented May 10 at 2:15
50

helm ls -a -n {namespace} will list all releases within a namespace, regardless of status.

You can also use helm ls -aA instead to list all releases in all namespaces -- in case you actually deployed the release to a different namespace (I've done that before)

2
  • 7
    to delete: helm uninstall {release} -n {namespace} Commented Jun 22, 2023 at 12:45
  • What is listed as NAME is the same as <release>/{release} parameter. Commented Jun 10 at 16:30
35

Try deleting the latest helm secret for the deployment and re-run your helm apply command.

kubectl get secret -A | grep <app-name>
kubectl delete secret <secret> -n <namespace>
5
  • I had this problem when a deployment failed due to a repository name clash. I was not able to use rollback. but deleting the secret did it for me! Thanks!
    – jonnyeom
    Commented Mar 7, 2023 at 1:59
  • 1
    you saved me from a lot of headache! deleting the secret worked a charm
    – iomv
    Commented Oct 3, 2023 at 9:52
  • 1
    to get only latest you can just add | tail -n 1 in the end of the first command so you will have: kubectl get secret -A | grep <app-name> | tail -n 1
    – Sysanin
    Commented Dec 5, 2023 at 10:54
  • kubectl get secret -A leads in my case to the following output Error from server (Forbidden): secrets is forbidden: User ... cannot list resource "secrets" in API group "" at the cluster scope: making cluster-scoped requests for namespaced resources is not allowed Commented May 28 at 9:33
  • 1
    This should be the first recommended approach, as it is less intrusive than other rollback suggestions
    – Oleksiy
    Commented Jun 20 at 21:27
10

I also got the same issue, I solved it by rolling back the Helm release from the OpenLens tool, It was showing as Pending-Upgrade before.

enter image description here

enter image description here

3

My exact error was

Helm upgrade failed: another operation (install/upgrade/rollback) is in progress" or "##[error]Error: UPGRADE FAILED: release <namespace> failed, and has been rolled back due to atomic being set: timed out waiting for the condition

The error has 2 potential solutions: Either delete the stuck deployment, or fix something in deployment pipeline.


Solution #1: Delete the stuck deployment

If this same pipeline has been working until now, it may just be stuck. Time to delete it in the Azure portal!

  1. Go the Azure Portal > Azure CLI (aka the icon called "Cloud Shell")

Azure portal address bar and header with Cloud Shell icon circled

  1. az account show // This will give which subscription are you in.

  2. az account set --subscription

  3. az aks get-credentials --name --resource-group

  4. kubectl get secrets -n

  5. In this situation, the latest release has a problem, so delete that release by running the following:

  6. kubectl delete secret -n sh.helm.release.v1..v88 (or whatever the last release was as shown after last command)

  7. Run the pipeline again, it works :-)


Solution #2: Fix the pipeline

If you've just updated your pipeline, or haven't run it yet, you may have a bug. That was the case for my particular Azure DevOps pipeline.

  1. The azure-pipeline.yml stage is missing some parameters (or whatever the problem is in your pipeline)
  2. Fix it.
  3. Deployment succeeds!

Successful deployment via Azure DevOps pipeline

3

Delete the helm secret associated with the release and re-run the command

Helm3 makes use of the Kubernetes Secrets object to store any information regarding a release. These secrets are basically used by Helm to store and read it's state every time we run "helm upgrade" or "helm install".

First we get all the secrets for the namespace by running: "kubectl get secrets -n <<namespace"

enter image description here

Now delete the helm secret associated with the pending release by running the following command:

enter image description here

Note: Here "gloo-system" is the namespace.

1
  • I have a similar problem which I asked here. My latest deployment was successful but every time I upgrade a new release, it fails first. Once I retry its successful then. How can I find more on whats going on in the secrets?
    – Sumitk
    Commented May 10 at 2:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.