10

I am getting issue while terminating the namesapce in the cluster, It's showing many parameters inside the namespace JSON. I followed this link https://medium.com/@craignewtondev/how-to-fix-kubernetes-namespace-deleting-stuck-in-terminating-state-5ed75792647e

 "spec": {},
    "status": {
        "conditions": [
            {
                "lastTransitionTime": "2021-01-11T08:41:48Z",
                "message": "All resources successfully discovered",
                "reason": "ResourcesDiscovered",
                "status": "False",
                "type": "NamespaceDeletionDiscoveryFailure"
            },
            {
                "lastTransitionTime": "2021-01-11T08:41:48Z",
                "message": "All legacy kube types successfully parsed",
                "reason": "ParsedGroupVersions",
                "status": "False",
                "type": "NamespaceDeletionGroupVersionParsingFailure"
            },
            {
                "lastTransitionTime": "2021-01-11T08:41:48Z",
                "message": "All content successfully deleted, may be waiting on finalization",
                "reason": "ContentDeleted",
                "status": "False",
                "type": "NamespaceDeletionContentFailure"
            },
            {
                "lastTransitionTime": "2021-01-11T08:42:09Z",
                "message": "All content successfully removed",
                "reason": "ContentRemoved",
                "status": "False",
                "type": "NamespaceContentRemaining"
            },
            {
                "lastTransitionTime": "2021-01-11T08:41:48Z",
                "message": "All content-preserving finalizers finished",
                "reason": "ContentHasNoFinalizers",
                "status": "False",
                "type": "NamespaceFinalizersRemaining"
            }
        ],
        "phase": "Terminating"
    }
}```
4
  • I've already seen this when there is a webhook still active. In this cases it took around 30 mins to delete the namespace. How long did you wait? Jan 11, 2021 at 13:39
  • Does this answer your question? Namespace "stuck" as Terminating, How do I remove it? Jan 11, 2021 at 14:06
  • Still in showing terminating since 215 min almost Jan 11, 2021 at 14:57
  • @SaikatChakrabortty no it's not helping even I tried kubectl delete ns fleet-system --grace-period=0 --force --namespace -n fleet-system warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. warning: deleting cluster-scoped resources, not scoped to the provided namespace namespace "fleet-system" force deleted And stuck here------ Jan 12, 2021 at 15:50

3 Answers 3

34

I have found the answer to terminate the stuck namespace.

for ns in $(kubectl get ns --field-selector status.phase=Terminating -o jsonpath='{.items[*].metadata.name}')
do
  kubectl get ns $ns -ojson | jq '.spec.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/$ns/finalize" -f -
done

for ns in $(kubectl get ns --field-selector status.phase=Terminating -o jsonpath='{.items[*].metadata.name}')
do
  kubectl get ns $ns -ojson | jq '.metadata.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/$ns/finalize" -f -
done
1
  • 3
    This solution magically works. Thanks @rajendra sharma for posting the solution Mar 30, 2022 at 11:35
11

Firstly export your namespace name in env which got struck in Terminating state

export NAMESPACE="monitoring"

Then run below command to delete the Terminating namespace

kubectl get namespace $NAMESPACE -o json   | tr -d "\n" | sed "s/\"finalizers\": \[[^]]\+\]/\"finalizers\": []/"   | kubectl replace --raw /api/v1/namespaces/$NAMESPACE/finalize -f -
5

The tutorial you've used is not proper because deleting the namespace by removing finalizers is not good way to go since it could leave resources registered to a non existing namespace. Please take a look at this post: finalizer-kubernetes-ns.

You can try to find out which resources in the namespace are pending deletion by:

  • Finding all resources that still exist using command kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n $yotur-ns-to-delete
  • Checking if any apiservice is unavailable and hence doesn't serve its resources by executing command kubectl get apiservice|grep False

Take a look also at this problem: ns-kubernetes-stuck-terminating.

5
  • 1
    kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n prometheus error: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: the server is currently unable to handle the request No resources found in prometheus namespace. kubectl get apiservice|grep False v1beta1.custom.metrics.k8s.io prometheus/prometheus-adapter False (ServiceNotFound) 60d Jan 12, 2021 at 6:06
  • Try to execute command: $ kubectl delete apiservice v1beta1.custom.metrics.k8s.io
    – Malgorzata
    Jan 12, 2021 at 10:52
  • Error from server (NotFound): apiservices.apiregistration.k8s.io "v1beta1.custom.metrics.k8s.io" not found Jan 12, 2021 at 14:36
  • Can you also try $ kubectl api-resources --verbs=list --namespaced -o name \ | xargs -n 1 kubectl get --show-kind --ignore-not-found -l <label>=<value> -n <namespace> I think error occurred due to previously deleted finalizer which is improper, steps I have mention you should have followed from beginning.
    – Malgorzata
    Jan 12, 2021 at 16:44
  • Thanks @Malgorzata I had wanted to clean up a prometheus-adapter PoC and blew away everything with a k delete -f but forgot my NS was in there. ``` kubectl get apiservice|grep False v1beta1.custom.metrics.k8s.io prometheus/g-prometheus-adaptor-prometheus-adapter False (ServiceNotFound) 5h9m ``` Deleting custom metrics api allowed my NS to finally terminate. ``` kubectl delete apiservice v1beta1.custom.metrics.k8s.io ```
    – GorginZ
    Jul 3 at 5:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.