I had a "stuck" namespace that I deleted showing in this eternal "terminating" status.
34 Answers
Assuming you've already tried to force-delete resources like: Pods stuck at terminating status, and your at your wits' end trying to recover the namespace...
You can force-delete the namespace (perhaps leaving dangling resources):
(
NAMESPACE=your-rogue-namespace
kubectl proxy &
kubectl get namespace $NAMESPACE -o json |jq '.spec = {"finalizers":[]}' >temp.json
curl -k -H "Content-Type: application/json" -X PUT --data-binary @temp.json 127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize
)
This is a refinement of the answer here, which is based on the comment here.
I'm using the
jq
utility to programmatically delete elements in the finalizers section. You could do that manually instead.kubectl proxy
creates the listener at127.0.0.1:8001
by default. If you know the hostname/IP of your cluster master, you may be able to use that instead.The funny thing is that this approach seems to work even when using
kubectl edit
making the same change has no effect.
-
1Would be better if you move the
kubectl edit
solution on top as it's much easier and does the same thing, applicable to all objects.– R.Jun 11, 2019 at 4:43 -
11This is bad since it leaves leftovers in the cluster. Its more tricking yourself rather than actually fixing the issue. There is a reason the namespace can't be removed and you better fix that rather than going for a cosmetic solution. Jun 23, 2019 at 8:59
-
5@AntonioGomezAlvarado: Right. This wouldn't normally be the first thing that you try. But, unfortunately, it isn't always easy to discover the underlying problem. In some cases, the need to recover the namespace may be so that you can continue to get your work done -- not just cosmetics. Jul 9, 2019 at 13:56
-
1better soln - stackoverflow.com/a/59667608/429476 please select that as the answer for new clusters - Aug 7, 2020 at 10:42
-
3You should state, that you can bypass the API authentication with curl if you use
kubectl proxy
.– dmorlockJan 13, 2021 at 10:04
This is caused by resources still existing in the namespace that the namespace controller is unable to remove.
This command (with kubectl 1.11+) will show you what resources remain in the namespace:
kubectl api-resources --verbs=list --namespaced -o name \
| xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace>
Once you find those and resolve and remove them, the namespace will be cleaned up
-
7This shows no resources on my cluster, I don't think is always the solution.– Ben MossJan 19, 2021 at 17:30
-
6But definitely better to try first. Always force deleting the namespace is not good advice.– pstFeb 24, 2021 at 5:33
-
6Windows Powershell equivalent:
Foreach($x in (kubectl api-resources --verbs=list --namespaced -o name)){ kubectl get --show-kind --ignore-not-found -n YOUR_NAMESPACE_HERE $x }
– VenryxDec 5, 2021 at 4:27 -
1This worked for me. I had a dangling CRD associated with that namespace.– 8675309Jun 1, 2022 at 12:32
-
3I get error:
unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
Jan 17 at 22:56
As mentioned before in this thread there is another way to terminate a namespace using API not exposed by kubectl by using a modern version of kubectl where kubectl replace --raw
is available (not sure from which version). This way you will not have to spawn a kubectl proxy
process and avoid dependency with curl (that in some environment like busybox is not available). In the hope that this will help someone else I left this here:
kubectl get namespace "stucked-namespace" -o json \
| tr -d "\n" | sed "s/\"finalizers\": \[[^]]\+\]/\"finalizers\": []/" \
| kubectl replace --raw /api/v1/namespaces/stucked-namespace/finalize -f -
-
6
-
Great. Only thing that worked for us when there were no nodes up in the cluster.– sshowSep 1, 2020 at 12:59
-
-
1export NS=istio-system && kubectl get namespace "$NS" -o json \ | tr -d "\n" | sed "s/\"finalizers\": [[^]]\+]/\"finalizers\": []/" \ | kubectl replace --raw /api/v1/namespaces/$NS/finalize -f -– neoakrisApr 15, 2021 at 1:10
-
Need to remove the finalizer for kubernetes.
Step 1:
kubectl get namespace <YOUR_NAMESPACE> -o json > <YOUR_NAMESPACE>.json
- remove kubernetes from finalizers array which is under spec
Step 2:
kubectl replace --raw "/api/v1/namespaces/<YOUR_NAMESPACE>/finalize" -f ./<YOUR_NAMESPACE>.json
Step 3:
kubectl get namespace
You can see that the annoying namespace is gone.
-
-
9This can also be achieved with
kubectl edit <stuck_resource>
and delete the item in the 'finalizers' list. This has the advantage that it can be used to delete other stuck resources without knowing their api path. I used it to remove a stubborn CRD.– SiHaJun 18, 2021 at 7:34 -
1
-
2
-
2If you have
jq
installed, the following snippet might help you to remove the stuck namespace:namespace=stuck-namespace; kubectl get namespace $namespace -o json | jq '.spec.finalizers= []' | kubectl replace --raw "/api/v1/namespaces/$namespace/finalize" -f -
Mar 2, 2022 at 15:15
Solution:
Use command below without any changes. it works like a charm.
NS=`kubectl get ns |grep Terminating | awk 'NR==1 {print $1}'` && kubectl get namespace "$NS" -o json | tr -d "\n" | sed "s/\"finalizers\": \[[^]]\+\]/\"finalizers\": []/" | kubectl replace --raw /api/v1/namespaces/$NS/finalize -f -
Enjoy
-
3Not the best idea. This will most likely leave some dangling resources which caused the initial problem with termination.– bazeuszDec 20, 2020 at 23:35
-
2
-
3@Dentrax This command remove first terminating namespace in your list. Fore example if you have 3 namespace by terminating status you shold run this commant 3 times. If not work try another way to solve. Apr 12, 2022 at 12:05
-
Note: If your kube/config contain token for autentication of the user maybe got error. You should coppy conf file from /etc/kubernetes by this command and replace it old to Config. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config Oct 18, 2022 at 8:24
I loved this answer extracted from here It is just 2 commands.
In one terminal:
kubectl proxy
In another terminal:
kubectl get ns delete-me -o json | \
jq '.spec.finalizers=[]' | \
curl -X PUT http://localhost:8001/api/v1/namespaces/delete-me/finalize -H "Content-Type: application/json" --data @-
-
3I tried a few here, this is the first one which worked - thank you! May 7, 2021 at 8:41
-
1Removing the finalisers could lead to unregistered hanging resources later on.– LucatDec 5, 2021 at 9:38
-
3This worked for me. If you're Mac, first install
jq
utility using the commandbrew install jq
May 31, 2022 at 8:14 -
2Thank you. I was have issue removing the terminating NS. This helped me to get rid of my terminating rook-ceph namespace.– SanjeevJun 27, 2022 at 18:11
-
4
Single line command
kubectl patch ns <Namespace_to_delete> -p '{"metadata":{"finalizers":null}}'
Option : 2
If Patching not working you check the error if it's like in status
kubectl get ns <namespace-name>
Discovery failed for some groups, 1 failing: unable to retrieve the 22 complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently 23 unable to handle the request
You get the error details or API details which is failing for you, list available apiservice
kubectl get apiservice
Look for ones the AVAILABLE is False
kubectl delete apiservice <apiservice-name>
Example
kubectl delete apiservice metrics.k8s.io/v1beta1
and run above patch command again or wait for while namespace will get removed.
Read article for more : https://medium.com/@harsh.manvar111/kubernetes-namespace-stuck-on-terminating-state-25d0cda8e3ff
Extra - Simple trick
You can edit namespace on console only kubectl edit <namespace name>
remove/delete "Kubernetes" from inside the finalizer section(Should be like "finalizers": [ ]
) and press enter or save/apply changes.
in one step also you can do it.
Trick : 1
kubectl get namespace annoying-namespace-to-delete -o json > tmp.json
then edit
tmp.json
and remove "kubernetes" from FinalizersOpen another terminal Run command
kubectl proxy
and run below Curl
curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json https://localhost:8001/api/v1/namespaces/
<NAMESPACE NAME TO DELETE>
/finalize
and it should delete your namespace.
Step by step guide
Start the proxy using command :
kubectl proxy
kubectl proxy & Starting to serve on 127.0.0.1:8001
find namespace
kubectl get ns
{Your namespace name} Terminating 1d
put it in file
kubectl get namespace {Your namespace name} -o json > tmp.json
edit the file tmp.json
and remove the finalizers
}, "spec": { "finalizers": [ "kubernetes" ] },
after editing it should look like this
}, "spec": { "finalizers": [ ] },
we are almost there simply now run the curl with updating namespace value in it
curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/{Your namespace name}/finalize
and it's gone
**
-
4
-
1
-
For us it was the metrics-server
crashing.
So to check if this is relevant to you'r case with the following run: kubectl api-resources
If you get
error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
Then its probably the same issue
Credits goes to @javierprovecho here
-
for us it was a service catalogue CRD leftover:
servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
Feb 4, 2019 at 9:41 -
7
-
@vonGohren AFAIR we removed it CR to solve the issue, but I can't remember how we've figured it out, sorry Oct 30, 2019 at 12:08
-
You can deploy the latest (0.4.1) version of
metrics-server
to resolve this: kubectl apply -f github.com/kubernetes-sigs/metrics-server/releases/download/…;– imrissDec 25, 2020 at 17:36 -
5@vonGohren at that point you should indeed remove that resource. as mentioned in another answer run
kubectl api-resources --verbs=list --namespaced -o name \ | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace>
to find it; I then removed it withkubectl delete APIService v1beta1.custom.metrics.k8s.io
– iomvApr 28, 2021 at 0:40
I've written a one-liner Python3 script based on the common answers here. This script removes the finalizers
in the problematic namespace.
python3 -c "namespace='<my-namespace>';import atexit,subprocess,json,requests,sys;proxy_process = subprocess.Popen(['kubectl', 'proxy']);atexit.register(proxy_process.kill);p = subprocess.Popen(['kubectl', 'get', 'namespace', namespace, '-o', 'json'], stdout=subprocess.PIPE);p.wait();data = json.load(p.stdout);data['spec']['finalizers'] = [];requests.put('http://127.0.0.1:8001/api/v1/namespaces/{}/finalize'.format(namespace), json=data).raise_for_status()"
💡 rename
namespace='<my-namespace>'
with your namespace. e.g.namespace='trust'
Full script: https://gist.github.com/jossef/a563f8651ec52ad03a243dec539b333d
-
2
-
Woah that worked after a few other methods didn't, including deleting the finalizer myself by editing! Thank you! Jul 7, 2021 at 15:06
-
-
-
I write simple script to delete your stucking namespace based on @Shreyangi Saxena 's solution.
cat > delete_stuck_ns.sh << "EOF"
#!/usr/bin/env bash
function delete_namespace () {
echo "Deleting namespace $1"
kubectl get namespace $1 -o json > tmp.json
sed -i 's/"kubernetes"//g' tmp.json
kubectl replace --raw "/api/v1/namespaces/$1/finalize" -f ./tmp.json
rm ./tmp.json
}
TERMINATING_NS=$(kubectl get ns | awk '$2=="Terminating" {print $1}')
for ns in $TERMINATING_NS
do
delete_namespace $ns
done
EOF
chmod +x delete_stuck_ns.sh
This Script can detect all namespaces in Terminating
state, and delete it.
PS:
This may not work in MacOS, cause the native
sed
in macos is not compatible with GNUsed
.you may need install GNU sed in your MacOS, refer to this answer.
Please confirm that you can access your kubernetes cluster through command
kubectl
.Has been tested on kubernetes version
v1.15.3
Update
I found a easier solution:
kubectl patch RESOURCE NAME -p '{"metadata":{"finalizers":[]}}' --type=merge
-
2
-
Do not remove the finalizers to force a deletion as some resources will most likely remain dangling.– LucatFeb 15, 2022 at 17:32
Run kubectl get apiservice
For the above command you will find an apiservice with Available Flag=Flase.
So, just delete that apiservice using kubectl delete apiservice <apiservice name>
After doing this, the namespace with terminating status will disappear.
-
1In my case, all are Available=True, so this does not help (at least not always)– AsuMar 25, 2020 at 17:17
-
1Was getting a 404 with the other described methods. This solved my issue Jan 21, 2021 at 22:42
Forcefully deleting the namespace or removing finalizers is definitely not the way to go since it could leave resources registered to a non existing namespace.
This is often fine but then one day you won't be able to create a resource because it is still dangling somewhere.
The upcoming Kubernetes version 1.16 should give more insights into namespaces finalizers, for now I would rely on identification strategies. A cool script which tries to automate these is: https://github.com/thyarles/knsk
However it works across all namespaces and it could be dangerous. The solution it s based on is: https://github.com/kubernetes/kubernetes/issues/60807#issuecomment-524772920
tl;dr
- Checking if any apiservice is unavailable and hence doesn't serve its resources:
kubectl get apiservice|grep False
- Finding all resources that still exist via
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n $your-ns-to-delete
(credit: https://github.com/kubernetes/kubernetes/issues/60807#issuecomment-524772920)
-
2Agree, definitely not the way to go. Stuck removal is most often due to some resources not being properly deleted thus blocking the namespace termination. Check and remove those and the namespace will finally be removed. It's worth to mention that the namespace stuck in 'terminating' will block another namespace removal. At least this is what has happened to me, looks like removal is sequential in GKE.– bazeuszDec 20, 2020 at 23:23
here is a (yet another) solution. This uses jq
to remove the finalisers block from the json, and does not require kubectl proxy
:
namespaceToDelete=blah
kubectl get namespace "$namespaceToDelete" -o json \
| jq 'del(.spec.finalizers)' \
| kubectl replace --raw /api/v1/namespaces/$namespaceToDelete/finalize -f -
1. Using Curl Command
Issue Mentioned: https://amalgjose.com/2021/07/28/how-to-manually-delete-a-kubernetes-namespace-stuck-in-terminating-state/
export NAMESPACE=<specifice-namespace>
kubectl get namespace $NAMESPACE -o json > tempfile.json
Edit the JSON file and remove all values from spec.finalizers
Save it and then apply this command on separate tab (Must be open in separate Tab)
kubectl proxy
And run this command on same tab:
curl -k -H "Content-Type: application/json" -X PUT --data-binary @tempfile.json http://127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize
Check namespace if terminating namespace is removed or not
kubectl get namespaces
2. Using Kubectl Command
Issue Mentioned: https://aws.amazon.com/premiumsupport/knowledge-center/eks- terminated-namespaces/
- Save a JSON file similar to the following:
export NAMESPACE=<specific-namespace>
kubectl get namespace $NAMESPACE -o json > tempfile.json
- Edit the JSON file and remove all values from spec.finalizers
- To apply the changes, run a command similar to the following:
kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f ./tempfile.json
- Verify that the terminating namespace is removed:
kubectl get namespaces
Please try with below command:
kubectl patch ns <your_namespace> -p '{"metadata":{"finalizers":null}}'
-
1From Review: Command/Code-only answers are discouraged on Stack Overflow because they don't explain how it solves the problem. Please edit your answer to explain what this code does and how it answers the question, so that it is useful to the OP as well as other users with similar issues. See: How do I write a good answer?. Thanks Feb 9, 2021 at 12:29
-
1Only this one worked for me. No other solutions using POST json to finalize api work. Aug 19, 2021 at 21:12
In my case the problem was caused by a custom metrics.
To know what is causing the issue, just run this command:
kubectl api-resources | grep -i false
That should give you which api resources are causing the problem. Once identified just delete it:
kubectl delete apiservice v1beta1.custom.metrics.k8s.io
Once deleted, the namespace should disappear.
-
2
false
was lowercase in my case. so maybe:kubectl api-resources | grep -i false
Aug 1, 2021 at 22:00 -
I tried 3-5 options to remove ns, but only this one works for me.
This sh file will remove all namespaces with Terminating status
$ vi force-delete-namespaces.sh
$ chmod +x force-delete-namespaces.sh
$ ./force-delete-namespaces.sh
#!/usr/bin/env bash
set -e
set -o pipefail
kubectl proxy &
proxy_pid="$!"
trap 'kill "$proxy_pid"' EXIT
for ns in $(kubectl get namespace --field-selector=status.phase=Terminating --output=jsonpath="{.items[*].metadata.name}"); do
echo "Removing finalizers from namespace '$ns'..."
curl -H "Content-Type: application/json" -X PUT "127.0.0.1:8001/api/v1/namespaces/$ns/finalize" -d @- \
< <(kubectl get namespace "$ns" --output=json | jq '.spec = { "finalizers": [] }')
echo
echo "Force-deleting namespace '$ns'..."
kubectl delete namespace "$ns" --force --grace-period=0 --ignore-not-found=true
done
-
This is the best script to delete all stuck namespace kubernetes. A great time saver! Thank you! Apr 27 at 12:07
-
trap "kill $proxy_pid" SIGINT SIGTERM ERR EXIT
is working, your single quotes aren't :) Apr 27 at 12:21
Replace ambassador with your namespace
Check if the namespace is stuck
kubectl get ns ambassador
NAME STATUS AGE
ambassador Terminating 110d
This is stuck from a long time
Open a admin terminal/cmd prompt or powershell and run
kubectl proxy
This will start a local web server: Starting to serve on 127.0.0.1:8001
Open another terminal and run
kubectl get ns ambassador -o json >tmp.json
edit the tmp.json using vi or nano
from this
{
"apiVersion": "v1",
"kind": "Namespace",
"metadata": {
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"name\":\"ambassador\"}}\n"
},
"creationTimestamp": "2021-01-07T18:23:28Z",
"deletionTimestamp": "2021-04-28T06:43:41Z",
"name": "ambassador",
"resourceVersion": "14572382",
"selfLink": "/api/v1/namespaces/ambassador",
"uid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
},
"spec": {
"finalizers": [
"kubernetes"
]
},
"status": {
"conditions": [
{
"lastTransitionTime": "2021-04-28T06:43:46Z",
"message": "Discovery failed for some groups, 3 failing: unable to retrieve the complete list of server APIs: compose.docker.com/v1alpha3: an error on the server (\"Internal Server Error: \\\"/apis/compose.docker.com/v1alpha3?timeout=32s\\\": Post https://0.0.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews: write tcp 0.0.0.0:53284-\u0026gt;0.0.0.0:443: write: broken pipe\") has prevented the request from succeeding, compose.docker.com/v1beta1: an error on the server (\"Internal Server Error: \\\"/apis/compose.docker.com/v1beta1?timeout=32s\\\": Post https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews: write tcp 0.0.0.0:5284-\u0026gt;10.96.0.1:443: write: broken pipe\") has prevented the request from succeeding, compose.docker.com/v1beta2: an error on the server (\"Internal Server Error: \\\"/apis/compose.docker.com/v1beta2?timeout=32s\\\": Post https://0.0.0.0:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews: write tcp 1.1.1.1:2284-\u0026gt;0.0.0.0:443: write: broken pipe\") has prevented the request from succeeding",
"reason": "DiscoveryFailed",
"status": "True",
"type": "NamespaceDeletionDiscoveryFailure"
},
{
"lastTransitionTime": "2021-04-28T06:43:49Z",
"message": "All legacy kube types successfully parsed",
"reason": "ParsedGroupVersions",
"status": "False",
"type": "NamespaceDeletionGroupVersionParsingFailure"
},
{
"lastTransitionTime": "2021-04-28T06:43:49Z",
"message": "All content successfully deleted",
"reason": "ContentDeleted",
"status": "False",
"type": "NamespaceDeletionContentFailure"
}
],
"phase": "Terminating"
}
}
to
{
"apiVersion": "v1",
"kind": "Namespace",
"metadata": {
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"name\":\"ambassador\"}}\n"
},
"creationTimestamp": "2021-01-07T18:23:28Z",
"deletionTimestamp": "2021-04-28T06:43:41Z",
"name": "ambassador",
"resourceVersion": "14572382",
"selfLink": "/api/v1/namespaces/ambassador",
"uid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
},
"spec": {
"finalizers": []
}
}
by deleting status and kubernetes inside finalizers
Now use the command and replace ambassador with your namespace
curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/ambassador/finalize
you will see another json like before then run
then run the command
kubectl get ns ambassador
Error from server (NotFound): namespaces "ambassador" not found
If it still says terminating or any other error make sure you format your json in a proper way and try the steps again.
There are a couple of things you can run. But what this usually means, is that the automatic deletion of namespace was not able to finish, and there is a process running that has to be manually deleted. To find this you can do these things:
Get all prossesse attached to the name space. If this does not result in anything move on to next suggestions
$ kubectl get all -n your-namespace
Some namespaces have apiserivces attached to them and it can be troublesome to delete. This can for that matter be whatever resources you want. Then you delete that resource if it finds anything
$ kubectl get apiservice|grep False
But the main takeaway, is that there might be some things that is not completly removed. So you can see what you initially had in that namespace, and then see what things is spun up with your YAMLs to see the processes up. Or you can start to google why wont service X be properly removed, and you will find things.
Run the following command to view the namespaces that are stuck in the Terminating state:
kubectl get namespaces
Select a terminating namespace and view the contents of the namespace to find out the finalizer. Run the following command:
kubectl get namespace -o yaml
Your YAML contents might resemble the following output:
apiVersion: v1
kind: Namespace
metadata:
creationTimestamp: 2019-12-25T17:38:32Z
deletionTimestamp: 2019-12-25T17:51:34Z
name: <terminating-namespace>
resourceVersion: "4779875"
selfLink: /api/v1/namespaces/<terminating-namespace>
uid: ******-****-****-****-fa1dfgerz5
spec:
finalizers:
- kubernetes
status:
phase: Terminating
Run the following command to create a temporary JSON file:
kubectl get namespace -o json >tmp.json
Edit your tmp.json file. Remove the kubernetes value from the finalizers field and save the file. Output would be like:
{
"apiVersion": "v1",
"kind": "Namespace",
"metadata": {
"creationTimestamp": "2018-11-19T18:48:30Z",
"deletionTimestamp": "2018-11-19T18:59:36Z",
"name": "<terminating-namespace>",
"resourceVersion": "1385077",
"selfLink": "/api/v1/namespaces/<terminating-namespace>",
"uid": "b50c9ea4-ec2b-11e8-a0be-fa163eeb47a5"
},
"spec": {
},
"status": {
"phase": "Terminating"
}
}
To set a temporary proxy IP and port, run the following command. Be sure to keep your terminal window open until you delete the stuck namespace:
kubectl proxy
Your proxy IP and port might resemble the following output:
Starting to serve on 127.0.0.1:8001
From a new terminal window, make an API call with your temporary proxy IP and port:
curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/your_terminating_namespace/finalize
Your output would be like:
{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name": "<terminating-namespace>",
"selfLink": "/api/v1/namespaces/<terminating-namespace>/finalize",
"uid": "b50c9ea4-ec2b-11e8-a0be-fa163eeb47a5",
"resourceVersion": "1602981",
"creationTimestamp": "2018-11-19T18:48:30Z",
"deletionTimestamp": "2018-11-19T18:59:36Z"
},
"spec": {
},
"status": {
"phase": "Terminating"
}
}
The finalizer parameter is removed. Now verify that the terminating namespace is removed, run the following command:
kubectl get namespaces
Edit: It is not recommended to remove finalizers. Correct approach would be:
- Delete all the resources in the namespace.
My usual workspace is a small k8s cluster which I frequently destroy and rebuild it back, and that's why removing finalizers method works for me.
Original answer: I usually run into same problem.
This is what I do
kubectl get ns your-namespace -o json > ns-without-finalizers.json
Edit ns-without-finalizers.json. replace all finalizers with empty array.
Run kubectl proxy ( usually run it on another terminal )
Then curl this command
curl -X PUT http://localhost:8001/api/v1/namespaces/your-namespace/finalize -H "Content-Type: application/json" --data @ns-without-finalizers.json
If the namespace stuck in Terminating while the resources in that namespace have been already deleted, you can patch the finalizers
of the namespace before deleting it:
kubectl patch ns ns_to_be_deleted -p '{"metadata":{"finalizers":null}}';
then
kubectl delete ns ns_to_be_deleted;
Edit:
Please check @Antonio Gomez Alvarado's Answer first. The root cause could be the metrics server
that mentioned in that answer.
The only way I found to remove a "terminating
" namespace is by deleting the entry inside the "finalizers" section. I've tried to --force
delete it and to --grace-period=0
none of them worked, however, this method did:
on a command line display the info from the namespace:
$ kubectl get namespace your-rogue-namespace -o yaml
This will give you yaml output, look for a line that looks similar to this:
deletionTimestamp: 2018-09-17T13:00:10Z
finalizers:
- Whatever content it might be here...
labels:
Then simply edit the namespace configuration and delete the items inside that finalizers container.
$ kubectl edit namespace your-rogue-namespace
This will open an editor (in my case VI), went over the line I wanted to delete and deleted it, I pressed the D key twice to delete the whole line.
Save it, quit your editor, and like magic. The rogue-namespace should be gone.
And to confirm it just:
$ kubectl get namespace your-rogue-namespace -o yaml
-
3this short-circuits cleanup of resources associated with the namespace and leaves orphaned resources associated with that namespace Sep 18, 2018 at 0:27
-
it might be crashing/missing metrics API server or service catalogue API server, see answers from JordanLiggitt and AntonioGomezAlvarado Feb 4, 2019 at 9:42
Completing the already great answer by nobar. If you deployed your cluster with Rancher there is a caveat.
Rancher deployments change EVERY api call, prepending /k8s/clusters/c-XXXXX/
to the URLs.
The id of the cluster on rancher (c-XXXXX
) is something you can easily get from the Rancher UI, as it will be there on the URL.
So after you get that cluster id c-xxxx, just do as nobar says, just changing the api call including that rancher bit.
(
NAMESPACE=your-rogue-namespace
kubectl proxy &
kubectl get namespace $NAMESPACE -o json |jq '.spec = {"finalizers":[]}' >temp.json
curl -k -H "Content-Type: application/json" \
-X PUT --data-binary @temp.json \
127.0.0.1:8001/k8s/clusters/c-XXXXX/api/v1/namespaces/$NAMESPACE/finalize
)
For anyone looking for few commands for later version of Kubernetes, this helped me.
NAMESPACE=mynamespace
kubectl get namespace $NAMESPACE -o json | sed 's/"kubernetes"//' | kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f -
Tested in Kubernetes v1.24.1
Note: the command replaces all kubernetes
strings in the json, please use it with care
Something similar happened to me in my case it was pv & pvc , which I forcefully removed by setting finalizers to null. Check if you could do similar with ns
kubectl patch pvc <pvc-name> -p '{"metadata":{"finalizers":null}}'
For namespaces it'd be
kubectl patch ns <ns-name> -p '{"spec":{"finalizers":null}}'
Debugging a similar issue.
Two important things to consider:
1 ) Think twice before deleting finalizers
from your namespace because there might be resources that you wouldn't want to automatically delete or at least understand what was deleted for troubleshooting.
2 ) Commands like kubectl api-resources --verbs=list
might not give you resources that were created by external crds.
In my case:
I viewed my namespace real state (that was stuck on Terminating
) with kubectl edit ns <ns-name>
and under status -> conditions
I saw that some external crds that I installed were failed to be deleted because they add a finalizers
defined:
- lastTransitionTime: "2021-06-14T11:14:47Z"
message: 'Some content in the namespace has finalizers remaining: finalizer.stackinstall.crossplane.io
in 1 resource instances, finalizer.stacks.crossplane.io in 1 resource instances'
reason: SomeFinalizersRemain
status: "True"
type: NamespaceFinalizersRemaining
curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json 127.0.0.1:8001/k8s/clusters/c-mzplp/api/v1/namespaces/rook-ceph/finalize
This worked for me, the namespace is gone.
Detailed explanation can be found in the link https://github.com/rook/rook/blob/master/Documentation/ceph-teardown.md.
This happened when I interrupted kubernetes installation(Armory Minnaker). Then I proceeded to delete the namespace and reinstall it. I was stuck with pod in terminating status due to finalizers. I got the namespace into tmp.json, removed finalizers from tmp.json file and did the curl command. Once I get past this issue, I used scripts for uninstalling the cluster to remove the residues and did a reinstallation.
-
perhaps you would like to break down your URL and explain what is going on?– ximbalJun 9, 2020 at 23:32
kubectl edit namespace ${stucked_namespace}
Then delete finalizers
in vi mode and save.
It worked in my case.
Editing NS yaml manually didn't work for me, no error was thrown on editing but changes did not take effect.
This worked for me:
In one session:
kubectl proxy
in another shell:
kubectl get ns <rouge-ns> -o json | jq '.spec.finalizers=[]' | curl -X PUT http://localhost:8001/api/v1/namespaces/<rouge-ns>/finalize -H "Content-Type: application/json" --data @-