Deleting Custom Resources in a Kubernetes Namespace

The custom resources I thought I deleted were in fact around all this time!?

When working with Kubernetes and custom resources, you need to be able to go back to a clean state.

Unfortunately, almost all kubectl commands are ignoring custom resources by default. Both when it comes to displaying and handling all resources.

At Stackable, we use custom resources and CRDs a lot to configure our operators.

Running kubectl delete all --all for example, will only affect built-in Kubernetes resources. Not custom ones.

Here are a few methods you can use to make sure that your namespace is as clean as possible.

A Word of Caution!

All of the methods below are meant for disposable development clusters. You probably don’t want to risk using them anywhere near a productive system, until you fully understand them!

Some of those methods should be used with utmost care. They will affect your whole cluster. Only use them in an environment where you are sure that no valuable data or workloads will be affected.

Method 1: Complete Destruction

The most thorough method to make sure no unwanted resources remain: delete the cluster and create a new one.

Radical, right? But it can also be simple and reliable. This method is only advisable when handling disposable clusters. Ones which are easy to recreate, preferably with a single command.

If you want to adopt such a workflow, “Kubernetes in Docker” (also known as kind) is a lightweight way to create and destroy Kubernetes clusters on your local machine with little overhead.

Here’s how the complete process can look:

$ kind create cluster --name tmp-cluster
# this creates a cluster called tmp-cluster

$ kind delete cluster --name tmp-cluster
# said cluster goes poof

$ kind create cluster --name tmp-cluster
# there it is again, new and shiny!

Method 2: Remove the CRDs

Danger! This can be just as destructive as deleting the whole cluster. Custom Resource Definitions (CRDs) are resources which are not namespaced. They are defined cluster-wide, and removing them will have implications on the complete cluster.

If you are SURE that no important CRDs exist in your cluster, besides the ones you want gone. If you remove CRDs, all custom resources which reference them will be removed as well. Here’s a command to get rid of all CRDs on the cluster:

$ kubectl delete crds --all

Handle with care!

Method 3: Delete the Namespace

Next on the list of “methods which work, but might be a bit too crude”: deleting the complete namespace.

Note: unlike the previous methods, this will not remove existing cluster-wide resources, like CRDs themselves. Only what’s inside the namespace.

You can only do this, if you have been working in a non-default and non-protected namespace. You won’t be able to delete the default namespace, nor kube-system for example.

However, if you created a new namespace for your work (my-random-namespace for example)? You’ll be able to delete and recreate it as you wish.

Note: Once again, only do this with namespaces where nothing important or valuable is located!

$ kubectl delete ns my-random-namespace
$ kubectl create ns my-random-namespace

Method 4: Only Delete Selected Custom Resources in One Specific Namespace

Finally, here is a way to only remove some custom resources in a single namespace. And only some of the CRDs on the cluster.

Labels are also really helpful for this kind of work! If you don’t love and value k8s labels, this is your chance to reconsider. We’ll rely on CRD names instead of labels here.

As an example, we’ll remove custom resources that are usually created by Stackable operators.

We are going to write a long-ish shell command to do it. I’m sure there are more elegant ways – view this as a flexible proof of concept.

You can see the complete command below. Don’t worry, we will break this up into parts below for understanding right away.

$ kubectl get crd | grep stackable | perl -anle 'print $F[0]' | xargs -I {} kubectl --namespace my-random-namespace delete {} --all

Intermission: What’s Going on Here?

Let’s go through all of those commands one by one and find out.

kubectl get crd

This first command gets information about all CRDs in the current k8s cluster. The output is piped into the next command:

grep stackable

which removes lines from the previous output which don’t contain the string “stackable”.

Note: all Stackable CRDs contain “stackable” in their name, making this grep approach possible.

perl -anle 'print $F[0]'

Now, Perl is used to cut out the names of the custom resources from each line. You could use something like awk instead, or cut, or sed (see this SO thread for the syntax).

Note: you could also tell kubectl to output something custom in the first command, but that’s another topic.

xargs -I {} kubectl --namespace my-random-namespace delete {} --all

This last command will call kubectl once for every line passed from the previous command. xargs replaces the second {} with every CRD name. If the output of the previous command is:

crd-name-1
crd-name-2
crd-name-3

The following kubectl command would be executed through xargs:

$ kubectl --namespace my-random-namespace delete crd-name-1 --all
$ kubectl --namespace my-random-namespace delete crd-name-2 --all
$ kubectl --namespace my-random-namespace delete crd-name-3 --all

Phew! That was a lot of steps. But in the end, all custom resources containing “stackable” in their CRD’s name are deleted from the my-random-namespace namespace.

All Clean!

I hope this writeup has helped you get an overview of a few ways to remove custom resources from your development cluster. Make sure to use these methods with care, as some of them may remove more than you intend.

Enjoy those empty and clean namespaces!