Using rabbitmqctl to reset a user password in RabbitMQ when using Kubernetes

We have a number of Kubernetes clusters running various things across our business. We use Azure’s flavour of ‘managed clusters’ AKS (Azure Kubernetes Service) and generally I find it great for removing some of the complexity of managing Kubernetes while still giving me the control I need compared to other ‘serverless’ hosting mechanisms like App Services and Serverless Containers.

One of the big upsides is upgrading the K8S version is usually just a button click away. But after upgrading 5 clusters, one of them encountered a strange issue, that after the update one of our rabbitMQ user’s passwords simply seemed to not work.

We use the excellent RabbitMQ Cluster Operator and it allows us to deploy rabbit very easily, you can port forward the management console and use the UI for lots of useful tasks. Its great…. if you can login.

When you google ‘reset rabbitmq user password’ there’s a lot of helpful answers around using some CLI tools like rabbitmqctl which initially I thought this was something that you could install locally and point at a cluster. But alas, after a bit of reading you find out that it is installed on the nodes themselves, and not something you can install with brew and use locally.

So… how do you use it when you have rabbit installed in a remote k8s cluster. That’s pretty simple, we use kubectl ‘s great exec function to run a command on one of our rabbit nodes.

Firstly, lets list everything to do with rabbit on our cluster

kubectl get all -l app.kubernetes.io/part-of=rabbitmq

That will spit out something that looks like this

NAME                            READY   STATUS    RESTARTS   AGE
pod/rabbitmq-cluster-server-0   1/1     Running   0          23h

NAME                             TYPE        CLUSTER-IP    EXTERNAL-IP   
service/rabbitmq-cluster         ClusterIP   10.0.128.91   <none>        
service/rabbitmq-cluster-nodes   ClusterIP   None          <none>

NAME                                       READY   AGE
statefulset.apps/rabbitmq-cluster-server   1/1     481d

(I removed some ports etc. for formatting here) 

So we can see that here I have one pod running a rabbitmq node rabbitmq-cluster-server-0
That’s the boy that will have rabbitmqctl installed. So using this, we can use the exec command to run the cli and use the reset password (or any other rabbitmqctl command) like this:
kubectl exec rabbitmq-cluster-server-0 -- /bin/sh -c "rabbitmqctl change_password mrrabbit somenewpassword"

and it worked a treat.