10

I am using RabbitMQ version 3.7.17

As my AWS hard disk was completely occupied(100% full). Due to which all the services stopped working
Solution to this: I extended AWS server memory and than tried to start all the API services after that it started throwing error. (Post this it started giving error)

Connection.open: (541) INTERNAL_ERROR - access to vhost '/' refused for user 'XYZ': vhost '/' is down

Restarted RabbitmMQ server using the below code still it was giving error:

sudo service rabbitmq-server restart

If I checked the permission for my user using:

sudo rabbitmqctl list_permissions --vhost /

Response shows that user has all the access.

Listing permissions for vhost "/" ...
user    configure   write   read
XYZ .*  .*  .*

Thank You.

5
  • @cabreracanal the server memory was expanded it was resolved but rabbitmq stopped working after that which was working fine before that. It started throwing above mention error.
    – Avi
    Commented Nov 6, 2019 at 10:53
  • I don't know if you are talking about server's memory (RAM) or server's volume capacity (disk). In your description, you said that expanding the memory did not solve the problem, so this is why I'm asking if you are talking about RAM or disk. Commented Nov 6, 2019 at 10:57
  • 2
    @cabreracanal I have updated the question. Initially it was working fine after aws hard disk was full and memory and it was expanded than rabbitmq started throwing error. There no problem with server it RabbitMQ that is giving error.
    – Avi
    Commented Nov 6, 2019 at 11:13
  • @cabreracanal your are telling me solution of aws memory which is been resolved and all other application are working fine such as apache2, celery etc only rabbitmq is throwing error.
    – Avi
    Commented Nov 6, 2019 at 11:40
  • Please refer my fix in stackoverflow.com/a/62019910/4817250 Commented May 26, 2020 at 10:24

1 Answer 1

27

As the Memory was full the RabbitMQ that was processing was not completed which resulted it in error in vhost.

When tried to restart vhost sudo rabbitmqctl restart_vhost it got error:

ERROR:

Failed to start vhost '/' on node 'rabbit@ip-172-31-16-172'Reason: {:shutdown, {:failed_to_start_child, :rabbit_vhost_process, {:error, {{{:function_clause, [{:rabbit_queue_index, :journal_minus_segment1, [{{true, <<230, 140, 82, 5, 193, 81, 136, 75, 11, 91, 31, 232, 119, 30, 99, 112, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 144>>, <<131, 104, 6, 100, 0, 13, 98, 97, 115, 105, 99, 95, 109, 101, 115, 115, 97, 103, 101, 104, 4, 100, 0, 8, 114, 101, 115, 111, 117, 114, ...>>}, :no_del, :no_ack}, {{true, <<230, 140, 82, 5, 193, 81, 136, 75, 11, 91, 31, 232, 119, 30, 99, 112, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 144>>, <<131, 104, 6, 100, 0, 13, 98, 97, 115, 105, 99, 95, 109, 101, 115, 115, 97, 103, 101, 104, 4, 100, 0, 8, 114, 101, 115, 111, 117, ...>>}, :del, :no_ack}], [file: 'src/rabbit_queue_index.erl', line: 1231]}, {:rabbit_queue_index, :"-journal_minus_segment/3-fun-0-", 4, [file: 'src/rabbit_queue_index.erl', line: 1208]}, {:array, :sparse_foldl_3, 7, [file: 'array.erl', line: 1684]}, {:array, :sparse_foldl_2, 9, [file: 'array.erl', line: 1678]}, {:rabbit_queue_index, :"-recover_journal/1-fun-0-", 1, [file: 'src/rabbit_queue_index.erl', line: 915]}, {:lists, :map, 2, [file: 'lists.erl', line: 1239]}, {:rabbit_queue_index, :segment_map, 2, [file: 'src/rabbit_queue_index.erl', line: 1039]}, {:rabbit_queue_index, :recover_journal, 1, [file: 'src/rabbit_queue_index.erl', line: 906]}]}, {:gen_server2, :call, [#PID<10397.473.0>, :out, :infinity]}}, {:child, :undefined, :msg_store_persistent, {:rabbit_msg_store, :start_link, [:msg_store_persistent, '/var/lib/rabbitmq/mnesia/rabbit@ip-172-31-16-172/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L', [], {#Function<2.32138423/1 in :rabbit_queue_index>, {:start, [{:resource, "/", :queue, "xx_queue"}, {:resource, "/", :queue, "app_xxx_queue"}, {:resource, "/", :queue, "default"}, {:resource, "/", :queue, "xx_priority_queue"}, {:resource, "/", :queue, "xxx_queue"}, {:resource, "/", :queue, "xxxx_queue"}, {:resource, "/", :queue, "yyy_queue"}, {:resource, "/", :queue, "zzz_queue"}, {:resource, "/", :queue, "aaa_queue"}]}}]}, :transient, 30000, :worker, [:rabbit_msg_store]}}}}}

STEPS TO SOLVE IT

  1. Stop your app node by below command.
sudo rabbitmqctl stop_app
  1. Reset your node by below command. Removes the node from any cluster it belongs to, removes all data from the management database, such as configured users and vhosts, and deletes all persistent messages.(Be careful while using it.) To backup your data before reset look here
sudo rabbitmqctl reset
  1. Start your Node by below command.
sudo rabbitmqctl start_app
  1. Restart your vhost by below commad.
sudo rabbitmqctl restart_vhost

And if you are using some application that is depended on RabbitMQ. Such as I using celery you will have to restart them again.

This was the link that helped me to solve it.

4
  • 1
    it works perfect for docker's container as well. Thank you very much
    – Alex Gore
    Commented Aug 25, 2021 at 19:24
  • 1
    If you don't want to remove all data (e.g. only one vhost is corrupted), use rabbitmqctl delete_vhost badvhost and then recreate it. If like me you have your rabbitmq definitions in a json file, that would be rabbitmqadmin import /path/to/definitions.json
    – Aaron D
    Commented May 24, 2022 at 15:27
  • That helped me too. But is there a way to save the message queue so I can resume where it left off?
    – ruslaniv
    Commented Nov 29, 2022 at 7:48
  • Hi @ruslaniv good question, I have added a link to backup your data before reset in my answer. Hope that is helpful.
    – Avi
    Commented Nov 29, 2022 at 12:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.