{"id":13,"date":"2015-11-20T14:09:44","date_gmt":"2015-11-20T22:09:44","guid":{"rendered":"http:\/\/iaastalk.com\/?p=13"},"modified":"2016-02-04T17:30:11","modified_gmt":"2016-02-05T01:30:11","slug":"rabbitmq-rescue","status":"publish","type":"post","link":"http:\/\/iaastalk.com\/?p=13","title":{"rendered":"RabbitMQ rescue"},"content":{"rendered":"

RabbitMQ cluster shows partitioned badly. I was trying to rebuild.<\/p>\n

First of all I had issue with stopping \/ starting.<\/p>\n

Start\/Stop hanging:<\/p>\n

    \n
  1. killall -u rabbitmq -q<\/li>\n
  2. backup exiting rabbitmq.config to rabbitmq.org\u00a0 (\/etc\/rabbitmq\/rabbitmq.config)<\/li>\n
  3. remove all other cluster members from rabbitmq.config and only keep current host.<\/li>\n
  4. rm -ef \/var\/lib\/rabbitmq\/mnesia\u00a0 (this needs to verified. Pls check \/etc\/rabbitmq\/rabbitmq-env.conf to see where RABBITMQ_MNESIA_BASE is pointing to.\u00a0 Another thing needs to pay attention to the permission on this directory. The owner has to be rabbitmq, otherwise rabbitmq will fail to start because it’s running as user rabbitmq and\u00a0 can not create directories and files without sufficient permission.<\/li>\n
  5. service rabbitmq-server restart<\/li>\n<\/ol>\n

    Now server should start without issue. now let’s do further clean up.<\/p>\n

      \n
    1. rabbitmqctl stop_app<\/li>\n
    2. rabbitmqctl force_reset<\/li>\n
    3. rabbitmqctl start_app<\/li>\n
    4. rbbitmqctl stop<\/li>\n<\/ol>\n

      There should be no errors<\/p>\n

      Now restore \/etc\/rabbitmq\/rabbitmq.config from backup rabbitmq.org and start rabbitmq again.<\/p>\n

        \n
      1. service rabbitmq-server start<\/li>\n<\/ol>\n

        Now we should have all nodes running<\/p>\n

        Assume node001 is master and start service on node001:<\/p>\n

          \n
        1. rabbitmqctl start_app<\/li>\n<\/ol>\n

           <\/p>\n

          All other nodes:<\/p>\n

            \n
          1. rabbitmqctl stop_app<\/li>\n
          2. rabbitmqctl join_cluster rabbit@node001<\/li>\n
          3. rabbitmqctl start_app<\/li>\n<\/ol>\n

             <\/p>\n

            Check cluster status on all nodes:<\/p>\n

              \n
            1. rabbitmqctl cluster_status<\/li>\n<\/ol>\n

              should have no more partition and all nodes should be running:<\/p>\n

              Cluster status of node rabbit@phx04rmqa001 …
              \n[{nodes,[{disc,[rabbit@node001,rabbit@node002, rabbit@node003]}]},
              \n{running_nodes,[[rabbit@node001,rabbit@node002, rabbit@node003]},
              \n{partitions,[]}]<\/p>\n

              partitions should be empty []\u00a0 .<\/p>\n

               <\/p>\n

              don’t forget to enabled HA queue etc.<\/p>\n

              rabbitmqctl set_policy ha-all “” ‘{“ha-mode”:”all”,”ha-sync-mode”:”automatic”}’<\/p>\n

              Another option is cluster_partition_handling in configuration that changes<\/span> how partition recovery works ( default is ignore):<\/p>\n

                \n
              • pause_minority<\/span><\/li>\n
              • {pause_if_all_down, [nodes], ignore | autoheal}<\/span><\/li>\n
              • autoheal<\/span><\/li>\n<\/ul>\n

                combined with loadbalancer, things could behave strange. Highly recommend no VIP to front mq server.<\/p>\n

                Another commonly see issue is rabbitmq stops responding due to messages flooded particular queues with no consumer. It may completely stopped responding or delay message delivering. A workaround is to put a size on the queue to force RMQ not to exhaust memry:<\/p>\n

                rabbitmqctl set_policy POLICY_NAME\u00a0 “QUEUE_NAME” ‘{“max-length”:100}’ –apply-to queues<\/p>\n

                 <\/p>\n

                 <\/p>\n

                Good luck<\/p>\n","protected":false},"excerpt":{"rendered":"

                RabbitMQ cluster shows partitioned badly. I was trying to rebuild. First of all I had issue with stopping \/ starting. Start\/Stop hanging: killall -u rabbitmq -q backup exiting rabbitmq.config to rabbitmq.org\u00a0 (\/etc\/rabbitmq\/rabbitmq.config) remove all other cluster members from rabbitmq.config and only keep current host. rm -ef \/var\/lib\/rabbitmq\/mnesia\u00a0 (this needs to verified. Pls check \/etc\/rabbitmq\/rabbitmq-env.conf to […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"_links":{"self":[{"href":"http:\/\/iaastalk.com\/index.php?rest_route=\/wp\/v2\/posts\/13"}],"collection":[{"href":"http:\/\/iaastalk.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/iaastalk.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/iaastalk.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/iaastalk.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=13"}],"version-history":[{"count":6,"href":"http:\/\/iaastalk.com\/index.php?rest_route=\/wp\/v2\/posts\/13\/revisions"}],"predecessor-version":[{"id":29,"href":"http:\/\/iaastalk.com\/index.php?rest_route=\/wp\/v2\/posts\/13\/revisions\/29"}],"wp:attachment":[{"href":"http:\/\/iaastalk.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=13"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/iaastalk.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=13"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/iaastalk.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=13"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}