记一次RabbitMQ服务器异常断电之后，服务重启异常的处理过程

2024-04-23 服务器 0

湖蓝几何球体LinkedIn Banner.png
转载说明：如果您喜欢这篇文章并打算转载它，请私信作者取得授权。感谢您喜爱本文，请文明转载，谢谢。

问题描述：

机房突然停电，rabbitmq的主机异常断电，集群服务全部需要重启。但是在执行service rabbitmq-server start 启动主节点服务的时候，没有反应，服务没有启动，命令也执行卡住了。必须Ctrl+C结束进程：

[root@master-2 rabbitmq]# service rabbitmq-server startStarting rabbitmq-server (via systemctl):  ^C[root@master-2 rabbitmq]#

查看/var/log/rabbitmq/startup_log 发现有如下报错信息：

[root@master-2 rabbitmq]# tail -1000 startup_log BOOT FAILED===========Timeout contacting cluster nodes: ['rabbit@s1-1','rabbit@slave-2'].BACKGROUND==========This cluster node was shut down while other nodes were still running.To avoid losing data, you should start the other nodes first, thenstart this one. To force this node to start, first invoke"rabbitmqctl force_boot". If you do so, any changes made on othercluster nodes after this one was shut down may be lost.DIAGNOSTICS===========attempted to contact: ['rabbit@s1-1','rabbit@slave-2']rabbit@s1-1:  * connected to epmd (port 4369) on s1-1  * epmd reports: node 'rabbit' not running at all                  no other nodes on s1-1  * suggestion: start the noderabbit@slave-2:  * unable to connect to epmd (port 4369) on slave-2: address (cannot connect to host/port)current node details:- node name: 'rabbit@master-2'- home dir: /var/lib/rabbitmq- cookie hash: oqRyxdQQXO31mzM8U0ysNA=={"init terminating in do_boot",timeout_waiting_for_tables}

解决方法1：

根据/var/log/rabbitmq/startup_log日志最后的报错信息{“init terminating in do_boot”,timeout_waiting_for_tables}，在网上查询到原因，和linux下rabbitmq大致有关系的，主要有这三种说法：
1、5672端口被占用了，导致服务起不来
2、/var/log/rabbitmq目录的权限不对，需要重新赋权限
3、/var/lib/rabbitmq/mnesia这个数据目录异常，删除原来的数据目录，重新启动服务

方法一：检查端口，发现并没有5672的这个端口：

[root@master-2 rabbitmq]# netstat -anp|grep 5672tcp        0      0 193.168.0.90:3306       131.10.10.120:56727     ESTABLISHED 3666/mysqld         tcp6       0      0 193.168.0.90:56727      193.168.0.93:9092       ESTABLISHED 4891/java           [root@master-2 rabbitmq]# netstat -ano|grep 5672tcp        0      0 193.168.0.90:3306       131.10.10.120:56727     ESTABLISHED keepalive (54.12/0/0)tcp6       0      0 193.168.0.90:56727      193.168.0.93:9092       ESTABLISHED keepalive (50.53/0/0)

方法二：修改/var/log/rabbitmq权限，进去/var/log/rabbitmq/目录，发现该目录下面的文件确实存在权限不统一的问题，于是修改权限重新启动服务，还是失败：

[root@master-2 rabbitmq]# cd /var/log/rabbitmq/[root@master-2 rabbitmq]# lltotal 11740-rw-r--r--  1 rabbitmq rabbitmq   29075 May 14 11:14 rabbit@master-2.log-rw-r--r--  1 rabbitmq rabbitmq  159053 Apr 29 03:19 rabbit@master-2.log-20180429.gz-rw-r--r--  1 rabbitmq rabbitmq 1756006 May  7 03:11 rabbit@master-2.log-20180507.gz-rw-r--r--  1 rabbitmq rabbitmq 9881632 May 13 03:17 rabbit@master-2.log-20180513-rw-r--r--  1 rabbitmq rabbitmq    3108 May 14 11:14 rabbit@master-2-sasl.log-rw-r--r--  1 rabbitmq rabbitmq     950 Apr 28 14:22 rabbit@master-2-sasl.log-20180429.gz-rw-r--r--  1 rabbitmq rabbitmq    1677 May  4 15:25 rabbit@master-2-sasl.log-20180507.gz-rw-r--r--  1 rabbitmq rabbitmq  159530 May 11 10:11 rabbit@master-2-sasl.log-20180513-rw-r--r--  1 root     root           0 May  7 15:14 shutdown_err-rw-r--r--  1 root     root          44 May  7 15:14 shutdown_log-rw-r--r--. 1 root     root         103 May 14 11:15 startup_err-rw-r--r--. 1 root     root        1323 May 14 11:15 startup_log[root@master-2 rabbitmq]# chown -R rabbitmq:rabbitmq /var/log/rabbitmq/[root@master-2 rabbitmq]# lltotal 11740-rw-r--r--  1 rabbitmq rabbitmq   29075 May 14 11:14 rabbit@master-2.log-rw-r--r--  1 rabbitmq rabbitmq  159053 Apr 29 03:19 rabbit@master-2.log-20180429.gz-rw-r--r--  1 rabbitmq rabbitmq 1756006 May  7 03:11 rabbit@master-2.log-20180507.gz-rw-r--r--  1 rabbitmq rabbitmq 9881632 May 13 03:17 rabbit@master-2.log-20180513-rw-r--r--  1 rabbitmq rabbitmq    3108 May 14 11:14 rabbit@master-2-sasl.log-rw-r--r--  1 rabbitmq rabbitmq     950 Apr 28 14:22 rabbit@master-2-sasl.log-20180429.gz-rw-r--r--  1 rabbitmq rabbitmq    1677 May  4 15:25 rabbit@master-2-sasl.log-20180507.gz-rw-r--r--  1 rabbitmq rabbitmq  159530 May 11 10:11 rabbit@master-2-sasl.log-20180513-rw-r--r--  1 rabbitmq rabbitmq       0 May  7 15:14 shutdown_err-rw-r--r--  1 rabbitmq rabbitmq      44 May  7 15:14 shutdown_log-rw-r--r--. 1 rabbitmq rabbitmq     103 May 14 11:15 startup_err-rw-r--r--. 1 rabbitmq rabbitmq    1323 May 14 11:15 startup_log

但是修改了权限之后，服务还是起不来：

[root@master-2 rabbitmq]# service rabbitmq-server start    Starting rabbitmq-server (via systemctl):  ^C[root@master-2 rabbitmq]#

方法三：删除原有的数据目录，然后重新启动服务

[root@master-2 rabbitmq]# cd /var/lib/rabbitmq/[root@master-2 rabbitmq]# lltotal 4020-rw-r----- 1 rabbitmq rabbitmq 4114398 May 14 11:15 erl_crash.dumpdrwxr-x--- 4 rabbitmq rabbitmq      94 May 14 11:38 mnesia[root@master-2 rabbitmq]# mv mnesia mnesia.bak[root@master-2 rabbitmq]# lltotal 4020-rw-r----- 1 rabbitmq rabbitmq 4114398 May 14 11:15 erl_crash.dumpdrwxr-x--- 4 rabbitmq rabbitmq      94 May 14 11:38 mnesia.bak

然后重新启动服务成功：

[root@master-2 rabbitmq]# service rabbitmq-server start    Starting rabbitmq-server (via systemctl):                  [  OK  ][root@master-2 rabbitmq]# ps -ef|grep rabbitmqrabbitmq  3131     1  0 May13 ?        00:00:00 /usr/lib64/erlang/erts-5.10.4/bin/epmd -daemonroot     19908     1  0 11:41 ?        00:00:00 /bin/sh /etc/rc.d/init.d/rabbitmq-server startroot     19910 19908  0 11:41 ?        00:00:00 /bin/bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /usr/sbin/rabbitmq-serverroot     19914 19910  0 11:41 ?        00:00:00 /bin/sh /usr/sbin/rabbitmq-serverroot     19932 19914  0 11:41 ?        00:00:00 su rabbitmq -s /bin/sh -c /usr/lib/rabbitmq/bin/rabbitmq-server rabbitmq 19935 19932  0 11:41 ?        00:00:00 /bin/sh /usr/lib/rabbitmq/bin/rabbitmq-serverrabbitmq 20158 19935 17 11:41 ?        00:00:04 /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -B i -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.12/ebin -noshell -noinput -s rabbit boot -sname rabbit@master-2 -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@master-2.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@master-2-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.12/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@master-2-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@master-2" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672rabbitmq 20316 20158  0 11:41 ?        00:00:00 inet_gethost 4rabbitmq 20317 20316  0 11:41 ?        00:00:00 inet_gethost 4root     20406 16497  0 11:42 pts/5    00:00:00 grep --color=auto rabbitmq[root@master-2 rabbitmq]#

注意，这只是主节点的处理方法，在两台从节点，需要做如下操作：
1、检查两台从节点的/var/lib/rabbitmq/.erlang.cookie文件内容是否和主节点是保持一致的
2、删除原有的数据/var/lib/rabbitmq/mnesia目录，执行rabbitmq-server -detached重新启动服务
3、在两台从节点上执行下面的命令，重新加入集群：

#rabbitmqctl stop_app #rabbitmqctl reset #rabbitmqctl join_cluster rabbit@master-2   # rabbit@master-2里面的master-2是主节点的主机名，注意修改#rabbitmqctl start_app

备节点执行完毕上面的步骤之后，需要在主节点验证集群的正确性：

[root@master-2 rabbitmq]# rabbitmqctl cluster_statusCluster status of node 'rabbit@master-2'[{nodes,[{disc,['rabbit@master-2','rabbit@s1-1','rabbit@slave-2']}]}, {running_nodes,['rabbit@s1-1','rabbit@slave-2','rabbit@master-2']}, {cluster_name,<<"rabbit@master-2">>}, {partitions,[]}, {alarms,[{'rabbit@s1-1',[]},          {'rabbit@slave-2',[nodedown]},          {'rabbit@master-2',[]}]}]

集群验证成功之后，使用主节点IP+端口登录，界面发现输入之前的用户名和密码，已经登录不进去了，需要在主节点重新创建管理用户并且赋予密码和访问权限

[root@master-2 rabbitmq]# rabbitmqctl add_user admin password123   #创建用户和密码Creating user "admin"[root@master-2 rabbitmq]# rabbitmqctl set_user_tags admin administrator   #给用户赋予管理员权限  Setting tags for user "admin" to [administrator][root@master-2 rabbitmq]#  rabbitmqctl  set_permissions -p "/" admin ".*" ".*" ".*"  #给管理员赋予访问权限  Setting permissions for user "admin" in vhost "/"

然后再使用主节点的IP+端口重新登录，输入用户名和密码，服务恢复。