高可用集群keepalived的应用以及部署

服务器 0

1.高可用集群

1.1.集群类型

  • LB:Load Balance 负载均衡

    LVS/haproxy/nginx(http/upstream,stream/upstream)

  • HA : High Point of Failure 高可用集群

    数据库、Redis

  • SPoF:single point Of failure 解决单点故障

    HPC:High performmance computing 高性能集群

1.2.系统可用性

SLA:Service-Level Agreement 服务等级协议(提供服务的企业与客户之间就服务的品质、水准、性能 等方面所达成的双方共同认可的协议或契约)

A = MTBF / (MTBF+MTTR)

99.95%:(60*24*30)*(1-0.9995)=21.6分钟 #一般按一个月停机时间统计

 1.3.系统故障

硬件故障:设计缺陷、wear out(损耗)、非人为不可抗拒因素

软件故障:设计缺陷 bug

1.4.实现高可用

提升系统高用性的解决方案:降低MTTR- Mean Time To Repair(平均故障时间) 解决方案:建立冗余机制

  • active/passive 主/备

  • active/active 双主

  • active --> HEARTBEAT --> passive

  • active <--> HEARTBEAT <--> active

1.5.VRRP:Virtual Router Redundancy Protocol

虚拟路由冗余协议,解决静态网关单点风险

  • 物理层:路由器、三层交换机

  • 软件层:keepalived

1.5.1VRRP相关术语

  • 虚拟路由器:Virtual Router

  • 虚拟路由器标识:VRID(0-255),唯一标识虚拟路由器

  • VIP:Virtual IP

  • VMAC:Virutal MAC (00-00-5e-00-01-VRID)

  • 物理路由器:

    • master:主设备

    • backup:备用设备

    • priority:优先级

1.5.2VRRP相关技术

通告:心跳,优先级等;周期性 工作方式:抢占式,非抢占式 安全认证:

  • 无认证

  • 简单字符认证:预共享密钥

  • MD5

工作模式

  • 主/备:单虚拟路由器

  • 主/主:主/备(虚拟路由器1)、备/注(虚拟路由器2),他们互为主备

2.keepalived部署

2.1keepalived简介及功能

简介

Keepalived软件起初是专为LVS负载均衡软件设计的,用来管理并监控LVS集群系统中各个服务节点的状态,后来又加入了可以实现高可用的VRRP功能。因此,Keepalived除了能够管理LVS软件外,还可以作为其他服务(例如:Nginx、Haproxy、MySQL等)的高可用解决方案软件。

功能:

基于vrrp协议完成地址流动

为vip地址所在的节点生成ipvs规则(在配置文件中预先定义)

为ipvs集群的各RS做健康状态检测

基于脚本调用接口完成脚本中定义的功能,进而影响集群事务,以此支持nginx、haproxy等服务

2.2keepalived工作原理

keepalived是以VRRP协议为实现基础的,VRRP全称Virtual Router Redundancy Protocol,即虚拟路由冗余协议。
虚拟路由冗余协议,可以认为是实现路由器高可用的协议,即将N台提供相同功能的路由器组成一个路由器组,这个组里面有一个master和多个backup,master上面有一个对外提供服务的vip(该路由器所在局域网内其他机器的默认路由为该vip),master会发组播,当backup收不到vrrp包时就认为master宕掉了,这时就需要根据VRRP的优先级来选举一个backup当master。这样的话就可以保证路由器的高可用了。

2.3keepalived架构

用户空间核心组件:
        vrrp stack:VIP消息通告
        checkers:监测real server
        system call:实现 vrrp 协议状态转换时调用脚本的功能
        SMTP:邮件组件
        IPVS wrapper:生成IPVS规则
        Netlink Reflector:网络接口
        WatchDog:监控进程
控制组件:提供keepalived.conf 的解析器,完成Keepalived配置
IO复用器:针对网络目的而优化的自己的线程抽象
内存管理组件:为某些通用的内存管理功能(例如分配,重新分配,发布等)提供访问权限

2.4keepalived相关文件

  • 软件包名:keepalived

  • 主程序文件:/usr/sbin/keepalived

  • 主配置件:/etc/keepalived/keepalived.conf

  • 配置文件示例:/usr/share/doc/keepalived/

  • UnitFile:/lib/systemd/system/keepalived.service

  • Unit File的环境配置文件:/etc/sysconfig/keepalived

2.6环境

我们基于rhel7

克隆四台rhel7的虚机,分别是

realserver1:172.252.50.110

realserver2:172.25.250.120

KA1:172.252.50.10

KA2:172.25.250.20

VIP:172.25.250.100

关闭防火墙和SElinux

realserver1、realserver2主机都下载Apache[root@realserver1 ~]# yum install httpd -y[root@realserver1 ~]# systemctl enable --now httpd[root@realserver1 ~]# echo realserver1 - 172.25.250.110 > /var/www/html/index.html[root@realserver2 ~]# yum install httpd -y[root@realserver2 ~]# systemctl enable --now httpd[root@realserver2 ~]# echo realserver2 - 172.25.250.120 > /var/www/html/index.html

2.7keepalived虚拟路由管理

2.7.1 ka1主机

###KA1和KA2主机下载keepalived[root@ka1 ~]# yum install keepalived -y[root@ka2 ~]# yum install keepalived -y[root@ka1 ~]# rpm -ql | grep keepalived    //查看文件[root@ka1 ~]# vim /etc/keepalived/keepalived.conf     //keepalived主配置文件

[root@ka1 ~]# systemctl enable --now keepalived.service

[root@ka1 ~]# ifconfig

2.7.2ka2主机

[root@ka1 ~]# scp /etc/keepalived/keepalived.conf root@172.25.250.20:/etc/keepalived/keepalived.conf   //将KA1里面/etc/keepalived/keepalived.conf 文件内容复制到KA2的/etc/keepalived/keepalived.conf##然后我们去KA2主机打开/etc/keepalived/keepalived.conf

[root@ka2 ~]# systemctl enable --now keepalived.service[root@ka2 ~]# ifconfig

然后我们关闭KA1主机的keepalived,再去KA2主机查看

[root@ka1 ~]# systemctl stop keepalived.service

2.8 虚拟路由的通信设定

最开始ka1主机或ka2主机是ping不通VIP的

因为默认情况会把VIP访问功能禁掉

iptables -nL

如果想要实现通信

[root@ka1 ~]# vim /etc/keepalived/keepalived.conf

2.9日志分离

[root@ka1 ~]# vim /etc/sysconfig/keepalived

[root@ka1 ~]# vim /etc/rsyslog.conf

[root@ka1 ~]# systemctl restart keepalived.service [root@ka1 ~]# systemctl restart rsyslog.service [root@ka1 ~]# ll /var/log/keepalived.log -rw-------. 1 root root 2121218 Aug 12 20:56 /var/log/keepalived.log

2.10独立子配置文件

当生产环境复杂时, /etc/keepalived/keepalived.conf 文件中内容过多,不易管理

将不同集群的配置,比如:不同集群的VIP配置放在独立的子配置文件中利用include 指令可以实现包含 子配置文件

[root@ka1 ~]# vim /etc/keepalived/keepalived.conf

然后创建目录

3.keepalived企业应用示例

3.1抢占模式和非抢占模式

3.1.1非抢占模式 nopreempt

默认为抢占模式preempt,即当高优先级的主机恢复在线后,会抢占低先级的主机的master角色,

这样会使vip在KA主机中来回漂移,造成网络抖动,

建议设置为非抢占模式 nopreempt ,即高优先级主机恢复后,并不会抢占低优先级主机的master角色

非抢占模块下,如果原主机down机, VIP迁移至的新主机, 后续也发生down时,仍会将VIP迁移回原主机

注意:要关闭 VIP抢占,必须将各 keepalived 服务器state配置为BACKUP

ka1主机

[root@ka1 ~]# vim /etc/keepalived/keepalived.conf

ka2主机

[root@ka2 ~]# vim /etc/keepalived/keepalived.conf

然后测试结果就是当ka1关闭keepalived时,VIP就会到ka2主机,然后ka1再次开启keepalived时,VIP不会回到ka1主机,而是继续在ka2主机

3.1.2延时抢占模式

抢占延迟模式,即优先级高的主机恢复后,不会立即抢回VIP,而是延迟一段时间(默认300s)再抢回 VIP

preempt_delay #    #指定抢占延迟时间为#s,默认延迟300s

注意:需要各keepalived服务器state为BACKUP,并且不要启用 vrrp_strict

ka1主机配置vrrp_instance VI_1 {   state BACKUP   interface eth0   virtual_router_id 20   priority 100 #优先级高   preempt_delay 10s #抢占延迟10s   advert_int 1   authentication {       auth_type PASS       auth_pass 1111   }   virtual_ipaddress {       172.25.250.100/24 dev eth0 label eth0:1   }}   #KA2主机配置vrrp_instance VI_1 {   state BACKUP   interface eth0   virtual_router_id 20   priority 80 #优先级低   advert_int 1   preempt_delay 10s #抢占延迟10S   authentication {       auth_type PASS       auth_pass 1111   }   virtual_ipaddress {     172.25.250.100/24 dev eth0 label eth0:1   }}

然后我们做完延时抢占模式实验后建议把延时注释掉,不然会影响后面实验

3.2VIP单播配置

默认keepalived主机之间利用多播相互通告消息,会造成网络拥塞,可以替换成单播,减少网络流量

注意:启用 vrrp_strict 时,不能启用单播

#在所有节点vrrp_instance语句块中设置对方主机的IP,建议设置为专用于对应心跳线网络的地址,而非使 用业务网络

unicast_src_ip <IPADDR> #指定发送单播的源IP

unicast_peer {      

         <IPADDR>#指定接收单播的对方目标主机IP  

......

}

#启用 vrrp_strict 时,不能启用单播,否则服务无法启动,并在messages文件中记录下面信息

ka1主机

[root@ka1 ~]# vim /etc/keepalived/keepalived.conf

ka2主机

[root@ka2 ~]# vim /etc/keepalived/keepalived.conf

然后我们关掉ka1主机的keepalived

[root@ka1 ~]# systemctl stop keepalived.service

3.3通知脚本配置

ka1和ka2主机都下载mailx

yum install mailx -y

qq邮箱通知

[root@ka2 ~]# vim /etc/mail.rc

 写脚本

[root@ka1 ~]# vim /etc/keepalived/mail.sh #!/bin/bashmail_dst=".....@qq.com"   //你自己的qq邮箱send_message(){        mail_sub="$HOSTNAME to be $1 vip move"        mail_msg="`date +%F/ %T`: vrrp move $HOSTNAME chage $1 "        echo $mail_msg | mail -s "$mail_sub" $mail_dst}case $1 in  master)  send_message master  ;;  backup)  send_message backup  ;;  fault)  send_message fault  ;;  *)  ;;esac[root@ka1 ~]# chmod +x /etc/keepalived/mail.sh    //给脚本执行权限

[root@ka1 ~]# vim /etc/keepalived/keepalived.conf 

然后

/etc/keepalived/mail.sh fualt    //你就会收到邮件

[root@ka1 ~]# systemctl stop keepalived.service

[root@ka1 ~]# systemctl restart keepalived.service

你也会收到对应的邮件

ka2主机上一样的操作

3.4双主机

ka1主机

[root@ka1 ~]# vim /etc/keepalived/keepalived.conf vrrp_instance VI_1 {    state MASTER    interface eth0    virtual_router_id 100    priority 100    advert_int 1    #preempt_delay 5s    authentication {        auth_type PASS        auth_pass 1111    }    virtual_ipaddress {        172.25.250.100/24 dev eth0 label eth0:1    }        unicast_src_ip 172.25.250.10        unicast_peer {                172.25.250.20        }}vrrp_instance VI_2 {    state BACKUP    interface eth0    virtual_router_id 200    priority 80    advert_int 1    #preempt_delay 5s    authentication {        auth_type PASS        auth_pass 1111    }    virtual_ipaddress {        172.25.250.200/24 dev eth0 label eth0:2    }        unicast_src_ip 172.25.250.10        unicast_peer {                172.25.250.20        }}

ka2主机 

[root@ka2 ~]# vim /etc/keepalived/keepalived.conf vrrp_instance VI_1 {    state BACKUP    interface eth0    virtual_router_id 100    priority 80    advert_int 1   # preempt_delay 5s    authentication {        auth_type PASS        auth_pass 1111    }    virtual_ipaddress {        172.25.250.100/24 dev eth0 label eth0:1    }        unicast_src_ip 172.25.250.20        unicast_peer {                172.25.250.10        }}vrrp_instance VI_2 {    state MASTER    interface eth0    virtual_router_id 200    priority 100    advert_int 1   # preempt_delay 5s    authentication {        auth_type PASS        auth_pass 1111    }    virtual_ipaddress {        172.25.250.200/24 dev eth0 label eth0:2    }        unicast_src_ip 172.25.250.20        unicast_peer {                172.25.250.10        }}

测试

3.5实现ipvs的高可用性

3.5.1实现单主机lvs-DR模式

准备web服务器并使用脚本绑定VIP至web服务器lo网卡

##rs1主机和rs2主机[root@realserver1 ~]# yum install httpd -y[root@realserver2 ~]# yum install httpd -y[root@realserver1 ~]# echo realserver1 - 172.25.250.110 > /var/www/html/index.html[root@realserver2 ~]# echo realserver2 - 172.25.250.120 > /var/www/html/index.html#rs1主机[root@realserver1 ~]# ip a a 172.25.250.100/32 dev lo[root@realserver1 ~]#echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore [root@realserver1 ~]#echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore [root@realserver1 ~]#echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce[root@realserver1 ~]#echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce#rs2主机[root@realserver2 ~]# ip a a 172.25.250.100/32 dev lo[root@realserver2 ~]#echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore [root@realserver2 ~]#echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore [root@realserver2 ~]#echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce[root@realserver2 ~]#echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce

 配置keepalived

#ka1主机[root@ka1 ~]# vim /etc/keepalived/keepalived.conf virtual_server 172.25.250.100 80 {    delay_loop 6    lb_algo wrr    lb_kind DR    protocol TCP    real_server 172.25.250.110 80 {        weight 1        HTTP_GET {            url {              path /              status_code 200            }            connect_timeout 3            nb_get_retry 2            delay_before_retry 2        }    }     real_server 172.25.250.120 80 {        weight 1        HTTP_GET {            url {              path /              status_code 200            }            connect_timeout 3            nb_get_retry 2            delay_before_retry 2        }    }}#ka2主机[root@ka2 ~]# vim /etc/keepalived/keepalived.conf virtual_server 172.25.250.100 80 {    delay_loop 6    lb_algo wrr    lb_kind DR    protocol TCP    real_server 172.25.250.110 80 {        weight 1        HTTP_GET {            url {              path /              status_code 200            }            connect_timeout 3            nb_get_retry 2            delay_before_retry 2        }    }    real_server 172.25.250.120 80 {        weight 1        HTTP_GET {            url {              path /              status_code 200            }            connect_timeout 3            nb_get_retry 2            delay_before_retry 2        }    }}#测试结果[root@ka1 ~]# ipvsadm -LnIP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags  -> RemoteAddress:Port           Forward Weight ActiveConn InActConnTCP  172.25.250.100:80 wrr  -> 172.25.250.110:80            Route   1      0          0           -> 172.25.250.120:80            Route   1      0          0         TCP  10.10.10.2:1358 rr persistent 50  -> 192.168.200.200:1358         Masq    1      0          0         TCP  10.10.10.3:1358 rr persistent 50

 

 

 模拟rs1故障

[root@realserver1 ~]# systemctl stop httpd

 3.6实现其它应用的高可用性 VRRP script

keepalived利用 VRRP Script 技术,可以调用外部的辅助脚本进行资源监控,并根据监控的结果实现优先 动态调整,从而实现其它应用的高可用性功能

3.6.1VRRP script配置

分两步实现

vrrp_script:自定义资源监控脚本,vrrp实例根据脚本返回值,公共定义,可被多个实例调用,定 义在vrrp实例之外的独立配置块,一般放在global_defs设置块之后。

通常此脚本用于监控指定应用的状态。一旦发现应用的状态异常,则触发对MASTER节点的权重减至 低于SLAVE节点,从而实现 VIP 切换到 SLAVE 节点

3.6.2利用脚本实现主从角色切换

[root@ka1 ~]# cat /mnt/check_file.sh   //脚本随便写在那个路径,唯一要注意的就是在配置文件里面的路径要与这里一致#!/bin/bash[ ! -f "/mnt/file" ][root@ka1 ~]# chmod +x /mnt/check_file.sh   //给脚本执行权限###################[root@ka1 ~]# sh /etc/keepalived/test.sh [root@ka1 ~]# echo $?0                         //如果没有/mnt/file这个文件,那么返回0,不会做任何改变[root@ka1 ~]# touch /mnt/file[root@ka1 ~]# sh /etc/keepalived/test.sh [root@ka1 ~]# echo $?1                         //如果有/mnt/file这个文件,返回为1,那么就会触发降低权重#####################[root@ka1 ~]# vim /etc/keepalived/keepalived.conf vrrp_script check_file {   script "/mnt/check_lee.sh"   #此脚本返回值为非0时,会触发下面OPTIONS执行   interval 1   weight -30   fall 2   rise 2   timeout 2}vrrp_instance VI_1 {    state MASTER    interface eth0    virtual_router_id 100    priority 100    advert_int 1    #preempt_delay 5s    authentication {        auth_type PASS        auth_pass 1111    }    virtual_ipaddress {        172.25.250.100/24 dev eth0 label eth0:1    }    teack_script {        check_file        }        unicast_src_ip 172.25.250.10        unicast_peer {                172.25.250.20        }}

 

3.7.3实现haproxy与keepalived的高可用

#在ka1和ka2主机安装haproxy[root@ka1 ~]# yum install haproxy -y[root@ka2 ~]# yum install haproxy -y#在ka1和ka2主机实现haproxy的配置#ka1主机[root@ka1 ~]# vim /etc/haproxy/haproxy.cfg listen webserver    bind 172.25.250.100:80    server web1 172.25.250.110:80 check    server web2 172.25.250.120:80 check#ka2主机[root@ka2 ~]# vim /etc/haproxy/haproxy.cfg listen webserver    bind 172.25.250.100:80    server web1 172.25.250.110:80 check    server web2 172.25.250.120:80 check
#在两个ka1和ka2先实现haproxy的配置backend webcluster    bind 172.25.250.100:80    mode http    balance roundrobin    server realserver1 172.25.250.110:80 check inter 3 fall 2 rise 5    server realserver2 172.25.250.120:80 check inter 3 fall 2 rise 5

#在两个ka1和ka2两个节点启用内核参数[root@ka1 ~]# cat /etc/sysctl.conf # sysctl settings are defined through files in# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.## Vendors settings live in /usr/lib/sysctl.d/.# To override a whole file, create a new file with the same in# /etc/sysctl.d/ and put new settings there. To override# only specific settings, add a file with a lexically later# name in /etc/sysctl.d/ and put new settings there.## For more information, see sysctl.conf(5) and sysctl.d(5).net.ipv4.ip_nonlocal_bind = 1         //这条参数     #在ka1中编写检测脚本[root@ka1 ~]# vim /etc/keepalived/scripts/haproxy.sh#!/bin/bash/usr/bin/killall -0 haproxy[root@ka1 ~]# chmod +X /etc/keepalived/scripts/haproxy.sh#在ka1中配置keepalived[root@ka1 ~]# vim /etc/keepalived/keepalived.confvrrp_script check_haproxy {   script "/etc/keepalived/scripts/haproxy.sh"   interval 1   weight -30   fall 2   rise 2   timeout 2}vrrp_instance web {   state MASTER   interface eth0   virtual_router_id 100   priority 100   advert_int 1   authentication {       auth_type PASS       auth_pass 1111   }   virtual_ipaddress {       172.25.250.100 dev eth0 label eth0:1   }   track_script {       check_haproxy   }}

 测试

我们把ka1主机的haproxy关掉,可以看到VIP飘到ka2主机上

然后客户机一直访问172.25.250.100 ,会发现访问不会断

这就是haproxy和keepalived联用实现高可用,利用keepalived检测haproxy是否存活,如果存活则不会有任何操作,如果不存活则会把VIP飘到另外一台haproxy存活的主机上,然后客户端主机访问不受影响

也许您对下面的内容还感兴趣: