linux · 2021年7月21日 0

Centos keepalive+DRBD 实例


简单讲解实现两台服务器硬盘数据块同步 防灾 高可用集群
仅供学习与交流,请勿用于商业用途

一,安装系统并设置静态IP以及主机名

设备相关信息

node1    192.168.4.110/24    Centos7 node1.example.com
node2    192.168.4.120/24    Centos7 node2.example.com
virtualip    192.168.4.130/24    

供用户使用,由keepalived实现
服务器分区要求
因为DRBD是块设备同步
所以单独创建分区挂载/var/svn
建议LVM分区,方便管理与扩容

NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   50G  0 disk 
├─sda1            8:1    0    1G  0 part /boot
└─sda2            8:2    0   49G  0 part 
 ├─centos-root 253:0    0 45.1G  0 lvm  /
 └─centos-swap 253:1    0  3.9G  0 lvm  [SWAP]
 └─centos-svn  253:2    0  3.8T  0 lvm  /var/svn

服务器双向免验证SSH
按需求设置,设置方法省略一万字
不会配置基本信息请关闭本文

二,部署DRBD实现数据同步

1.安装elrepo认证密匙,安装elrepo源,安装DRBP

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
yum-config-manager --add-repo=https://mirrors.tuna.tsinghua.edu.cn/epel/7/x86_64/
yum install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm
yum install -y kmod-drbd84.x86_64 kmod-devel.x86_64 drbd84-utils.x86_64

2.安装开发工具

yum groupinstall "Development Tools" -y

3.修改全局文件

vim /etc/drbd.d/global_common.conf
global {
        usage-count no;
        udev-always-use-vnr; # treat implicit the same as explicit volumes
}
common {
        handlers {
                 pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
                 pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
                local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
        }
        startup {
        }
        options {
        }
        disk {     
                ##on-io-error detach;   
                ##配置I/O错误处理策略为分离
        }
        net {
            protocol C;
        }
        syncer {
        rate 1024M;    #设置主备节点同步时的网络速率,
    }
}

on-io-error 策略可能为以下选项之一

detach 分离:这是默认和推荐的选项,如果在节点上发生底层的硬盘I/O错误,它会将设备运行在Diskless无盘模式下

pass_on:DRBD会将I/O错误报告到上层,在主节点上,它会将其报告给挂载的文件系统,但是在此节点上就往往忽略(因此此节点上没有可以报告的上层)

local-in-error:调用本地磁盘I/O处理程序定义的命令;这需要有相应的local-io-error调用的资源处理程序处理错误的命令;这就给管理员有足够自由的权力命令命令或是脚本调用local-io-error处理I/O错误

  1. 创建配置文件
vim /etc/drbd.d/svn0.res
resource r0 {
  device    /dev/drbd1;
  disk      /dev/centos/svn;
  meta-disk internal;
  on node1.example.com {
    address   192.168.4.110:7789;
  }
  on node2.example.com {
    address   192.168.4.120:7789;
  }
}

5.两台主机的防火墙放行自定义端口
使用富规则 保证安全性

firewall-cmd --permanent --add-rich-rule="rule family=ipv4 source address=192.168.4.120 port port=7789 protocol=tcp accept"
firewall-cmd --reload

6.初始化块设备 并上线

drbdadm create-md r0
drbdadm up r0
drbdadm primary --force r0

过程比较久,需要一定时间,因硬盘速度而决定
查看块设备初始化状态

cat /proc/drbd 

7.启动服务
两台设备同时启动服务

systemctl enable drbd
systemctl start drbd

8.创建文件系统
使用lsblk命令可以查看到一块新磁盘上线
格式化这块新磁盘,并挂载到业务目录

mkfs.xfs /dev/drbd1
mount /dev/drbd1 /var/svn

三,服务器部署SVN

1.两台服务器安装SVN

yum install subversion -y

2.在node1简单配置SVN
修改默认目录的selinux

yum install libsemanage-static.x86_64 libsemanage-devel.x86_64 -y
semanage fcontext -a -t svnserve_content_t '/var/svn(/.*)?'
restorecon -Rv '/var/svn/'

在SVN默认目录创建测试文件夹

mkdir -p /var/svn/svntest
svnadmin create /var/svn/svntest/

修改该目录SVN配置
添加用户

cd /var/svn/svntest/
vim conf/passwd
[users]
admin = admin
long =long

修改用户权限

vim conf/authz
[/]
admin = rw
long =r

3.两台设备启动服务,防火墙放行,可按需求编写富规则

systemctl enable svnserve.service 
systemctl start svnserve.service 
firewall-cmd --permanent --add-service=svn 
firewall-cmd --reload 

四,两台机器部署keepalive并编写监控脚本

1,两台机器安装keepalived服务

yum install keepalived -y

2,编辑keepalived全局配置
主机:

! Configuration File for keepalived
global_defs {
   notification_email {
        root@localhost
   }
   script_user root
   enable_script_security
   notification_email_from root@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id DRBD_HA_MASTER
}

vrrp_script chk_drbd {
script "/etc/keepalived/chk_drbd.sh"
interval 1
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 7789
    }
    track_script {
        chk_drbd
        }
    notify_master /etc/keepalived/notify_master.sh
    notify_stop   /etc/keepalived/notify_stop.sh
    virtual_ipaddress {
        192.168.4.130
    }
}

备机:

! Configuration File for keepalived

global_defs {
   notification_email {
        root@localhost
   }
   notification_email_from root@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id DRBD_HA_BACKUP
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 7789
    }
    notify_master /etc/keepalived/notify_master.sh
    notify_backup /etc/keepalived/notify_backup.sh
    virtual_ipaddress {
        192.168.4.130/24
    }
}

3.编写四个脚本,供keepalived启动 故障 主备切换使用
主机监测svn服务故障转移脚本

vim chk_drbd.sh
#!/bin/bash
# power by guanjianlong
# Communcate with me moulongxiong0831@163.com
# the script for cheak drbd service
systemctl is-active svnserve.service &>/dev/null
if [ $? -ne 0 ];then
        systemctl restart svnserve.service
        systemctl status svnserve.service &>/dev/null
        if [ $? -ne 0 ];then
                umount /dev/drbd1
                #degrade drbd
                drbdadm secondary r0
                systemctl stop keepalived
                #change vip
        fi
fi

当机器升级为master的时候执行以下脚本,两台机器都要有的脚本

vim notify_master.sh
#!/bin/bash  
time=`date "+%F  %H:%M:%S"`
echo -e "$time    ------notify_master------\n" >> /etc/keepalived/logs/notify_master.log
drbdadm primary r0 &>> /etc/keepalived/logs/notify_master.log
mount /dev/drbd1 /var/svn &>> /etc/keepalived/logs/notify_master.log
systemctl restart svnserve &>> /etc/keepalived/logs/notify_master.log
echo -e "\n" >> /etc/keepalived/logs/notify_master.log

当主机的vip停止时需要运行的脚本

vim notify_stop.sh
#!/bin/bash
time=`date "+%F  %H:%M:%S"`
echo -e "$time  ------notify_stop------\n" >> /etc/keepalived/logs/notify_stop.log
systemctl stop svnserve.service &>> /etc/keepalived/logs/notify_stop.log
umount /var/svn &>> /etc/keepalived/logs/notify_stop.log
drbdadm secondary r0 &>> /etc/keepalived/logs/notify_stop.log
echo -e "\n" >> /etc/keepalived/logs/notify_stop.log

当备机降级到backup的脚本

#!/bin/bash
time=`date "+%F  %H:%M:%S"`
echo -e "$time    ------notify_backup------\n" >> /etc/keepalived/logs/notify_backup.log
systemctl stop svnserve.service &>> /etc/keepalived/logs/notify_backup.log
umount /dev/drbd1 &>> /etc/keepalived/logs/notify_backup.log
drbdadm secondary r0 &>> /etc/keepalived/logs/notify_backup.log
echo -e "\n" >> /etc/keepalived/logs/notify_backup.log

4.修改脚本文件的上下文

semanage fcontext -a -t keepalived_unconfined_script_exec_t /path/script
restorecon -Rv /path/script

5.启动keepalived

systemctl enable keepalived
systemctl start keepalived

五,检查主机服务状态以及测试故障迁移

1.检查服务

[root@node1 ~]# systemctl is-active svnserve.service keepalived.service drbd.service 
active
active
active

2.检查挂载

[root@node1 ~]# lsblk 
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   50G  0 disk 
├─sda1            8:1    0    1G  0 part /boot
└─sda2            8:2    0   49G  0 part 
  ├─centos-root 253:0    0 45.1G  0 lvm  /
  └─centos-swap 253:1    0  3.9G  0 lvm  [SWAP]
sdb               8:16   0   20G  0 disk 
└─sdb1            8:17   0   20G  0 part 
  └─drbd1       147:1    0   20G  0 disk /var/svn

3.linux端服务测试

[root@node2 ~]# svn list svn://192.168.4.130/repos/project
branches/
tags/
trunk/

4.windows端SVN测试

六.模拟故障试验

场景1:
主机svn服务故障,自动重启

[root@node1 keepalived]# systemctl stop svnserve.service ;  sleep 5 ; systemctl status svnserve.service 
● svnserve.service - Subversion protocol daemon
   Loaded: loaded (/usr/lib/systemd/system/svnserve.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-27 22:51:18 CST; 4s ago
  Process: 55623 ExecStart=/usr/bin/svnserve --daemon --pid-file=/run/svnserve/svnserve.pid $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 55624 (svnserve)
   CGroup: /system.slice/svnserve.service
           └─55624 /usr/bin/svnserve --daemon --pid-file=/run/svnserve/svnserve.pid -r /var/svn

Oct 27 22:51:18 node1.example.com systemd[1]: Starting Subversion protocol daemon...
Oct 27 22:51:18 node1.example.com systemd[1]: Started Subversion protocol daemon.

场景2:
屏蔽SVN,模拟服务暴毙,自动转移到备用机

[root@node1 keepalived]# systemctl mask svnserve.service
[root@node1 keepalived]# systemctl stop svnserve.service
[root@node1 keepalived]# systemctl status svnserve.service 
● svnserve.service
   Loaded: masked (/dev/null; bad)
   Active: inactive (dead) since Sun 2019-10-27 22:52:38 CST; 26s ago
 Main PID: 55624 (code=killed, signal=TERM)

Oct 27 22:51:18 node1.example.com systemd[1]: Starting Subversion protocol daemon...
Oct 27 22:51:18 node1.example.com systemd[1]: Started Subversion protocol daemon.
Oct 27 22:52:38 node1.example.com systemd[1]: Stopping svnserve.service...
Oct 27 22:52:38 node1.example.com systemd[1]: Stopped svnserve.service.

查看备机

[root@node2 keepalived]# lsblk 
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   50G  0 disk 
├─sda1            8:1    0    1G  0 part /boot
└─sda2            8:2    0   49G  0 part 
  ├─centos-root 253:0    0 45.1G  0 lvm  /
  └─centos-swap 253:1    0  3.9G  0 lvm  [SWAP]
sdb               8:16   0   20G  0 disk 
└─sdb1            8:17   0   20G  0 part 
  └─drbd1       147:1    0   20G  0 disk /var/svn
sr0              11:0    1  4.4G  0 rom

场景3:
主机暴力断电

[root@node1 ~]# poweroff -f

检查备用机

[root@node2 ~]# lsblk 
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   50G  0 disk 
├─sda1            8:1    0    1G  0 part /boot
└─sda2            8:2    0   49G  0 part 
  ├─centos-root 253:0    0 45.1G  0 lvm  /
  └─centos-swap 253:1    0  3.9G  0 lvm  [SWAP]
sdb               8:16   0   20G  0 disk 
└─sdb1            8:17   0   20G  0 part 
  └─drbd1       147:1    0   20G  0 disk /var/svn
sr0              11:0    1  4.4G  0 rom  
[root@node2 ~]# systemctl status svnserve.service 
● svnserve.service - Subversion protocol daemon
   Loaded: loaded (/usr/lib/systemd/system/svnserve.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-27 22:58:29 CST; 18s ago
  Process: 30638 ExecStart=/usr/bin/svnserve --daemon --pid-file=/run/svnserve/svnserve.pid $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 30639 (svnserve)
   CGroup: /system.slice/svnserve.service
           └─30639 /usr/bin/svnserve --daemon --pid-file=/run/svnserve/svnserve.pid -r /var/svn

Oct 27 22:58:29 node2.example.com systemd[1]: Starting Subversion protocol daemon...
Oct 27 22:58:29 node2.example.com systemd[1]: Started Subversion protocol daemon.