1. Related introduction
1. rsync (remote synchronize)
rsync is a remote data synchronization tool under Liunx/Unix. It can quickly synchronize files and directories between multiple hosts through LAN/WAN.
There are generally two ways to synchronize files between Linux, namely rsync and scp. scp is equivalent to copying and pasting. If the file does not exist, it will be created. If it exists, it will be overwritten. rsync will compare the files on both sides to see if they are the same. If they are not the same, they will be updated. . Therefore, there is a big difference between rsync and scp when the folder exists, because scp copies and overwrites, and rsync is superior in terms of execution performance. Moreover, rsync can also save information such as folder and file permissions.
However, rsync also has certain shortcomings. When synchronizing data, all files need to be scanned and compared. If the number of files is quite large, scanning files is very time-consuming and performance-consuming. Secondly, rsync cannot monitor and synchronize data in real time, which may lead to data inconsistency in some time periods. The solution to this problem is real-time synchronization, so you need to use the rsync + inotify combination.
2. inotify
inotify is a powerful, fine-grained, asynchronous file system event monitoring mechanism. The Linux kernel has added support for inotify starting from version 2.6.13. Through inotify, you can monitor various events such as addition, deletion, modification, and movement in the file system. Using this kernel interface, inotify-tools can monitor various changes in files under the file system.
2. Preparation
1. Close selinux and firewall (or open ports in the firewall)
[root@localhost ~]# setenforce 0
[root@localhost ~]# vi /etc/sysconfig/selinux
2. Check whether the kernel supports inotify
[root@localhost ~]# ll /proc/sys/fs/inotify
If the following three files appear, it means that the system supports inotify by default
3. Overall architecture diagram
3. Synchronous node deployment (rsync)
Note: Generally, the rsync service is pre-installed on the system. If it is not installed, you can directly use yum install -y rsync
1. Write configuration file
[root@localhost ~]# vi /etc/rsyncd.conf
Add the following:
uid = nobody gid = nobody use chroot = yes max connections = 10 strict mode=yes pid file = /var/run/rsyncd.pid lock file=/var/run/rsync.lock log file=/var/log/rsyncd.log [backup] #Shared module name, customizable path = /usr/local/ #Synchronize data to the specified directory comment = this is rsync for nginx #Shared module remarks, customized according to needs ignore errors read only=no write only=no hosts allow=192.168.7.98 #Source server node IP hosts deny=* list=false uid=root gid=root auth users=ffs #Username for mutual authentication between servers, can be customized, but must be consistent with the source server secrets file=/etc/rsync.password #Password file used for authentication between servers
[root@localhost ~]# vi /etc/rsync.password
ffs:123456
[root@localhost ~]# chmod 600 /etc/rsync.password
Note: The permissions of the password file must be set to 600, otherwise the authentication will fail.
2. Start the service
[root@localhost ~]# rsync –daemon –config=/etc/rsyncd.conf
Note:
(1) System self-starting files can be added
[root@localhost ~]# echo “/usr/bin/rsync –daemon” >> /etc/rc.local
[root@localhost ~]# chmod 755 /etc/rc.local
(2) If the firewall is turned on, the corresponding port needs to be opened
[root@localhost ~]# firewall-cmd –add-port=873/tcp –permanent
[root@localhost ~]# firewall-cmd –reload
3. Check whether the rsync service is started
4. Source server node deployment (rsync + inotify)
1. Configure rsync configuration file
[root@localhost ~]# vi /etc/rsyncd.conf
uid = nobody gid = nobody use chroot = yes max connections = 10 strict mode=yes pid file = /var/run/rsyncd.pid lock file=/var/run/rsync.lock log file=/var/log/rsyncd.log [backup] #Shared module name, customizable path = /usr/local/ #The path is only written to the upper level of the folder that needs to be synchronized, and the script will automatically synchronize the entire directory. comment = this is rsync for nginx #Shared module remarks, customized according to needs ignore errors read only=no write only=no hosts allow=192.168.7.97 #Synchronization node IP hosts deny=* list=false uid=root gid=root auth users=ffs #The username is the same as the source server secrets file=/etc/rsync.password #The password is the same as the source server. You only need to write the password in this file, no user name is required.
[root@localhost ~]# vi /etc/rsync.password
123456
[root@localhost ~]# chmod 600 /etc/rsync.password
2. Start the service
[root@localhost ~]# rsync –daemon –config=/etc/rsyncd.conf
Note:
(1) System self-starting files can be added
[root@localhost ~]# echo “/usr/bin/rsync –daemon” >> /etc/rc.local
[root@localhost ~]# chmod 755 /etc/rc.local
(2) If the firewall is turned on, the corresponding port needs to be opened
[root@localhost ~]# firewall-cmd –add-port=873/tcp –permanent
[root@localhost ~]# firewall-cmd –reload
3. Install inotify
[root@localhost ~]# yum install -y inotify-tools
Note: inotify depends on the epel source. If there is no yum source, you need to perform the yum -y install epel-relese operation.
5. Create a shell script for rsync synchronization on the source server
[root@localhost work]# cat inotify.sh
#!/bin/bash
host1=192.168.7.97 #Synchronization node IP
src=/usr/local/ffmpeg_flie #The source server needs to be synchronized to the data directory of the synchronization node
dst1=backup #The shared module name defined in the rsync configuration file
user1=ffs #User name defined in rsync configuration file
/usr/bin/inotifywait -mrq --timefmt '%d/%m/%y %H:%M' --format '%T %w%f%e' -e close_write,delete,create ,attrib $src | while read files
do /usr/bin/rsync -vzrtopg --delete --progress --password-file=/etc/rsync.password $src $user1@$host1::$dst1 > /dev/null 2> & amp;1
echo "${files} was rsynced." >> /tmp/rsync.log 2> & amp;1
done
[root@localhost work]# chmod 755 inotify.sh
Run the script in the background
[root@localhost work]# nohup inotify.sh & amp;
Note: If necessary, you can add it to the system self-starting file.
[root@localhost work]# echo “/root/work/inotify.sh & amp;” >> /etc/rc.local
After the above steps are completed, you can test whether the real-time backup is successful. Create the test.txt file in the /usr/local/ffmpeg_flie/ directory of 192.168.7.98, and then check whether it is successful in the /usr/local/ffmpeg_flie/ directory of 192.168.7.87. It has been synchronized (the same applies to deleting files).