Rsync remote synchronization & inotify monitoring

Introduction to Rsync

rsync (Remote Sync) is an open source fast backup tool that can mirror and synchronize the entire directory tree between different hosts, support incremental backup, and maintain links and permissions.

In a remote synchronization task,the client responsible for initiating rsync synchronization operations is called the initiator, and the server responsible for responding to rsync synchronization operations from the client is called the synchronization source. During the synchronization process, the synchronization source is responsible for providing the original location of the file, and the initiating end should have read permissions to this location.

Rsync is a data mirroring backup tool under Linux systems. The fast incremental backup tool Remote Sync can be used to synchronize remotely and synchronize between different hosts. It can realize full backup and incremental backup, maintain links and permissions, and adopt optimized Synchronization algorithm, compression is performed before transmission, so it is very suitable for applications such as centralized backup or off-site backup. At the same time, Rsync supports local replication or synchronization with other SSH and Rsync hosts.

Rsync backup method

1. Initiator: The client responsible for rsync synchronization operation is called the initiator. It notifies the server that I want to back up your data.
2. Backup source: The server responsible for responding to rsync synchronization operations from the client is called the backup source, and the server that needs to be backed up
3. Server: Run the rsyncd service. Generally speaking, the server needs to be backed up.
4. Client: store backup data

Rsync synchronization method

1. Full backup: Each backup backs up all files or directories from the backup source to the destination.
2. Differential backup: Backs up data that has changed since the last full backup (he targets the last full backup, and does not clear the archive attributes during the backup process)
3. Incremental backup: Back up data that has changed since the last backup (it doesn’t care what type of backup it is, just back up the changed data, and it will clear the archive attributes)

Rsync synchronization source

rsync synchronization source: refers to the remote server for backup operations, also called backup source.

For example: Server A synchronizes the data of Server B, and Server B is the backup source – downlink synchronization

The opposite is – upstream synchronization

Rsync common command options

< /table>

Two expression methods of configuration source

Format 1

Username@host address::shared module name
 
rsync -avz [email protected]::wwwroot /opt/

Format 2

rsync://username@host address/shared module name
 
rsync -avz rsync://[email protected]/wwwroot /opt/

Rsync Experiment

Local copy

Configuring server and client

Install Rsync

systemctl stop firewall
systemctl disbale firewalld
setenforce 0
yum -y install rsync

Create the /etc/rsyncd.conf configuration file

uid = root
gid=root
use chroot = yes #imprisoned in the source directory
address = 192.168.10.18 #Listening address
port 873
#Listening port tcp/udp 873, can be viewed through cat /etc/services | grep rsync
 
log file = /var/log/rsyncd.log #Log file location
pid file = /var/run/rsyncd.pid #The file location where the process ID is stored
hosts allow = 192.168.10.0/24 #Client address allowed to be accessed
dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z
#File types that are no longer compressed during synchronization
 
[wwwroot] #Shared module name
path = /var/www/html #The actual path of the source directory
comment = Document Root of www.kgc.com
read only = yes #Whether it is read-only
auth users = backuper #Authorized accounts, multiple accounts separated by spaces
secrets file = /etc/rsyncd_users.db #Data file to store account information

Create data files for backup accounts

Ensure that all users have read permissions to the source directory /var/www/html

Normal synchronization completed

Password-free synchronization

Add a specified password file in the middle for password-free login
rsync -avz --password-file=/etc/server.passbackupera192.168,154.10: :wwwroot /opt

Usage of –delete option

rsync -avz -delete --password-file=/etc/server.pass [email protected]::wwwroot /abc

Introduction to inotify

Can monitor file system changes and respond to notifications

vim /etc/sysctl.conf (kernel parameter file)
max_queue_events #Monitor event queue size
max_user_instances #Maximum number of monitoring instances
max_user_watches #Maximum number of monitored files per instance
 
#Command tool
inotifywait #for continuous monitoring and real-time output results
inotifywatch #Used for short-term monitoring, the results will be output after the task is completed

Common command options

Options	Function
-a	Regulations Mode, recursive and retaining object properties, equivalent to -rlptgoD
-v	Display synchronization process information
-z	Compress during transmission
-H	Keep hard link files
-A	Retain ACL attribute information
–delete	Delete files that exist in the target location but not in the synchronization source
–checksum	Determine whether to skip the file based on the checksum of the object

Options	Function
-m	Continuous Monitoring
-r	Recursive monitoring All sub-objects
-q	Simplify output information
-e	Specify which event types to monitor
modify	Modify
create	Create
move	Move
delete	Delete

rsync + inotify implementation configuration

Server side configuration

Modify configuration file

vim /etc/rsyncd.conf
read only=no
 
kill (cat /var/run/rsyncd.pid)
rsync --daemon
netstat -natp |grep rsync

Modify the kernel file (an optimization operation)

Client configuration

Modify configuration file

Write a trigger synchronization script

vim /opt/inotify.sh
 
#!/bin/bash
INOTIFY_CMD="inotifywait -mrq -e create,delete,move,modify,attrib /abc"
RSYNC_CMD="rsync -azH --delete --password-file=/etc/server.pass /abc/ [email protected]::wwwroot"
 
$INOTIFY_CMD | while read DIRECTORY EVENT FILE
do
    if [ $(pgrep rsync | wc -l) -le 0 ] ; then
        $RSYNC_CMD
fi
done
 
#Script explanation
First define a variable to monitor changes in information such as creation, deletion, movement, modification, attributes, etc. in this directory.
Define another variable for rsync synchronization to synchronize the information in the directory in /abc to the specified directory in the rsync server
Write another while loop, first execute the monitoring variable, and then execute the while loop.
The content of the loop is to read the directory event file, and if the rsync process synchronization operation does not exist, execute the variables for the synchronization operation.
 

chmod + x /opt/inotify.sh
./inotify.sh

---------------#Join self-start-----------------------
chmod +x /etc/rc.d/rc.local
echo "/opt/inotify.sh" >>/etc/rc.d/rc.local

Client-side verification

4. Summary

Advantages

• Can be real-time

Disadvantages

• When bandwidth is particularly abundant, the synchronization rate of rsync is slower than scp.

• To synchronize millions of data, parameters must be written strictly as required, otherwise synchronization may fail.