rsync backup tool (with rsync+inotify real-time synchronization deployment example)

rsync backup tool (with rsync + inotify real-time synchronization deployment example)

  • 1. Overview of rsync
    • 1.1 About rsync
    • 1.2 Characteristics of rsync
    • 1.3 Working principle
  • 2. rsync related commands
    • 2.1 Basic formats and common options
    • 2.2 Start and shut down the rsync service
    • 2.3 Basic format of downlink synchronization
    • 2.4 Basic format of uplink synchronization
    • 2.5 Interaction-free
      • 2.5.1 Specify password file
      • 2.5.2rsync-daemon method
      • 2.5.3rsync-ssh method
    • 2.6 Regular synchronization
  • 3. Deploy rsync for regular synchronization
    • 3.1 Configure rsync source server (node 1)
    • 3.2 Initiator (Node 2)
    • 3.3 Initiator configuration rsync + inotify
  • 4. Use rsync to quickly delete a large number of files

1. Overview of rsync

1.1 About rsync

Rsync (Remote sync) is a remote data synchronization tool, a fast incremental backup tool, used on multiple platforms such as unix/Linux/windows.

Rsync uses the so-called “Rsync algorithm” to synchronize files between the local and remote hosts. This algorithm only transmits different parts of the two files instead of transmitting the entire copy each time, so it is quite fast.

The machine running Rsync server is also called backup server. One Rsync server can back up the data of multiple clients at the same time; multiple Rsync servers can also back up the data of one client.

Rsync can be used with rsh or ssh or even in daemon mode.

The Rsync server will open a service channel (port) 873 and wait for the other party’s Rsync connection.
When connecting, the Rsync server will check whether the password matches. If the password passes the check, the file transfer can begin.

When the first connection is completed, the entire file will be transferred once, and the next time only the different parts of the two files will be transferred.

Official website: http://rsync.samba.org

Features of 1.2rsync

  • The entire directory tree and file system can be mirrored;
  • It is easy to maintain the original file permissions, time, soft and hard links, etc.;
  • No special permissions are required to install;
  • Optimized process and high file transfer efficiency;
  • You can use rcp, ssh, etc. to transfer files, and of course you can also connect through direct socket;
  • Supports anonymous transmission.

1.3 Working Principle


In a remote synchronization task, the client responsible for initiating the rsync synchronization operation is called the initiator, and the server responsible for responding to the rsync synchronization operation from the client is called the rsync synchronization source.

First, server B (original source) performs data backup to server A (synchronization source), and backs up its own data to server A.

When the data in server B is lost or incremented, the data will be synchronized from server A.

If the data on server B is lost, the lost part of the data will be synchronized from server A.

When the data of server B increases, the data will be backed up to server A again, but the backup will not be a complete backup, but an incremental backup, that is, the data that is not in the synchronization source will be backed up.

2. rsync related commands

2.1 Basic format and common options

#Basic format
rsync [options] origin destination destination

2.2 Starting and shutting down the rsync service

  • Start service
//Start the rsync service and run it as an independent listening service (daemon process)
rsync --daemon
  • Close service
//Close rsync service
kill $(cat /var/run/rsyncd.pid)
rm -rf /var/run/rsyncd.pid

2.3 Basic format of downlink synchronization

Pull data from source server

rsync [options] source server location local location

##Give an example#
#Format one
rsync -avz [email protected]::message /opt/

#Format 2
rsync -avz rsync://[email protected]/message /opt/

#test is the authorized account in the configuration file
#IP address is the synchronization source address
#message is the shared module defined in the configuration file

2.4 Basic format of uplink synchronization

Push data to source server

rsync [options] local location source server location

2.5 Interaction-free

2.5.1 Specify password file

echo "abc123" > /etc/server.pass
chmod 600 /etc/server.pass

2.5.2rsync-daemon mode

rsync -avz --delete --password-file=/opt/userlist [email protected]::wwwky31 /opt/data/ #rsync-daemon method

2.5.3rsync-ssh mode

rsync -avz --delete -e 'sshpass -p abc1234 ssh -p 22' /etc/yum.repos.d [email protected]:/opt/data #rsync-ssh method

2.6 Regular synchronization

Combined with crontab scheduled tasks, regular synchronization can be achieved.

#Give an example
crontab -e
30 22 * * * /usr/bin/rsync -az --delete --password-file=/etc/server.pass [email protected]::wwwroot /opt/
#In order to avoid entering a password during the synchronization process, you need to create a password file to save the password of the backuper user, such as /etc/server.pass.
#Use the option "--password-file=/etc/server.pass" to specify when performing rsync synchronization.

systemctl restart crond
systemctl enablecrond

3. Deploy rsync for regular synchronization

3.1 Configure rsync source server (node 1)

yum install -y rsync
//Install rsync service


rpm -qc rsync //View the location of the rsync configuration file

#Create the /etc/rsyncd.conf configuration file
vim /etc/rsyncd.conf //Add the following configuration items

uid=root
gid=root
use chroot = yes #imprisoned in the source directory
max connections = 4
address = 192.168.190.100 #Listening address
port = 873 #Listen to port tcp/udp 873, which can be viewed through cat /etc/services | grep rsync
log file = /var/log/rsyncd.log #Log file location
pid file = /var/run/rsyncd.pid #The file location where the process ID is stored
hosts allow = 192.168.190.0/24 #Client address allowed to be accessed
dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z #File types that are no longer compressed during synchronization

[wwwroot] #Shared module name
path = /var/www/html #The actual path of the source directory
comment = document Root of www
read only = no #Whether it is read-only
auth users = fsj #Authorized accounts, multiple accounts separated by spaces
secrets file = /opt/rsyncd_userlist#Data file to store account information
#If you use anonymous mode, just remove the "auth users" and "secrets file" configuration items.
#Create data files for backup accounts



vim /opt/rsyncd_userlist
fsj:abc123 #No need to create a system user with the same name


chmod 600 /etc/rsyncd_users.db


mkdir -p /var/www/html //Create folder

rsync --daemon #Start the rsync service and run it as an independent listening service (daemon process)


netstat -anpt | grep rsync


cd /var/www/html

cp /etc/fatab ./

3.2 Initiator (Node 2)

yum install -y rsync
//Install rsync service


Start the rsync service program
rsync --daemon
#Start the rsync service and run it as an independent listening service (daemon process)

netstat -anpt | grep rsync


cd /opt //Switch to the opt directory
ls
mkdir data //Create the data directory under the opt directory
cd data //switch to the data directory
ls


#Download the specified resources to the local/opt directory for backup.
Format one:
rsync -avz [email protected]::wwwky31 /opt/data #Password abc123


Format 2:
rsync -avz rsync://[email protected]/wwwky31 /opt/data


Interaction-free format configuration:
echo "abc123" > /etc/server.pass
chmod 600 /etc/server.pass


rsync -avz --delete --password-file=/opt/rsync_passwd [email protected]::wwwky31 /opt/data



3.3 Initiator configuration rsync + inotify

Using the inotify notification interface, you can monitor various changes in the file system, such as file access, deletion, movement, modification, etc. Using this mechanism, you can easily implement file change alarms, incremental backups, and respond promptly to changes in directories or files. Combining the inotify mechanism with the rsync tool can achieve triggered backup (real-time synchronization), that is, as long as the document in the original location changes, the incremental backup operation will be started immediately; otherwise, it will be in a silent waiting state. In this way, problems such as delays and excessive cycles that exist when backing up on a fixed cycle are avoided.
Because the inotify notification mechanism is provided by the Linux kernel, it is mainly used for local monitoring and is more suitable for upstream synchronization when applied in triggered backup.

Modify the rsync source server configuration file
vim /etc/rsyncd.conf
...
read only = no #Turn off read-only, upstream synchronization needs to be writeable

kill $(cat /var/run/rsyncd.pid)
rm -rf /var/run/rsyncd.pid
rsync --daemon
netstat -anpt | grep rsync

chmod 777 /var/www/html/

Adjust inotify kernel parameters
vim /etc/sysctl.conf
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576

Installing inotify-tools To use the inotify mechanism, you also need to install inotify-tools to provide inotifywait and inotifywatch auxiliary tool programs to monitor and summarize changes.
inotifywait: can monitor various events such as modify (modification), create (creation), move (move), delete (delete), attrib (attribute change) and other events, and output the results immediately as soon as there is a change.
inotifywatch: can be used to collect file system changes and output the summary changes after running.

cd /opt
tar zxvf inotify-tools-3.14.tar.gz
cd /opt/inotify-tools-3.14
./configure
make & amp; & amp; make install



Then open a new terminal to add files to the /var/www/html directory, move files, and track the screen output results in the original terminal.
inotifywait -mrq -e modify,create,move,delete /var/www/html


Write a triggered synchronization script in another terminal (note that the script name cannot contain the rsync string, otherwise the script may not take effect)

yum install -y rsync
//Install rsync service


cd /opt
tar zxvf inotify-tools-3.14.tar.gz
cd /opt/inotify-tools-3.14
./configure
make & amp; & amp; make install


vim inotify.sh
#!/bin/bash
INOTIFY_CMD="inotifywait -mrq -e modify,create,attrib,move,delete /var/www/html/"
RSYNC_CMD="rsync -azH --delete --password-file=/opt/rsync_passwd /vopt/data/[email protected]::wwwky31/"
#Use while and read to continuously obtain monitoring results. Based on the results, you can further determine whether the output monitoring records have been read.
$INOTIFY_CMD | while read DIRECTORY EVENT FILE
do
    if [ $(pgrep rsync | wc -l) -le 0 ] ; then
#If rsync is not executing, start immediately
        $RSYNC_CMD
    fi
done


chmod + x inotify.sh


echo admin123 > /opt/rsync_passwd
chmod 600 /opt/rsync_passwd


4. Use rsync to quickly delete a large number of files

When using rm -rf * to delete a large number of files, it is less efficient.

At this time, using the replacement principle of rsync and combining it with the –delete option, you can quickly delete a large number of files, such as the service cache.