Table of Contents
1. Introduction to rsync:
2. Principle of rsync copying:
3. rsync synchronization method:
4. Basic format:
1. Commonly used options:
2. Two methods of synchronization:
3. Interaction-free format configuration:
5. Configure rsync source server:
1. Preparation:
2. Write configuration file:
3. Create a data password file for the backup account:
4. Add permissions to the synchronized directory:
5. Start the rsync service program:
6. Synchronization:
6. Configure rsync + inotify:
1. The source server modifies the rsync source server configuration file:
2. Adjust inotify kernel parameters
3． Install inotify-tools:
3.1 inotifywait option:
4. Write a triggered synchronization script in another terminal:
7. Use rsync to quickly delete a large number of files
1. Introduction to rsync:
- An open source fast backup tool that can mirror and synchronize the entire directory tree between different hosts and support incremental backup
- Compression is performed before transfer, making it ideal for off-site backups, mirror servers, etc.
- The client responsible for initiating rsync synchronization operations is called the initiator, and the server responsible for responding to rsync synchronization operations from the client is called the synchronization source. During the synchronization process, the synchronization source is responsible for providing the original location of the file, to which the initiating end should have read permissions.
- Official website: https://rsync.samba.org/
2. Principle of rsync copying:
- First, compare the source file and the file in the target location to find the differences.
- Consistency synchronization is performed based on the differences between the files in the target location and the source files.
3. rsync synchronization method:
1. Local copy: First, compare the source file and the destination file to find the differences, and then synchronize the differences between the source file and the destination location consistently.
2. Downstream synchronization: The client and server synchronize content, which can be understood as the client downloading content from the server.
3. Upstream synchronization: The server and the client synchronize content, which can be understood as the client uploading content to the server.
4. Basic format:
rsync [options] original location target location
1. Commonly used options:
-r: Recursive mode, including all files in the directory and subdirectories.
-l: Symbolic link files are still copied as symbolic link files.
-v: Display detailed (verbose) information about the synchronization process.
-z: Compress when transferring files.
-a: Archive mode, retains file permissions, attributes and other information, equivalent to the combination option “-rlptgoD”.
-p: Preserves the file’s permission flags.
-t: Keep the timestamp of the file.
-g: Preserve the file’s group tag (only for superusers).
-o: Keep the file’s owner mark (for superuser use only).
-H: Keep hard-linked files.
-A: Preserve ACL attribute information.
-D: Preserve device files and other special files.
–delete: Delete files that exist in the target location but not in the original location.
–checksum: Determine whether to skip files based on checksum (rather than file size or modification time).
2. Two methods of synchronization:
Format 1: rsync -avz [email protected]::wwwroot /opt/ #Password abc123 Format two: rsync -avz rsync://[email protected]/wwwroot /opt/
3. Interaction-free format configuration:
echo "abc123" > /etc/server.pass chmod 600 /etc/server.pass
5. Configure rsync source server:
1. Preparation:
systemctl stop firewalld setenforce 0 rpm -q rsync ##General systems have rsync installed by default
2. Write configuration file:
vim /etc/rsyncd.conf #Add the following configuration items uid=root gid=root use chroot = yes #imprisoned in the source directory address = 192.168.80.10 #Listening address port = 873 #Listen to port tcp/udp 873, which can be viewed through cat /etc/services | grep rsync log file = /var/log/rsyncd.log #Log file location pid file = /var/run/rsyncd.pid #The file location where the process ID is stored hosts allow = 192.168.80.0/24 #Client address allowed to be accessed dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z #File types that are no longer compressed during synchronization [wwwroot] #Shared module name path = /var/www/html #The actual path of the source directory comment = Document Root of www.kgc.com read only = yes #Whether it is read-only auth users = backuper #Authorized accounts, multiple accounts separated by spaces secrets file = /etc/rsyncd_users.db #Data file to store account information #If you use anonymous mode, just remove the "auth users" and "secrets file" configuration items.
3. Create a data password file for the backup account:
vim /etc/rsyncd_users.db backuper:abc123 #No need to create a system user with the same name chmod 600 /etc/rsyncd_users.db
4. Add permissions to the synchronized directory:
Ensure that all users have read permissions to the source directory /var/www/html
chmod + r /var/www/html/ ls -ld /var/www/html/
5. Start the rsync service program:
#Start rsync service program rsync --daemon #Start the rsync service and run it as an independent listening service (daemon process) netstat -anpt | grep rsync #Close rsync service kill $(cat /var/run/rsyncd.pid) rm -rf /var/run/rsyncd.pid
6. Sync:
crontab -e 30 22 * * * /usr/bin/rsync -az --delete --password-file=/etc/server.pass [email protected]::wwwroot /opt/ #In order to avoid entering a password during the synchronization process, you need to create a password file to save the password of the backuper user, such as /etc/server.pass. Just specify the option "--password-file=/etc/server.pass" when performing rsync synchronization.
At this time, we can combine the scheduled tasks to synchronize every two minutes, but it does not achieve real-time synchronization. If the server goes down in this world, there will still be losses. Next, we can use inotify to solve this problem< /strong>
6. Configure rsync + inotify:
- Using the inotify notification interface, you can monitor various changes in the file system, such as file access, deletion, movement, modification, etc. Using this mechanism, you can easily implement file change alarms, incremental backups, and respond promptly to changes in directories or files.
- Combining the inotify mechanism with the rsync tool can achieve triggered backup (real-time synchronization), that is, as long as the document in the original location changes, the incremental backup operation will be started immediately; otherwise, it will be in a silent waiting state. In this way, problems such as delays and excessive cycles that exist when backing up on a fixed cycle are avoided.
- Because the inotify notification mechanism is provided by the Linux kernel, it is mainly used for local monitoring. It is more suitable for upstream synchronization when applied in triggered backup.
1. The source server modifies the rsync source server configuration file:
vim /etc/rsyncd.conf ... read only = no #Turn off read-only, upstream synchronization needs to be writeable kill $(cat /var/run/rsyncd.pid) rm -rf /var/run/rsyncd.pid rsync --daemon netstat -anpt | grep rsync #Restart the rsyncd service to refresh the configuration file chmod 777 /var/www/html/ #Authorize the synchronized directory
2. Adjust inotify kernel parameters
When the number of directories and files to be monitored is large or changes frequently, it is recommended to increase the values of these three parameters.
vim /etc/sysctl.conf fs.inotify.max_queued_events = 16384 monitoring event queue, the default value is 16384 fs.inotify.max_user_instances = 1024 Maximum number of monitoring instances, the default value is 128 fs.inotify.max_user_watches = 1048576 Maximum number of monitoring instances, the default value is 128 sysctl-p
3. Install inotify-tools:
To use the inotify mechanism, you also need to install inotify-tools to provide inotifywait and inotifywatch auxiliary tool programs to monitor and summarize changes.
- inotifywait: can monitor various events such as modify (modification), create (creation), move (move), delete (delete), attrib (attribute change) and other events, and output the results immediately as soon as there is a change.
- inotifywatch: can be used to collect file system changes and output the summary changes after running.
tar zxvf inotify-tools-3.14.tar.gz -C /opt/ cd /opt/inotify-tools-3.14 ./configure make & amp; & amp; make install
3.1 inotifywait option:
#Option “-e”: used to specify which events to monitor
#Option “-m”: indicates continuous monitoring
#Option “-r”: indicates recursively the entire directory
#Option “-q”: simplify output information
4. Write a triggered synchronization script in another terminal:
vim /opt/inotify.sh #!/bin/bash INOTIFY_CMD="inotifywait -mrq -e modify,create,attrib,move,delete /var/www/html/" RSYNC_CMD="rsync -azH --delete --password-file=/etc/server.pass /var/www/html/ [email protected]::wwwroot/" #Use while and read to continuously obtain monitoring results. Based on the results, you can further determine whether the output monitoring records have been read. $INOTIFY_CMD | while read DIRECTORY EVENT FILE do if [ $(pgrep rsync | wc -l) -le 0 ] ; then #If rsync is not executing, start immediately $RSYNC_CMD fi done chmod +x /opt/inotify.sh chmod 777 /var/www/html/ chmod +x /etc/rc.d/rc.local echo '/opt/inotify.sh' >> /etc/rc.d/rc.local #Add automatic execution at boot
7. Use rsync to quickly delete a large number of files
If you want to delete a large number of files under Linux, such as 1 million or 10 million, such as the nginx cache of /usr/local/nginx/proxy_temp, etc., then rm -rf * may not be useful because you have to wait for a long time. a period of time. In this case we can use rsync to handle it cleverly. rsync actually uses the substitution principle.
mkdir /home/blank Use rsync to delete the target directory: rsync --delete-before -a -H -v --progress --stats /home/blank/ /usr/local/nginx/proxy_temp/ In this way, the target directory will be cleared quickly. Option description: --delete-before The receiver performs delete operations during transmission -a archive mode, which means to transfer files recursively and keep all file attributes -H keep hard-linked files -v verbose output mode --progress displays the transfer progress during transfer --stats gives the transfer status of certain files
The knowledge points of the article match the official knowledge files, and you can further learn relevant knowledge. Cloud native entry-level skills treeHomepageOverview 16,365 people are learning the system