[Comprehensive explanation of Linux commands] 130.rsync: Fast remote data synchronization tool

Article directory

  • rsync
    • Additional information
    • grammar
    • Options
    • Example
        • SSH mode
        • Backend service mode
  • Learn `python` from scratch

rsync

Remote data synchronization tool

Supplementary instructions

The rsync command is a remote data synchronization tool that can quickly synchronize files between multiple hosts through LAN/WAN. rsync uses the so-called “rsync algorithm” to synchronize files between the local and remote hosts. This algorithm only transmits different parts of the two files instead of transmitting the entire file each time. So it’s quite fast. rsync is a very powerful tool, and its command also has many functional options. We will analyze and explain its options one by one below.

Grammar

rsync [OPTION]... SRC DEST
rsync [OPTION]... SRC [USER@]host:DEST
rsync [OPTION]... [USER@]HOST:SRC DEST
rsync [OPTION]... [USER@]HOST::SRC DEST
rsync [OPTION]... SRC [USER@]HOST::DEST
rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]

Corresponding to the above six command formats, rsync has six different working modes:

  1. Copy local files. This working mode is started when neither the SRC nor the DES path information contains a single colon “:” separator. For example: rsync -a /data /backup
  2. Use a remote shell program (such as rsh, ssh) to copy the contents of the local machine to the remote machine. This mode is enabled when the DST path address contains a single colon “:” delimiter. For example: rsync -avz *.c foo:src
  3. Use a remote shell program (such as rsh, ssh) to copy the contents of the remote machine to the local machine. This mode is enabled when the SRC address path contains a single colon “:” delimiter. For example: rsync -avz foo:src/bar /data
  4. Copy files from the remote rsync server to the local machine. This mode is enabled when the SRC path information contains the “::” delimiter. For example: rsync -av [email protected]::www /databack
  5. Copy files from the local machine to the remote rsync server. This mode is enabled when the DST path information contains the “::” delimiter. For example: rsync -av /databack [email protected]::www
  6. List files on remote machine. This is similar to the rsync transfer, but just omit the local machine information in the command. For example: rsync -v rsync://192.168.78.192/www

Options

  • -v, --verbose: Verbose mode output.
  • -q, --quiet: Streamlined output mode.
  • -c, --checksum: Turn on the verification switch to force verification of file transfers.
  • -a, --archive: Archive mode, which means to transfer files recursively and keep all file attributes, equal to -rlptgoD.
  • -r, --recursive: Process subdirectories in recursive mode.
  • -R, --relative: Use relative path information.
  • -b, --backup: Create a backup, that is, when the same file name already exists for the destination, rename the old file to ~filename. You can use the --suffix option to specify a different backup file prefix.
  • --backup-dir: Store backup files (such as ~filename) in the directory.
  • --suffix=SUFFIX: Define the backup file prefix.
  • -u, --update: Just update, that is, skip all files that already exist in DST and are later than those to be backed up file, updated files are not overwritten.
  • -l, --links: Keep soft links.
  • -L, --copy-links: Treat soft links like regular files.
  • --copy-unsafe-links: Only copies links pointing outside the SRC path directory tree.
  • --safe-links: Ignore links pointing outside the SRC path directory tree.
  • -H, --hard-links: Keep hard links.
  • -p, --perms: Keep file permissions.
  • -o, --owner: Keep file owner information.
  • -g, --group: Keep file group information.
  • -D, --devices: Keep device file information.
  • -t, --times: Keep file time information.
  • -S, --sparse: Perform special processing on sparse files to save DST space.
  • -n, --dry-run: Show which files will be transferred.
  • -w, --whole-file: Copy the file without incremental detection.
  • -x, --one-file-system: Do not cross file system boundaries.
  • -B, --block-size=SIZE: The block size used by the verification algorithm, the default is 700 bytes.
  • -e, --rsh=command: Specify the use of rsh and ssh for data synchronization.
  • --rsync-path=PATH: Specify the path information of the rsync command on the remote server.
  • -C, --cvs-exclude: Use the same method as CVS to automatically ignore files to exclude files that you do not want to transfer.
  • --existing: Only update files that already exist in DST, without backing up newly created files.
  • --delete: Delete files that are not in SRC in DST.
  • --delete-excluded: Also delete files on the receiving end that are excluded by this option.
  • --delete-after: Delete after the transfer is completed.
  • --ignore-errors: Delete IO errors if they occur in a timely manner.
  • --max-delete=NUM: Delete at most NUM files.
  • --partial: Preserve files that were not completely transferred for some reason to speed up subsequent transfers.
  • --force: Force the directory to be deleted, even if it is not empty.
  • --numeric-ids: Do not match numeric user and group IDs to user and group names.
  • --timeout=time: IP timeout time, unit is seconds.
  • -I, --ignore-times: Do not skip files with the same time and length.
  • --size-only: When deciding whether to back up a file, only look at the file size without considering the file age.
  • --modify-window=NUM: The timestamp window used to determine whether files have the same time, the default is 0.
  • -T, --temp-dir=DIR: Create temporary files in DIR.
  • --compare-dest=DIR: Also compare the files in DIR to determine whether backup is needed.
  • -P: Equivalent to --partial.
  • --progress: Display the backup process.
  • -z, --compress: Compress the backup files during transmission.
  • --exclude=PATTERN: Specifies the file pattern to exclude that does not need to be transferred.
  • --include=PATTERN: Specifies the file pattern that needs to be transferred without excluding it.
  • --exclude-from=FILE: Exclude files with the pattern specified in FILE.
  • --include-from=FILE: Do not exclude files matching the pattern specified by FILE.
  • --version: Print version information.
  • --address: Bind to a specific address.
  • --config=FILE: Specify other configuration files and do not use the default rsyncd.conf file.
  • --port=PORT: Specify other rsync service ports.
  • --blocking-io: Use blocking IO for remote shells.
  • -stats: Gives the transfer status of certain files.
  • --progress: Display the transfer process during transfer.
  • --log-format=formAT: Specify the log file format.
  • --password-file=FILE: Get the password from FILE.
  • --bwlimit=KBPS: Limit I/O bandwidth, KBytes per second.
  • -h, --help: Display help information.

Example

SSH mode

First start the ssh service on the server:

service sshd start

Sync using rsync

Next, you can use the rsync command on the client to back up the data on the server. The SSH method is used to back up the system user, as follows:

rsync -vzrtopg --progress -e ssh --delete [email protected]:/www/* /databack/experiment/rsync


[email protected]'s password:
receiving file list...
5 files to consider
test/
a
0 100% 0.00kB/s 527:35:41 (1, 20.0% of 5)
b
67 100% 65.43kB/s 0:00:00 (2, 40.0% of 5)
c
0 100% 0.00kB/s 527:35:41 (3, 60.0% of 5)
dd
100663296 100% 42.22MB/s 0:00:02 (4, 80.0% of 5)
sent 96 bytes received 98190 bytes 11563.06 bytes/sec
total size is 100663363 speedup is 1024.19

The above information describes the entire backup process and the total size of the backup data.

Backend service mode

Start the rsync service, edit the /etc/xinetd.d/rsync file, change disable=yes to disable=no, and restart the xinetd service, as follows:

vi /etc/xinetd.d/rsync

#default: off
# description: The rsync server is a good addition to an ftp server, as it \
# allows crc checksumming etc.
service rsync {<!-- -->
disable=no
socket_type = stream
wait=no
user=root
server = /usr/bin/rsync
server_args = --daemon
log_on_failure + = USERID
}

/etc/init.d/xinetd restart
Stop xinetd: [OK]
Start xinetd: [OK]

Create a configuration file. After the rsync program is installed by default, the main configuration file of rsync will not be automatically created. It needs to be created manually. The main configuration file is “/etc/rsyncd.conf”. Create the file and insert the following content:

vi /etc/rsyncd.conf

uid=root
gid=root
max connections=4
log file=/var/log/rsyncd.log
pid file=/var/run/rsyncd.pid
lock file=/var/run/rsyncd.lock
secrets file=/etc/rsyncd.passwd
hosts deny=172.16.78.0/22

[www]
comment= backup web
path=/www
read only=no
exclude=test
auth users=work

Create a password file. In this way, the system user cannot be used to authenticate the client, so you need to create a password file in the format of “username:password”. The username and password can be defined arbitrarily. It is best not to be consistent with the system account. , and at the same time set the permissions of the created password file to 600, which is described in detail in the previous module parameters.

echo "work:abc123" > /etc/rsyncd.passwd
chmod 600 /etc/rsyncd.passwd

Backup, after completing the above work, you can now back up the data, as follows:

rsync -avz --progress --delete [email protected]::www /databack/experiment/rsync

Password:
receiving file list...
6 files to consider
./
a
0 100% 0.00kB/s 528:20:41 (1, 50.0% of 6)
b
67 100% 65.43kB/s 0:00:00 (2, 66.7% of 6)
c
0 100% 0.00kB/s 528:20:41 (3, 83.3% of 6)
dd
100663296 100% 37.49MB/s 0:00:02 (4, 100.0% of 6)
sent 172 bytes received 98276 bytes 17899.64 bytes/sec
total size is 150995011 speedup is 1533.75

Recovery, when there is a problem with the server’s data, then the server needs to be restored through the client’s data, but the premise is that the server allows the client to have write permission, otherwise the server cannot be restored directly on the client. , the method of using rsync to recover data is as follows:

rsync -avz --progress /databack/experiment/rsync/ [email protected]::www

Password:
building file list...
6 files to consider
./
a
b
67 100% 0.00kB/s 0:00:00 (2, 66.7% of 6)
c
sent 258 bytes received 76 bytes 95.43 bytes/sec
total size is 150995011 speedup is 452080.87

Synchronize source directory to target directory

$ rsync -r source destination

In the above command, -r means recursion, that is, including subdirectories. Note that -r is required, otherwise rsync will not run successfully. The source directory represents the source directory, and destination represents the destination directory.

Synchronize multiple files or directories

$ rsync -r source1 source2 destination

In the above command, source1 and source2 will be synchronized to the destination directory.

Sync meta information

The -a parameter can replace -r. In addition to recursive synchronization, it can also synchronize meta-information (such as modification time, permissions, etc.). Since rsync by default uses file size and modification time to determine whether a file needs to be updated, -a is more useful than -r. The following usage is the common way of writing.

$ rsync -a source destination

The target directory becomes a mirror copy of the source directory

By default, rsync simply ensures that all contents of the source directory (except explicitly excluded files) are copied to the target directory. It doesn’t leave the two directories the same, and it doesn’t delete the files. If you want to make the target directory a mirror copy of the source directory, you must use the --delete parameter, which will delete files that only exist in the target directory and not in the source directory.

$ rsync -av --delete source/destination

Exclude files

Sometimes, we want to exclude certain files or directories during synchronization. In this case, we can use the --exclude parameter to specify the exclusion mode.

$ rsync -av --exclude='*.txt' source/destination
# or
$ rsync -av --exclude '*.txt' source/destination

The above command excludes all TXT files.

Note that rsync will synchronize hidden files starting with “dot”. If you want to exclude hidden files, you can write --exclude=".*" like this.

If you want to exclude all files in a directory, but do not want to exclude the directory itself, you can write it as follows.

$ rsync -av --exclude 'dir1/*' source/destination

For multiple exclusion modes, multiple --exclude parameters can be used.

$ rsync -av --exclude 'file1.txt' --exclude 'dir1/*' source/destination

Multiple exclusion patterns can also take advantage of Bash’s large-expansion feature, using just a single --exclude parameter.

$ rsync -av --exclude={'file1.txt','dir1/*'} source/ destination

If there are many exclusion patterns, you can write them to a file, one line per pattern, and then specify this file with the --exclude-from parameter.

$ rsync -av --exclude-from='exclude-file.txt' source/destination

Specify file modes that must be synchronized

The --include parameter is used to specify the file mode that must be synchronized, and is often used in conjunction with --exclude.

$ rsync -av --include="*.txt" --exclude='*' source/destination

The above command specifies that when synchronizing, all files will be excluded, but TXT files will be included.

Learn python

from scratch

[Learn python from scratch] 92. Use Python’s requests library to send HTTP requests and process responses
[Learn python from scratch] 91. A simple web application that uses decorators and dictionaries to manage request paths
[Learn python from scratch] 93. Use dictionaries to manage request paths
[Learn python from scratch] 89. Use WSGI to build a simple and efficient web server
[Learn python from scratch] 88. Detailed explanation of WSGI interface: achieving simple and efficient web development
[Learn python from scratch] 87. Manually build Python implementation of HTTP server and multi-thread concurrent processing
[Learn python from scratch] 86. In-depth understanding of the HTTP protocol and its role in browser and server communication
[Learn python from scratch] 85. Application of parallel computing technology in Python process pool
[Learn python from scratch] 84. In-depth understanding of threads and processes
[Learn python from scratch] 83. Python multi-process programming and the use of process pools
[Learn python from scratch] 82. Implementation of chat program based on multi-threading
[Learn python from scratch] 81. Application of Python multi-thread communication and queue
[Learn python from scratch] 80. Thread access to global variables and thread safety issues
[Learn python from scratch] 79. Thread access to global variables and thread safety issues
[Learn python from scratch] 78. File download case
[Learn python from scratch] 77. TCP server programming and precautions
[Learn python from scratch] 76. Server and client: key components of network communication
[Learn python from scratch] 75. TCP protocol: a reliable connection-oriented transport layer communication protocol
[Learn python from scratch] 74. UDP network program: detailed explanation of port issues and binding information
[Learn python from scratch] 73. UDP network program-send data
[Learn python from scratch] 72. In-depth understanding of Socket communication and methods of creating sockets
[Learn python from scratch] 71. Network ports and their functions
[Learn python from scratch] 70. Network communication methods and their applications: from direct communication to routers connecting multiple networks
[Learn python from scratch] 69. Network communication and IP address classification analysis
[Learn python from scratch] 68. Greedy and non-greedy modes in Python regular expressions
[Learn python from scratch] 67. The re module in Python: regular replacement and advanced matching technology
[Learn python from scratch] 66. In-depth understanding of regular expressions: a powerful tool for pattern matching and text processing
[Learn python from scratch] 65. Detailed explanation of Python regular expression modifiers and their applications
[Learn python from scratch] 64. Detailed explanation of the use of re.compile method in Python regular expressions
[Learn python from scratch] 63. Introduction to the re.Match class and its attributes and methods in regular expressions
[Learn python from scratch] 62. Python regular expressions: a powerful string matching tool
[Learn python from scratch] 61. Detailed explanation and application examples of property attributes in Python
[Learn python from scratch] 60. Exploration generator: a flexible tool for iteration
[Learn python from scratch] 59. Iterator: an efficient tool for optimizing data traversal
[Learn python from scratch] 58. Custom exceptions in Python and methods of raising exceptions
[Learn python from scratch] 57. Use the with keyword in Python to correctly close resources
[Learn python from scratch] 56. The importance and application of exception handling in programming
[Learn python from scratch] 55. Serialization and deserialization in Python, application of JSON and pickle modules
[Learn python from scratch] 54. Write data in memory
[Learn python from scratch] 53. CSV files and Python’s CSV module
[Learn python from scratch] 52. Reading and writing files – Python file operation guide
[Learn python from scratch] 51. Opening and closing files and their application in Python
[Learn python from scratch] 49. Object-related built-in functions in Python and their usage
[Learn python from scratch] 48. Detailed explanation of inheritance and multiple inheritance in Python
[Learn python from scratch] 47. The concept and basic use of inheritance in object-oriented programming
[Learn python from scratch] 46. Analysis of __new__ and __init__ methods and singleton design pattern in Python
[Learn python from scratch] 45. Class methods and static methods in Python
[Learn python from scratch] 44. Private properties and methods in object-oriented programming
[Learn python from scratch] 43. Instance attributes and class attributes in Python object-oriented programming
[Learn python from scratch] 42. Built-in properties and methods in Python
[Learn python from scratch] 41. python magic method (2)
[Learn python from scratch] 40. python magic method (1)
[Learn python from scratch] 39. Basic object-oriented syntax and application examples
[Learn python from scratch] 38. How to use and import Python packages
[Learn python from scratch] 37. The use and precautions of Python custom modules
[Learn python from scratch] 36. Methods and techniques for using pip for third-party package management in Python
[Learn python from scratch] 35. Python common system modules and their usage
[Learn python from scratch] 34. Detailed explanation of how to import and use Python modules
[Learn python from scratch] 33. The role of decorators (2)
[Learn python from scratch] 32. The role of decorators (1)
[Learn python from scratch] 31. In-depth understanding of higher-order functions and closures in Python
[Learn python from scratch] 30. In-depth understanding of recursive functions and anonymous functions
[Learn python from scratch] 29. “Detailed explanation of function parameters” – Understand the different uses of Python function parameters
[Learn python from scratch] 28. Local variables and global variables in Python
[Learn python from scratch] 27. Use of Python functions and nested calls
[Learn python from scratch] 25. Function: a powerful tool to improve the efficiency of code writing
[Learn python from scratch] 24. String operations and traversal methods in Python
[Learn python from scratch] 23. How to use sets (set) and common operations in Python
[Learning python from scratch] 22. Addition, deletion, modification, and dictionary variables in Python
[Learn python from scratch] 21. Tuples and dictionaries in Python
[Learn python from scratch] 20. Python list operation skills and examples
[Learn python from scratch] 19. Application of looping through lists and list nesting
[Learn python from scratch] 18. Detailed explanation of the basic operations of Python lists (1)
[Learn python from scratch] 17. Python string format method (2)
[Learn python from scratch] 16. Python string format method (1)
[Learn python from scratch] 15. In-depth understanding of string and character set encoding
[Learn python from scratch] 14. Common operations on Python strings (2)
[Learn python from scratch] 13. Common operations on Python strings (1)
[Learn python from scratch] 12. Python string operations and applications
[Learn python from scratch] 11. Python loop statements and control flow
[Learn python from scratch] 10. Detailed explanation of Python conditional statements and if nesting
[Learn python from scratch] 09. Conditional judgment statements in Python
[Learn python from scratch] 08. Python understands bitwise operators and operator precedence
[Learn python from scratch] 07. Detailed explanation of Python operators: assignment, comparison and logical operators
[Learn python from scratch] 06. Use arithmetic operators for calculation and string concatenation in Python
[Learn python from scratch] 05. Output and input in Python
[Learn python from scratch] 04. Python programming basics: variables, data types and identifiers
[Learn python from scratch] 03. Python interactive programming and detailed explanation of comments
[Learn python from scratch] 02. Introduction to development tools
[Learn python from scratch] 01. Install and configure python