Nosql redis high availability and persistence

Nosql redis high availability and persistence

  • 1. redis high availability
  • 2. redis persistence
    • 2.1redis persistence
    • 2.2Redis persistence method
    • 2.3RDB persistence
      • 2.3.1 How RDB persistence works
      • 2.3.2 Trigger conditions
      • 2.3.3 Other automatic triggering mechanisms
      • 2.3.4 Execution process
      • 2.3.5 Loading at startup
    • 2.4AOF persistence
      • 2.4.1AOF persistence principle
      • 2.4.2 Turn on AOF
      • 2.4.3 Execution process
      • 2.4.4 File rewriting process
      • 2.4.5 Loading at startup
    • 2.5 Advantages and Disadvantages of RDB and AOF
      • 2.5.1RDB persistence
      • 2.5.2AOF persistence
  • 3. Redis performance management
    • 3.1 Check Redis memory usage
    • 3.2 Memory fragmentation rate
    • 3.3 Track memory fragmentation rate
    • 3.4 Memory usage
    • Recycle key within 3.5
  • 4. Summary

1. redis high availability

Redis high availability and persistence are very important technologies in enterprises. When a single point of failure occurs, high availability must be used to resist risks. To ensure data security, data must be persisted and written to disk.

redis high availability method

  • Persistence: Persistence is the simplest high-availability method (sometimes not even classified as a high-availability method). Its main function is data backup, that is, storing data on the hard disk to ensure that the data will not be lost when the process exits.
  • Master-slave replication: Master-slave replication is the basis of high-availability Redis. Both Sentinel and Cluster achieve high availability based on master-slave replication. Master-slave replication mainly implements multi-machine backup of data, as well as load balancing and simple fault recovery for read operations. Disadvantages: Failure recovery cannot be automated; write operations cannot be load balanced; storage capacity is limited by a single machine.
  • Sentinel: Based on master-slave replication, Sentinel implements automated fault recovery. Disadvantages: Write operations cannot be load balanced; storage capacity is limited by a single machine.
  • Cluster cluster: Through clustering, Redis solves the problem that write operations cannot be load balanced and storage capacity is limited by a single machine, achieving a relatively complete high-availability solution.

2. redis persistence

2.1redis persistence

Persistence function: Redis is an in-memory database, and data is stored in memory. In order to avoid permanent loss of data after the Redis process exits abnormally due to server power outages and other reasons, the data in Redis needs to be stored in some form (data or command) is saved from the memory to the hard disk; when Redis restarts next time, the persistent file is used to achieve data recovery. In addition, persistent files can be copied to a remote location for disaster backup purposes.

2.2Redis persistence method

  • RDB persistence: The principle is to regularly save Reids database records in memory to disk.
  • AOF persistence (append only file): The principle is to write Reids’ operation log to the file in an appending manner, similar to MySQL’s binlog.

Since AOF persistence has better real-time performance, that is, less data is lost when the process exits unexpectedly, AOF is currently the mainstream persistence method, but RDB persistence still has its place.

2.3RDB persistence

2.3.1 Working principle of RDB persistence

RDB persistence refers to generating a snapshot of the data in the current process in memory and saving it to the hard disk within a specified time interval (so it is also called snapshot persistence). It is stored using binary compression and the saved file suffix is rdb; when Redis restarts , the snapshot file can be read to restore data.

2.3.2 Trigger conditions

The triggering of RDB persistence is divided into two types: manual triggering and automatic triggering.

  • manual trigger

Both the save command and the bgsave command can generate RDB files.

1. The save command will block the Redis server process until the RDB file is created. While the Redis server is blocked, the server cannot process any command requests.
2. The bgsave command will create a child process, which will be responsible for creating the RDB file, while the parent process (ie the Redis main process) will continue to process requests.
3. During the execution of the bgsave command, only the fork child process will block the server, while for the save command, the entire process will block the server. Therefore, save has been basically abandoned, and the use of save must be eliminated in the online environment.

  • Automatic trigger

When RDB persistence is automatically triggered, Redis will also choose bgsave instead of save for persistence.

save m n
The most common case of automatic triggering is through save m n in the configuration file, which specifies that bgsave will be triggered when n changes occur within m seconds.

vim /etc/redis/6379.conf
--Line 219--When any of the following three save conditions is met, it will cause bgsave to be called.
save 900 1: When the time reaches 900 seconds, if the redis data has changed at least once, execute bgsave
save 300 10: When the time reaches 300 seconds, if the redis data has changed at least 10 times, execute bgsave
save 60 10000: When the time reaches 60 seconds, if the redis data has changed at least 10000 times, execute bgsave
--Line 254--Specify the RDB file name
dbfilenamedump.rdb
--Line 264--Specify the directory where the RDB file and AOF file are located
dir /var/lib/redis/6379
--Line 242--Whether to enable RDB file compression
rdbcompression yes

2.3.3 Other automatic trigger mechanisms

In addition to save m n, there are some other situations that trigger bgsave:

  • In the master-slave replication scenario, if the slave node performs a full copy operation, the master node will execute the bgsave command and send the rdb file to the slave node.
  • When executing the shutdown command, RDB persistence is automatically performed.

2.3.4 Execution Process

(1) The Redis parent process first determines whether it is currently executing save or the child process of bgsave/bgrewriteaof. If it is executing, the bgsave command returns directly. The child processes of bgsave/bgrewriteaof cannot be executed at the same time, mainly due to performance considerations: two concurrent child processes perform a large number of disk write operations at the same time, which may cause serious performance problems.
(2) The parent process performs a fork operation to create a child process. During this process, the parent process is blocked and Redis cannot execute any commands from the client.
(3) After the parent process forks, the bgsave command returns the “Background saving started” message and no longer blocks the parent process, and can respond to other commands.
(4) The child process creates an RDB file, generates a temporary snapshot file based on the memory snapshot of the parent process, and atomically replaces the original file after completion.
(5) The child process sends a signal to the parent process to indicate completion, and the parent process updates statistical information.

2.3.5 Loading at startup

The loading of RDB files is automatically executed when the server starts, and there is no special command. However, because AOF has a higher priority, when AOF is turned on, Redis will give priority to loading the AOF file to restore data; only when AOF is turned off, the RDB file will be detected and automatically loaded when the Redis server starts. The server is blocked while loading the RDB file until the loading is completed.
When Redis loads an RDB file, it will verify the RDB file. If the file is damaged, an error will be printed in the log and Redis will fail to start.

2.4AOF persistence

2.4.1AOF persistence principle

RDB persistence writes process data to files, while AOF persistence records each write and delete command executed by Redis into a separate log file, and query operations are not recorded; when Redis restarts, the AOF file is executed again command to recover data.
Compared with RDB, AOF has better real-time performance, so it has become a mainstream persistence solution.

2.4.2 Turn on AOF

//Enable AOF
The Redis server turns on RDB by default and turns off AOF. To turn on AOF, you need to configure it in the configuration file:
vim /etc/redis/6379.conf
--Line 700--Modify, enable AOF
appendonly yes
--Line 704--Specify the AOF file name
appendfilename "appendonly.aof"
--Line 796--Whether to ignore the last potentially problematic instruction
aof-load-truncated yes<br>Start redis<br>/etc/init.d/redis_6379 restart

2.4.3 Execution process

Since each write command of Redis needs to be recorded, AOF does not need to be triggered. The execution process of AOF is introduced below.
The execution process of AOF includes:

  • Command append (append): Append the Redis write command to the buffer aof_buf;
  • File writing (write) and file synchronization (sync): synchronize the content in aof_buf to the hard disk according to different synchronization strategies;
  • File rewrite (rewrite): Rewrite AOF files regularly to achieve compression.

(1) Command append (append)
Redis first appends the write command to the buffer instead of writing it directly to the file. This is mainly to avoid writing the write command directly to the hard disk every time, causing the hard disk IO to become the bottleneck of Redis load.
The format of the command append is the protocol format requested by the Redis command. It is a plain text format and has the advantages of good compatibility, strong readability, easy processing, simple operation and avoidance of secondary overhead. In the AOF file, except for the select command used to specify the database (such as select 0 to select database No. 0), which is added by Redis, the others are write commands sent by the client.
(2) File writing (write) and file synchronization (sync)
Redis provides a variety of synchronization file strategies for the AOF cache area. The strategies involve the write function and fsync function of the operating system. The description is as follows:
In order to improve file writing efficiency, in modern operating systems, when users call the write function to write data to a file, the operating system usually temporarily stores the data in a memory buffer. When the buffer is filled or exceeds the specified After the time limit, the buffer data is actually written to the hard disk. Although such an operation improves efficiency, it also brings security issues: if the computer shuts down, the data in the memory buffer will be lost; therefore, the system also provides synchronization functions such as fsync and fdatasync, which can force the operating system to immediately transfer the data in the buffer to The data is written to the hard disk to ensure data security.

There are three synchronization methods for the synchronization file strategy of the AOF cache area. They are:
vim /etc/redis/6379.conf
--729--
appendfsync always: Immediately after the command is written to aof_buf, the system fsync operation is called to synchronize to the AOF file. The thread returns after fsync is completed. In this case, every write command must be synchronized to the AOF file, and hard disk IO becomes a performance bottleneck. Redis can only support about a few hundred TPS writes, seriously reducing the performance of Redis; even if a solid-state drive (SSD) is used, It can only handle tens of thousands of commands per second, and it will greatly reduce the life of the SSD.
appendfsync no: After the command is written to aof_buf, the system write operation is called, and fsync synchronization of the AOF file is not performed; the synchronization is handled by the operating system, and the synchronization period is usually 30 seconds. In this case, the file synchronization time is uncontrollable, and there will be a lot of data accumulated in the buffer, and data security cannot be guaranteed.
appendfsync everysec: After the command is written into aof_buf, the system write operation is called. The thread returns after the write is completed; the fsync synchronization file operation is called once per second by a dedicated thread. everysec is a compromise between the two aforementioned strategies, a balance between performance and data security. Therefore, it is the default configuration of Redis and our recommended configuration.

(3) File rewrite (rewrite)
As time goes by, the Redis server executes more and more write commands, and the AOF file will become larger and larger; an excessively large AOF file will not only affect the normal operation of the server, but also cause data recovery to take too long.
File rewriting refers to rewriting AOF files regularly to reduce the size of AOF files. It should be noted that AOF rewriting converts the data in the Redis process into write commands and synchronizes them to the new AOF file; no reading or writing operations will be performed on the old AOF file!
Another point to note about file rewriting is: for AOF persistence, although file rewriting is highly recommended, it is not necessary; even without file rewriting, data can be persisted and started in Redis Time import; therefore in some realities, automatic file rewriting will be turned off, and then executed regularly at a certain time every day through a scheduled task.

#The reason why file rewriting can compress AOF files is because:
●Expired data will no longer be written to the file
●Invalid commands are no longer written to the file: for example, some data are repeatedly set (set mykey v1, set mykey v2), some data are deleted (set myset v1, del myset), etc.
●Multiple commands can be merged into one: for example, sadd myset v1, sadd myset v2, sadd myset v3 can be merged into sadd myset v1 v2 v3.

As can be seen from the above, since AOF executes fewer commands after rewriting, file rewriting can not only reduce the space occupied by the file, but also speed up recovery.
#The triggering of file rewriting is divided into manual triggering and automatic triggering:
●Manual triggering: Directly call the bgrewriteaof command. The execution of this command is somewhat similar to bgsave: both fork sub-processes to perform specific work, and both are blocked only during fork.
●Automatic triggering: Automatically execute BGREWRITEAOF by setting the auto-aof-rewrite-min-size option and the auto-aof-rewrite-percentage option.
Only when the two options auto-aof-rewrite-min-size and auto-aof-rewrite-percentage are satisfied at the same time, AOF rewriting, that is, the bgrewriteaof operation will be automatically triggered.

vim /etc/redis/6379.conf
--729--
auto-aof-rewrite-percentage 100: When the current AOF file size (i.e. aof_current_size) is twice the AOF file size (aof_base_size) during the last log rewrite, the BGREWRITEAOF operation occurs
auto-aof-rewrite-min-size 64mb: The minimum value for executing the BGREWRITEAOF command on the current AOF file to avoid frequent BGREWRITEAOF caused by the small file size when Reids is first started.<br><br>Regarding the process of file rewriting, there are two points that need special attention: (1) The rewriting is performed by the parent process forking the child process; (2) The write command executed by Redis during the rewriting needs to be appended to the new AOF file. For this reason Redis introduces aof_rewrite_buf cache.

2.4.4 File rewriting process

(1) The Redis parent process first determines whether there is currently a child process executing bgsave/bgrewriteaof. If it exists, the bgrewriteaof command will return directly. If the bgsave command exists, it will wait until the bgsave execution is completed before executing it.
(2) The parent process performs a fork operation to create a child process. During this process, the parent process is blocked.
(3.1) After the parent process forks, the bgrewriteaof command returns the “Background append only file rewrite started” message and no longer blocks the parent process.
and can respond to other commands. All Redis write commands are still written to the AOF buffer and synchronized to the hard disk according to the appendfsync policy to ensure the correctness of the original AOF mechanism.
(3.2) Since the fork operation uses copy-on-write technology, the child process can only share the memory data during the fork operation. Since the parent process is still responding to the command, Redis uses the AOF rewrite buffer (aof_rewrite_buf) to save this part of the data to prevent this part of the data from being lost during the generation of the new AOF file. In other words, during the execution of bgrewriteaof, the Redis write command is appended to the two buffers aof_buf and aof_rewirte_buf at the same time.
(4) The child process writes to the new AOF file according to the memory snapshot and the command merging rules.
(5.1) After the child process writes the new AOF file, it sends a signal to the parent process, and the parent process updates the statistical information, which can be viewed through info persistence.
(5.2) The parent process writes the data in the AOF rewrite buffer to the new AOF file, thus ensuring that the database state saved in the new AOF file is consistent with the current state of the server.
(5.3) Use the new AOF file to replace the old file and complete the AOF rewriting.

2.4.5 Loading at startup

When AOF is turned on, Redis will load the AOF file first to restore data when it starts; only when AOF is turned off, the RDB file will be loaded to restore data.
When AOF is turned on but the AOF file does not exist, the RDB file will not be loaded even if it exists.
When Redis loads an AOF file, it will verify the AOF file. If the file is damaged, an error will be printed in the log and Redis will fail to start. However, if the end of the AOF file is incomplete (a sudden machine downtime can easily cause the end of the file to be incomplete), and the aof-load-truncated parameter is turned on, a warning will be output in the log, and Redis will ignore the end of the AOF file and start successfully. The aof-load-truncated parameter is enabled by default.

2.5 Advantages and Disadvantages of RDB and AOF

2.5.1RDB persistence

Advantages: RDB files are compact, small in size, fast in network transmission, suitable for full copy; recovery speed is much faster than AOF. Of course, one of the most important advantages of RDB is that it has a relatively small impact on performance compared to AOF.

Disadvantages: The fatal disadvantage of RDB files is that the persistence method of data snapshots determines that real-time persistence cannot be achieved. Today, when data is becoming more and more important, a large amount of data loss is often unacceptable, so AOF persistence become mainstream. In addition, RDB files need to meet a specific format and have poor compatibility (for example, the old version of Redis is not compatible with the new version of RDB files).
For RDB persistence, on the one hand, the Redis main process will be blocked when bgsave performs a fork operation. On the other hand, the child process writing data to the hard disk will also bring IO pressure.

2.5.2AOF persistence

Corresponding to RDB persistence, AOF has the advantage of supporting second-level persistence and good compatibility. The disadvantage is that the file is large, the recovery speed is slow, and the performance is greatly affected.

For AOF persistence, the frequency of writing data to the hard disk is greatly increased (second level under the everysec policy), and the IO pressure is greater, which may even cause AOF append blocking problems. The rewriting of AOF files is similar to RDB’s bgsave, and there will be problems of blocking during fork and IO pressure of the child process. Relatively speaking, because AOF writes data to the hard disk more frequently, it will have a greater impact on the performance of the Redis main process.

3. Redis performance management

3.1 View Redis memory usage

info memory

3.2 Memory fragmentation rate

The memory value used_memory_rss allocated by the operating system is divided by the memory value used_memory used by Redis. The calculated memory fragmentation is caused by the inefficient allocation/recycling of physical memory by the operating system (discontinuous physical memory allocation).

3.3 Tracking memory fragmentation rate

Tracking memory fragmentation rate is very important to understand the resource performance of a Redis instance:

  • It is reasonable for the memory fragmentation rate to be slightly greater than 1. This value indicates that the memory fragmentation rate is relatively low.
  • If the memory fragmentation rate exceeds 1.5, it means that Redis consumes 150 of the actual physical memory required, of which 50 is the memory fragmentation rate. You need to enter the shutdown save command on the redis-cli tool and restart the Redis server.
  • If the memory fragmentation rate is lower than 1, it means that the Redis memory allocation exceeds the physical memory and the operating system is performing memory swap. It is necessary to increase the available physical memory or reduce the memory usage of Redis.

3.4 Memory Usage

The memory usage of the redis instance exceeds the maximum available memory, and the operating system will start exchanging memory and swap space.

Ways to avoid memory swapping:

  • Select the Redis instance to install based on cache data size
  • Use Hash data structure storage as much as possible
  • Set the expiration time of the key

Recover key within 3.5

Ensure reasonable allocation of redis’ limited memory resources.

When the set maximum threshold is reached, a key recycling strategy needs to be selected. By default, the recycling strategy prohibits deletion. Modify the maxmemory-policy attribute value in the configuration file:

vim /etc/redis/6379.conf
 
--598--
maxmemory-policy noenviction //Modify the max-memory-policy attribute value in the configuration file
volatile-lru: Use the LRU algorithm to eliminate data from a data collection with an expiration time set
volatile-ttl: Select data that is about to expire from the data collection with an expiration time set and eliminate it.
volatile-random: Randomly select data for elimination from the data set with an expiration time set
allkeys-lru: Use the LRU algorithm to eliminate data from all data sets
allkeys-random: arbitrarily select data to eliminate from the data collection
noenviction: disable obsolete data

4. Summary

Redis high availability mainly includes two methods: master-slave replication and Redis cluster. By dispersing data to multiple nodes, data backup and read-write separation are realized, which improves system availability and performance. Choosing a high-availability solution that suits your business scenario can ensure the reliability and stability of Redis data. Redis persistence mainly includes two methods: RDB and AOF. RDB saves Redis data to the hard disk in the form of snapshots, which is suitable for scenarios where the amount of data is large and requires regular backup. AOF writes Redis operation records to files, which can achieve more accurate data recovery. Choosing a persistence solution that suits your business scenario can ensure the reliability and stability of Redis data.