Redis persistent AOF and RDB

Why does Redis need persistence?

When thinking about this problem, I think we can think about it in reverse, assuming that Redis does not support persistence, what will be the consequences.
First of all, Redis is a memory-based database. Once the Redis service goes down or is restarted, the data in Redis will be lost immediately. At this time, all data requests will reach the back-end database, causing huge pressure on the database, resulting in a decrease in performance. Even the back-end database is down, causing system paralysis.

Redis supports two persistence solutions: AOF log and RDB snapshot.

AOF log

Unlike MySql, Redis is a “write-after” log. Redis executes the command first, writes the data into the memory, and then records the log. The log records every command received by Redis. These commands are saved in text form.

Why use post-write logging

  • Avoid checking overhead
  • Will not block the current write operation

Disadvantages of post-write log

  • If the computer crashes before writing the log, data will be lost.
  • The main thread is under great pressure to write to the disk, causing disk writing to be slow and affecting subsequent operations.

Enable AOF log

AOF is turned off by default. You can enable AOF logs in the configuration file redis.conf

# The appendonly parameter enables AOF persistence
appendonly no
#The file name of AOF persistence, the default is appendonly.aof
appendfilename "appendonly.aof"

AOF implementation

Redis is a memory-based database. After each write operation, if you want to write the AOF log, you need to switch from the kernel mode to the user mode and write to the disk. Therefore, how to implement AOF is a key factor affecting performance.
AOF log records each write command of Redis. The steps are divided into: command append (append), file writing (write) and file synchronization (sync).

  • The command appends:
    • When the AOF persistence function is turned on, after the server executes a write command, it will append the executed write command to the server’s aof_buf buffer in protocol format.
  • File writing and synchronization:
    • Regarding when to write the contents of the aof_buf buffer to the AOF file, Redis provides three writeback strategies:
      Always : Synchronous writeback: After each write command is executed, the log will be written back to the disk synchronously;
      Everysec: Write back every second: After each write command is executed, the log is first written to the memory buffer of the AOF file, and the contents of the buffer are written to the disk every second;
      No: Operating system controlled writeback: After each write command is executed, the log is first written to the memory buffer of the AOF file, and the operating system decides when to write the buffer contents back to the disk.

AOF rewriting

As operations on Redis increase, the AOF file will become larger and larger, and there will be many redundant or commands that have no impact on the current data (such as deleting the data after writing the modification), so The command will only occupy space and affect Redis data recovery, so AOF rewriting is to create a new AOF file to replace the existing AOF file. The data generated after the execution of the two file commands is the same, but the new AOF file has no redundancy. Order.

AOF rewriting mechanism

AOF rewriting is caused by the main thread forking out of the background bgrewriteaof sub-process. The fork will copy the memory of the main thread to the bgrewriteaof sub-process, which contains the latest data of the database. Then the bgrewriteaof child process can write the copied data into operations one by one without affecting the main thread, and include it in the rewrite diary.

Therefore, when aof is rewritten, the main thread will be blocked when forking the process.

When will the AOF log be rewritten

There are two configuration items that control the triggering of AOF rewriting:
auto-aof-rewrite-min-size: Indicates the minimum size of the file when running AOF rewrite, the default is 64MB.
auto-aof-rewrite-percentage: This value is calculated as the difference between the current aof file size and the aof file size after the last rewrite, divided by the aof file size after the last rewrite. That is, the incremental size of the current AOF file compared to the last rewritten AOF file, and the ratio of the AOF file size after the last rewrite.

When rewriting the entire log process, where will the main thread be blocked?

  1. When forking a child process, the virtual page table needs to be copied, which will block the main thread.
  2. When the main process writes bigkey, the operating system will create a copy of the page and copy the original data, which will block the main thread.
  3. After the sub-process rewrite log is completed, the main thread may be blocked when the main process appends the AOF rewrite buffer.

Why AOF rewriting does not reuse the original AOF log

  1. Writing the same file between parent and child processes will cause competition problems and affect the performance of the parent process.
  2. If the AOF rewrite process fails, it is equivalent to contaminating the original AOF file and cannot be used to recover data.

RDB Snapshot

RDB snapshot is a solution to save Redis data at a certain moment to the disk. Since it is data at a certain moment, the RDB data may be earlier than Redis data at this moment.
After the RDB snapshot, a binary file is generated, which stores the data at the time of the Redis snapshot. The binary file therefore restores data from the RDB file with Redis much faster than from the AOF log file. but not readable

RDB triggering method

There are two triggering methods for RDB snapshots: manual triggering and automatic triggering.

Manual trigger

There are two commands for manual triggering: save and bgsave

  • save command
    • The save command will block the current Redis service until the snapshot of the RDB is completed. Therefore, when the amount of data is large, it may cause long-term blocking. It is not recommended to be used in an online environment.
  • bgsave command
    • The Redis process performs a fork operation to create a child process. The RDB persistence process is responsible for the child process and ends automatically after completion. Blocking only occurs in the fork phase, usually for a short time

The specific process of bgsave is as follows:

  1. The redis client executes the bgsave command or automatically triggers the bgsave command;
  2. The main process determines whether there is currently an executing child process. If it exists, the main process returns directly;
  3. If there is no executing child process, then fork a new child process to persist the data. The fork process is blocked. After the fork operation is completed, the main process can perform other operations;
  4. The child process first writes the data into the temporary rdb file, and then atomically replaces the old rdb file after the snapshot data is written;
  5. At the same time, a signal is sent to the main process to notify the main process that rdb persistence is completed, and the main process updates relevant statistical information (rdb_* related options under info Persitence).

Automatically triggered

It will be automatically triggered in the following 4 situations:

  1. Save m n is configured in redis.conf, that is, when there are n modifications within m seconds, bgsave is automatically triggered to generate an rdb file;
  2. During master-slave replication, when the slave node wants to fully copy from the master node, the bgsave operation will also be triggered, and the snapshot at that time will be generated and sent to the slave node;
  3. Executing the debug reload command to reload redis will also trigger the bgsave operation;
  4. By default, when executing the shutdown command, if aof persistence is not enabled, the bgsave operation will also be triggered.

redis.conf configuration file
There is a default save m n configuration in the redis.conf file. The configuration of save m n means: when there are n keys within m seconds When the information changes, the bgsave operation is performed. Although the configuration is written as save, the bgsave operation is performed to perform RDB snapshot;

When operations modify Redis data during bgsave, how to ensure data consistency?

Every time RDB forks a process to take a snapshot, because the amount of data is often relatively large, it may last for a long time. During this period, the Redis service is still running, and there are frequent write or modification operations. So what should I do at this time? Ensure the consistency of RDB file data.

In fact, the idea of RDB operation is Copy-on-Write. When data changes during the snapshot, the fork sub-process will create a copy of the changed data and store it in another area of the memory. When the snapshot ends, the sub-process Then write the copy to the RDB file.

When the service crashes during bgsave, will the data be lost?

If the service crashes during bgsave, then the RDB snapshot will fail this time, but this will not affect the last snapshot file. The RDB snapshot does not directly write the RDB file that overwrites the previous snapshot, but writes a new one first. Temporary data files, the original files will be replaced only after the RDB snapshot is successfully completed. Therefore, when the service crashes during bgsave, at most the data from the last RDB to the crash will be lost. Data can be recovered from the last RDB file

RDB and AOF hybrid method

Redis 4.0 proposes a method of mixing AOF logs and memory snapshots. Simply put, memory snapshots are executed at a certain frequency. Between two snapshots, AOF logs are used to record all command operations during this period.

What is the hybrid method of RDB and AOF?

It is known that RDB snapshots take a long time to execute and are not suitable for real-time persistence. However, RDB files are small and data recovery is fast. AOF can be persisted in real time, with less data lost, but the log file is large.

Example: After a full snapshot, use the AOF log to record data. After several modifications, take a full snapshot. At this time, the snapshot already has all the data, so the AOF file can be no longer used and can be deleted. As a result, not only are the number of full snapshots reduced, but the AOF file is also smaller, which reduces AOF rewriting and makes file recovery faster. You can not only enjoy the benefits of fast recovery of RDB files, but also enjoy the simple advantage of AOF only recording operation commands, which is widely used in actual environments.

Recover data

  1. When redis restarts, it determines whether aof is enabled. If aof is enabled, the aof file will be loaded first;
  2. If aof exists, then load the aof file. If the loading is successful, redis will restart successfully. If the aof file fails to load, a log will be printed indicating that the startup failed. At this time, you can repair the aof file and restart;
  3. If the aof file does not exist, then redis will load the rdb file instead. If the rdb file does not exist, redis will start successfully;
  4. If the rdb file exists, the rdb file will be loaded to restore the data. If the loading fails, a log will be printed to indicate that the startup failed. If the loading is successful, then redis restarts successfully and the rdb file is used to restore the data.

Reference article: https://pdai.tech/md/db/nosql-redis/db-redis-x-rdb-aof.html