I have encountered such a requirement: to use Redis to save 50 million key-value pairs, and each key-value pair is about 512B. In order to quickly deploy and provide external services, we use cloud hosts to run Redis instances. Then, the How to choose the memory capacity of the cloud host?
I roughly calculated that the memory space occupied by these key-value pairs is about 25GB (50 million*512B). So, at that time, the first solution I thought of was: choose a cloud host with 32GB memory to deploy Redis. Because 32GB of memory can save all data, and there is 7GB left to ensure the normal operation of the system. At the same time, I also use RDB to persist the data to ensure that the data can be recovered from the RDB after the Redis instance fails.
However, in the process of using it, I found that the response of Redis is sometimes very slow. Later, we used the INFO command to view the latest_fork_usec indicator value of Redis (indicating the time-consuming of the latest fork), and the results showed that the indicator value was extremely high, almost to the second level.
This has something to do with the persistence mechanism of Redis. When using RDB for persistence, Redis will fork subprocesses to complete. The time spent on fork operations is positively related to the amount of data in Redis, and fork will block the main thread during execution. The larger the amount of data, the longer the main thread will be blocked by the fork operation. Therefore, when using RDB to persist 25GB of data, the amount of data is large, and the child process running in the background blocks the main thread when the fork is created, which slows down the response of Redis.
It seems that the first option is obviously not feasible, we have to find other options. At this time, we noticed the Redis slice cluster. Although it is cumbersome to set up a slice cluster, it can save a large amount of data and has less impact on the blocking of the Redis main thread.
A slice cluster, also called a slice cluster, refers to starting multiple Redis instances to form a cluster, and then dividing the received data into multiple shares according to certain rules, each of which is saved by an instance. Going back to the scene we just saw, if we divide the 25GB data into 5 parts (of course, you don’t need to divide it evenly), and use 5 instances to save it, each instance only needs to save 5GB of data. As shown below:
Then, in a sliced cluster, when an instance generates an RDB for 5GB of data, the amount of data is much smaller, and the fork child process generally does not block the main thread for a long time. After using multiple instances to save data slices, we can not only save 25GB of data, but also avoid the sudden slowdown of response caused by the fork child process blocking the main thread.
In the actual application of Redis, with the expansion of users or business scale, it is usually unavoidable to save a large amount of data. The slice cluster is a very good solution. In this lesson, let’s learn about it.
How to save more data?
In the case just now, in order to save a large amount of data, we used two methods: large-memory cloud host and slice cluster. In fact, these two methods correspond to two solutions for Redis to deal with the increase in data volume: vertical expansion (scale up) and horizontal expansion (scale out).
-
Vertical expansion: upgrade the resource configuration of a single Redis instance, including increasing memory capacity, increasing disk capacity, and using higher-configured CPUs. As shown in the figure below, the original instance memory is 8GB, and the hard disk is 50GB.
-
After scaling up, the memory increases to 24GB and the disk increases to 150GB. Horizontal expansion: Horizontally increase the number of current Redis instances, as shown in the figure below, where one instance with 8GB memory and 50GB disk was originally used, and now three instances with the same configuration are used.
So, what are the advantages and disadvantages of these two methods?
First, the benefit of vertical scaling is that it is simple and straightforward to implement. However, this scheme also faces two potential problems.
The first problem is that when using RDB to persist data, if the amount of data increases, the required memory will also increase, and the main thread may block when forking the child process (such as the situation in the example just now). However, if you do not require persistent storage of Redis data, then vertical scaling would be a good choice.
However, at this time, you have to face the second problem: vertical expansion will be limited by hardware and cost. This is easy to understand. After all, it is easy to expand the memory from 32GB to 64GB. However, if you want to expand to 1TB, you will face limitations in hardware capacity and cost.
Scaling out is a more scalable solution than scaling up. This is because, if you want to save more data, if you adopt this solution, you only need to increase the number of Redis instances, without worrying about the hardware and cost limitations of a single instance. When facing the scale of millions or tens of millions of users, the scale-out Redis slice cluster will be a very good choice.
However, when only a single instance is used, it is very clear where the data is stored and where the client accesses it. However, the sliced cluster inevitably involves the distributed management of multiple instances. In order to use sliced clusters, we need to solve two major problems:
-
After the data is sliced, how is it distributed among multiple instances?
-
How does the client determine which instance the data it wants to access resides on?
Next, we will solve them one by one.
The corresponding distribution relationship between data slices and instances
In a sliced cluster, data needs to be distributed on different instances, so how do data and instances correspond? This is related to the Redis Cluster solution I will talk about next. However, we must first understand the connection and difference between sliced clusters and Redis Cluster.
In fact, sliced clusters are a general mechanism for storing large amounts of data, and this mechanism can have different implementations. Before Redis 3.0, the official did not provide specific solutions for sliced clusters. Starting from 3.0, the official provides a solution called Redis Cluster for implementing slice clusters. The corresponding rules for data and instances are stipulated in the Redis Cluster scheme.
Specifically, the Redis Cluster solution uses hash slots (Hash Slots, which I will directly call Slots) to handle the mapping relationship between data and instances. In the Redis Cluster scheme, a slice cluster has a total of 16384 hash slots. These hash slots are similar to data partitions, and each key-value pair is mapped to a hash slot according to its key.
The specific mapping process is divided into two steps: first, according to the key of the key-value pair, calculate a 16-bit value according to the CRC16 algorithm; then, use this 16-bit value to take the modulus of 16384 to obtain the modulus in the range of 0~16383, Each modulus represents a correspondingly numbered hash slot. Regarding the CRC16 algorithm, it is not the focus of this lesson, you can simply read the information in the link.
So, how are these hash slots mapped to specific Redis instances?
When we deploy the Redis Cluster solution, we can use the cluster create command to create a cluster. At this time, Redis will automatically distribute these slots evenly among the cluster instances. For example, if there are N instances in the cluster, then the number of slots on each instance is 16384/N.
Of course, we can also use the cluster meet command to manually establish connections between instances to form a cluster, and then use the cluster addslots command to specify the number of hash slots on each instance.
For example, assuming that the memory sizes of different Redis instances in the cluster are configured differently, if the hash slots are evenly divided among the instances, when saving the same number of key-value pairs, compared with instances with large memory, those with small memory The instance will have greater capacity pressure. In this case, you can use the cluster addslots command to manually allocate hash slots according to the resource configuration of different instances.
In order to facilitate your understanding, I will draw a schematic diagram to explain the mapping distribution of data, hash slots, and instances.
There are 3 instances in the slice cluster in the schematic diagram, and assuming that there are 5 hash slots, we can first manually allocate hash slots through the following command: instance 1 saves hash slot 0 and 1, instance 2 saves hash slot 2 and 3, instance 3 holds hash slot 4.
redis-cli -h 172.16.19.3 –p 6379 cluster addslots 0,1 redis-cli -h 172.16.19.4 –p 6379 cluster addslots 2,3 redis-cli -h 172.16.19.5 –p 6379 cluster addslots 4
During the running of the cluster, after calculating the CRC16 value of key1 and key2, the total number of hash slots 5 is taken modulo, and then can be mapped to the corresponding instance 1 and instance 3 according to the respective modulus results.
In addition, I will give you another reminder. When manually assigning hash slots, you need to allocate all 16384 slots, otherwise the Redis cluster will not work properly.
Well, through the hash slot, the slice cluster realizes the allocation of data to the hash slot, and then to the instance. However, even if the instance has the mapping information of the hash slot, how does the client know which instance the data to be accessed is on? Next, I’ll come and talk to you.
How does the client locate the data?
When locating key-value pair data, the hash slot it is in can be obtained by calculation, and this calculation can be performed when the client sends a request. However, to further locate the instance, it is also necessary to know which instance the hash slot is distributed on.
Generally speaking, after the client establishes a connection with the cluster instance, the instance will send the hash slot allocation information to the client. However, when the cluster is just created, each instance only knows which hash slots it is assigned to, but does not know the hash slot information owned by other instances.
So, why can the client get all the hash slot information when accessing any instance? This is because the Redis instance will send its own hash slot information to other instances connected to it to complete the diffusion of hash slot allocation information. When the instances are connected to each other, each instance has a mapping relationship of all hash slots.
After the client receives the hash slot information, it will cache the hash slot information locally. When a client requests a key-value pair, it first calculates the hash slot corresponding to the key, and then sends a request to the corresponding instance.
However, in a cluster, the correspondence between instances and hash slots is not static. There are two most common changes:
-
In the cluster, if instances are added or deleted, Redis needs to reallocate hash slots
-
For load balancing, Redis needs to redistribute the hash slots on all instances
At this time, instances can also obtain the latest hash slot allocation information by passing messages to each other, but the client cannot actively perceive these changes. This will lead to an inconsistency between the allocation information it caches and the latest allocation information, so what should I do?
The Redis Cluster scheme provides a redirection mechanism. The so-called “redirection” means that when the client sends data read and write operations to an instance, there is no corresponding data on the instance, and the client needs to send a new instance Send an operation command.
Then how does the client know the access address of the new instance during redirection? When the client sends an operation request of a key-value pair to an instance, if there is no hash slot for the key-value pair mapping on the instance, then the instance will return the following MOVED command response result to the client, This result contains the access address of the new instance.
GET hello:key (error) MOVED 13320 172.16.19.5:6379
Among them, the MOVED command indicates that the hash slot 13320 where the key-value pair requested by the client is located is actually on the instance 172.16.19.5. By returning the MOVED command, it is equivalent to telling the client the information of the new instance where the hash slot is located. In this way, the client can directly connect to 172.16.19.5 and send operation requests.
Let me draw a picture to illustrate how to use the MOVED redirection command. It can be seen that due to load balancing, the data in Slot 2 has been migrated from instance 2 to instance 3, but the client cache still records the information that “Slot 2 is in instance 2”, so the command will be sent to instance 2. Instance 2 returns a MOVED command to the client, and returns the latest location of Slot 2 (that is, on instance 3) to the client, and the client will send a request to instance 3 again, and at the same time update the local cache to set the Slot 2 The corresponding relationship with the instance is updated.
It should be noted that in the above figure, when the client sends a command to instance 2, all the data in slot 2 has been migrated to instance 3. In actual application, if there is a lot of data in Slot 2, a situation may occur: the client sends a request to instance 2, but at this time, only part of the data in Slot 2 is migrated to instance 3, and some data There is no migration. In the case where this migration is partially completed, the client will receive an ASK error message as follows:
GET hello:key (error) ASK 13320 172.16.19.5:6379
The ASK command in this result indicates that the hash slot 13320 where the key-value pair requested by the client is located is on the instance 172.16.19.5, but this hash slot is being migrated. At this point, the client needs to send an ASKING command to the instance of 172.16.19.5 first. The meaning of this command is to let this instance allow the execution of the command sent by the client next. Then, the client sends a GET command to this instance to read the data.
It seems a bit complicated, let me explain with the help of pictures.
In the figure below, Slot 2 is migrating from instance 2 to instance 3, key1 and key2 have been migrated, and key3 and key4 are still in instance 2. After the client requests key2 from instance 2, it will receive the ASK command returned by instance 2.
The ASK command has two meanings: first, it indicates that the Slot data is still being migrated; second, the ASK command returns the latest instance address of the data requested by the client to the client. At this time, the client needs to send the ASKING command to instance 3 , and then send the operation command.
Unlike the MOVED command, the ASK command does not update the hash slot allocation information cached by the client. Therefore, in the above figure, if the client requests the data in Slot2 again, it will still send the request to instance 2. That is to say, the function of the ASK command is only to allow the client to send a request to the new instance, unlike the MOVED command, which changes the local cache so that all subsequent commands are sent to the new instance.
Summary
Learned the advantages of slicing clusters in saving large amounts of data, as well as the hash slot-based data distribution mechanism and the method of client-side locating key-value pairs.
When dealing with data volume expansion, although the vertical expansion method of increasing memory is simple and straightforward, it will cause the database memory to be too large, resulting in slow performance. The Redis slice cluster provides a horizontal expansion mode, that is, multiple instances are used, and a certain number of hash slots are configured for each instance. Data can be mapped to the hash slots through the hash value of the key, and then dispersed through the hash slots. Save to a different instance. The advantage of this is that it has good scalability, no matter how much data there is, the slice cluster can handle it.
In addition, the increase or decrease of instances in the cluster, or the redistribution of data for load balancing, will lead to changes in the mapping relationship between hash slots and instances. When the client sends a request, it will receive an error message when the command is executed. Knowing the MOVED and ASK commands, you won’t have a headache for this kind of error.