Redis sharding cluster construction and its principle

1. Redis fragmentation cluster
- 1.1. Build a shard cluster
- 1.2. Prepare instance and configuration
- 1.3. Startup
- 1.4. Create a cluster
- 1.5. Testing
2. Hash slot principle
- 2.1. Slot principle
- 2.2. Summary
3. Cluster scaling
- 3.1. Demand analysis
- 3.2. Create a new redis instance
- 3.3. Add new nodes to redis
- 3.4. Transfer slot
4. Failover
- 4.1. Automatic Failover
- 4.2. Manual failover
5. RedisTemplate access to fragmented cluster

1. Redis Fragmentation Cluster

1.1. Build a shard cluster

Master-slave and sentry can solve the problem of high availability and high concurrent reading. But there are still two unresolved issues:

Mass data storage problem
The problem of high concurrent writing

Using fragmented clusters can solve the above problems, as shown in the figure:

Sharded cluster features:

There are multiple masters in the cluster, and each master saves different data
Each master can have multiple slave nodes
The master monitors each other’s health status through ping
Client requests can access any node in the cluster and will eventually be forwarded to the correct node

Sharded clusters require a large number of nodes. Here we build a minimal sharded cluster, including 3 master nodes, and each master contains a slave node. The structure is as follows:

Here we will open 6 redis instances in the same virtual machine to simulate a fragmented cluster. The information is as follows:

IP	PORT	Role
192.168.150.101	7001	master
192.168.150.101	7002	master
192.168.150.101	7003	master
192.168.150.101	8001	slave
192.168.150.101	8002	slave
192.168.150.101	8003	slave

1.2. Prepare instance and configuration

Delete the previous directories 7001, 7002, and 7003, and recreate the directories 7001, 7002, 7003, 8001, 8002, and 8003:

# Enter the /tmp directory
cd /tmp
# Delete old ones to avoid configuration interference
rm -rf 7001 7002 7003
# Create a directory
mkdir 7001 7002 7003 8001 8002 8003

Prepare a new redis.conf file under /tmp with the following content:

port 6379
# Enable the cluster function
cluster-enabled yes
# The name of the configuration file of the cluster, we do not need to create it, it is maintained by redis itself
cluster-config-file /tmp/6379/nodes.conf
# Timeout for node heartbeat failure
cluster-node-timeout 5000
# Persistence file storage directory
dir /tmp/6379
# bind address
bind 0.0.0.0
# Let redis run in the background
daemonize yes
# registered instance ip
replica-announce-ip 192.168.150.101
# protected mode
protected-mode no
# number of databases
databases 1
# log
logfile /tmp/6379/run.log

Copy this file into each directory:

# Enter the /tmp directory
cd /tmp
# Execute the copy
echo 7001 7002 7003 8001 8002 8003 | xargs -t -n 1 cp redis.conf

Modify redis.conf in each directory, and modify 6379 to be consistent with the directory:

# Enter the /tmp directory
cd /tmp
# Modify the configuration file
printf '%s\
' 7001 7002 7003 8001 8002 8003 | xargs -I{} -t sed -i 's/6379/{}/g' {}/redis.conf

1.3. Startup

Since the background startup mode has been configured, the service can be started directly:

# Enter the /tmp directory
cd /tmp
# Start all services with one click
printf '%s\
' 7001 7002 7003 8001 8002 8003 | xargs -I{} -t redis-server {}/redis.conf

View status via ps:

ps -ef | grep redis

Discovery services have been started normally:

If you want to close all processes, you can execute the command:

ps -ef | grep redis | awk '{print $2}' | xargs kill

or (recommended this way):

printf '%s\
' 7001 7002 7003 8001 8002 8003 | xargs -I{} -t redis-cli -p {} shutdown

1.4. Create a cluster

Although the service is started, each service is currently independent without any association.

We need to execute commands to create a cluster. It is troublesome to create a cluster before Redis5.0. After 5.0, cluster management commands are integrated into redis-cli.

1) Before Redis5.0

Cluster commands before Redis 5.0 were implemented using src/redis-trib.rb under the redis installation package. Because redis-trib.rb is written in ruby language, it needs to install ruby environment.

# install dependencies
yum -y install zlib ruby rubygems
gem install redis

Then use the command to manage the cluster:

# Enter the src directory of redis
cd /tmp/redis-6.2.4/src
# Create a cluster
./redis-trib.rb create --replicas 1 192.168.150.101:7001 192.168.150.101:7002 192.168.150.101:7003 192.168.150.101:8001 192.168.150.101: 8002 192.168.150.101:8003

2) After Redis5.0

We are using Redis6.2.4 version, cluster management and integration into redis-cli, the format is as follows:

redis-cli --cluster create --cluster-replicas 1 192.168.150.101:7001 192.168.150.101:7002 192.168.150.101:7003 192.168.150.101:8001 192.168.150 .101:8002 192.168.150.101:8003

Command description:

redis-cli --cluster or ./redis-trib.rb: represents the cluster operation command
create: means to create a cluster
--replicas 1 or --cluster-replicas 1: Specifies that the number of replicas of each master in the cluster is 1, and the total number of nodes ÷ (replicas + 1) What you get is the number of masters. Therefore, the first n in the node list is the master, and the other nodes are all slave nodes, which are randomly assigned to different masters.

What it looks like after running:

Enter yes here, and the cluster will start to be created:

You can view the cluster status with the command:

redis-cli -p 7001 cluster nodes

1.5. Test

Try to connect to node 7001 and store a piece of data:

# connection
redis-cli -p 7001
# Storing data
set num 123
# read data
get num
# store again
set a 1

The result is tragic:

This is because the a key has been set in other nodes

During cluster operation, you need to add the -c parameter to redis-cli:

redis-cli -c -p 7001

This time it works:

2. Hash slot principle

2.1. Slot principle

Redis will map each master node to a total of 16384 slots (hash slots) ranging from 0 to 16383, which can be seen when viewing the cluster information:

Data keys are not bound to nodes, but to slots. Redis will calculate the slot value based on the effective part of the key, in two cases:

The key contains “{}”, and “{}” contains at least 1 character, and the part in “{}” is a valid part
The key does not contain “{}”, the entire key is a valid part

For example: if the key is num, then calculate according to num, if it is {zqd}num, then calculate according to zqd. The calculation method is to use the CRC16 algorithm to obtain a hash value, and then take the remainder of 16384, and the result obtained is the slot value.

As shown in the figure, when set a 1 is executed on node 7001, a hash operation is performed on a, and the remainder of 16384 is obtained, and the result is 15495, so it needs to be stored in node 103.

After arriving at 7003, when executing get num, perform a hash operation on num, take the remainder of 16384, and the result is 2765, so you need to switch to 7001 node

2.2. Summary

How does Redis determine which instance a key should be in?

Allocate 16384 slots to different instances
Calculate the hash value according to the effective part of the key, and take the remainder of 16384
The remainder is used as the slot, just find the instance where the slot is located

How to save the same type of data in the same Redis instance?

This type of data uses the same effective part, for example, keys are all prefixed with {typeId}

3. Cluster scaling

redis-cli –cluster provides many commands to operate the cluster, which can be viewed in the following ways:

For example, the command to add a node:

3.1. Demand analysis

Requirement: Add a new master node to the cluster and store num = 10 in it

Start a new redis instance with port 7004
Add 7004 to the previous cluster and act as a master node
Assign a slot to the 7004 node so that the key num can be stored in the 7004 instance

Two new functions are needed here:

Add a node to the cluster
Assign some slots to new slots

3.2. Create a new redis instance

Create a folder:

mkdir 7004

Copy the configuration file:

cp redis.conf /7004

Modify the configuration file:

sed /s/6379/7004/g 7004/redis.conf

start up

redis-server 7004/redis.conf

3.3. Add new nodes to redis

The syntax for adding a node is as follows:

Excuting an order:

redis-cli --cluster add-node 192.168.150.101:7004 192.168.150.101:7001

Check the cluster status with the command:

redis-cli -p 7001 cluster nodes

As shown in the figure, 7004 has joined the cluster and is a master node by default:

However, it can be seen that the number of slots of the 7004 node is 0, so no data can be stored on the 7004

3.4. Transfer slot

We want to store num to node 7004, so we need to see how many slots num has:

As shown above, the slot of num is 2765.

We can transfer the slots of 0~3000 from 7001 to 7004, the command format is as follows:

The specific commands are as follows:

establish connection:

Get the following feedback:

Ask how many slots to move, we plan to be 3000:

Here comes the new problem:

Which node to receive these slots? ?

Obviously it is 7004, so what is the id of node 7004?

Copy this id, and then copy it to the console just now:

Ask here, where did your slot move from?

all: represents all, that is, each of the three nodes transfers a part
Specific id: the id of the target node
done: no more

Here we want to get from 7001, so fill in the id of 7001:

Once filled, click done, and the slot transfer is ready:

Are you sure you want to transfer? Enter yes:

Then, view the results with the command:

can be seen:

The goal is achieved.

4. Failover

The initial state of the cluster is as follows:

Among them, 7001, 7002, and 7003 are masters, and we plan to shut down 7002.

4.1. Automatic failover

What happens when a master in the cluster goes down?

Stop a redis instance directly, such as 7002:

redis-cli -p 7002 shutdown

1) First, the instance loses connection with other instances

2) Then there is a suspected downtime:

3) Finally, it is determined to go offline and automatically promote a slave to the new master:

4) When the 7002 starts up again, it will become a slave node:

4.2. Manual failover

Using the cluster failover command, you can manually shut down a master in the cluster and switch to the slave node that executes the cluster failover command to realize data migration without awareness. The process is as follows:

This failover command can specify three modes:

Default: The default process, as shown in Figure 1~6 steps
force: omits the consistency check of offset
takeover: Execute step 5 directly, ignoring data consistency, master status and other master opinions

Case requirements: Perform manual failover on the slave node 7002 to regain the status of master

Proceed as follows:

1) Use redis-cli to connect to node 7002

2) Execute the cluster failover command

As shown in the picture:

Effect:

5. RedisTemplate access to shard cluster

The underlying layer of RedisTemplate also implements the support of fragmented clusters based on lettuce, and the steps used are basically the same as the sentinel mode:

1) Introduce the starter dependency of redis

2) Configure the shard cluster address

3) Configure read-write separation

Compared with sentinel mode, only the configuration of fragmented clusters is slightly different, as follows:

spring:
  redis:
    cluster:
      nodes:
        - 192.168.150.101:7001
        - 192.168.150.101:7002
        - 192.168.150.101:7003
        - 192.168.150.101:8001
        - 192.168.150.101:8002
        - 192.168.150.101:8003