Redis high availability solution: Redis cluster, practical integration with Spring Cloud

A collection of columns that you can save for emergencies

Spring Cloud practical column: https://blog.csdn.net/superdangbo/category_9270827.html

Python practical column: https://blog.csdn.net/superdangbo/category_9271194.html

Logback detailed explanation column: https://blog.csdn.net/superdangbo/category_9271502.html

tensorflow column: https://blog.csdn.net/superdangbo/category_8691332.html

Redis column: https://blog.csdn.net/superdangbo/category_9950790.html

Spring Cloud actual combat:

Spring Cloud Practical Combat | Decrypting the underlying principles of Feign, including practical source code

Spring Cloud Practical Combat | Decrypting the underlying principles of load balancing Ribbon, including practical source code

1024 Programmers Day special article:

1024 Programmers Carnival Special | ELK + collaborative filtering algorithm builds a personalized recommendation engine to intelligently realize “thousands of people, thousands of faces”

1024 Programmer’s Day Special | Deciphering Spring Cloud Hystrix Meltdown to Improve System Availability and Fault Tolerance

1024 Programmer’s Day Special | ELK + user portraits build a personalized recommendation engine to intelligently realize “thousands of people, thousands of faces”

1024 Programmer’s Day Special | OKR VS KPI, who is more suitable?

1024 Programmers Day Special | Spring Boot Practical MongoDB Sharding or Replica Set Operation

Spring practical series of articles:

Spring Practical | Spring AOP Core Tips – Sunflower Collection

Spring Practice | The secret that Spring IOC cannot tell?

National Day and Mid-Autumn Festival special series of articles:

National Day and Mid-Autumn Festival Special (8) How to use JPA in Spring Boot projects

National Day and Mid-Autumn Festival Special (7) 20 Common Programming Interview Questions for Java Software Engineers

National Day and Mid-Autumn Festival Special (6) 30 Common Treasure Programming Interview Questions for College Students

National Day and Mid-Autumn Festival Special (5) How to performance tune MySQL? Next article

National Day and Mid-Autumn Festival Special (4) How to performance tune MySQL? Previous article

National Day and Mid-Autumn Festival Special (3) Use Generative Adversarial Network (GAN) to generate paintings with a festive atmosphere, implemented by the deep learning frameworks TensorFlow and Keras

National Day and Mid-Autumn Festival Special (2) Romantic Blessings Using Generative Adversarial Networks (GAN) to generate paintings with a festive atmosphere

National Day and Mid-Autumn Festival Special (1) Romantic Blessing Method Use Recurrent Neural Network (RNN) or Long Short-Term Memory Network (LSTM) to generate blessing poems

1. Core principles of Redis cluster

The core principles of Redis cluster mainly include data sharding, node roles, distributed hash tables, data replication, failover and recovery, etc. These core principles and related core code will be briefly analyzed below.

Data sharding:
Redis cluster achieves distributed storage of data by distributing data to multiple nodes based on the hash value of the key. The core code is as follows:

// RedisClusterClient.c
int RedisClusterClient::clusterCreate(const std::string & amp;node_ip, uint16_t node_port) {
    // ...
    int ret = redisClusterCreate(node_ip.c_str(), node_port);
    if (ret == REDIS_CLUSTER_OK) {
        //Add to cluster
        redisClusterAddNode(node_ip.c_str(), node_port);
    }
    return ret;
}

Node role:
The nodes in the Redis cluster are divided into three roles: master node (Master), slave node (Slave) and idle node (Idle). The core code is as follows:

// RedisClusterClient.c
int RedisClusterClient::clusterSetNodeRole(const std::string & amp;node_ip, uint16_t node_port, RedisClusterNodeRole role) {
    // ...
    return redisClusterSetNodeRole(node_ip.c_str(), node_port, role);
}

Distributed hash table:
Redis cluster uses distributed hash tables (DHT) to store and manage data. DHT can ensure distributed storage and consistency of data between different nodes. The core code is as follows:

// RedisClusterClient.c
int RedisClusterClient::clusterSet(const std::string & amp;key, const std::string & amp;value, uint32_t expiration) {
    // ...
    return redisClusterSet(key.c_str(), value.c_str(), expiration);
}

Data copy:
Redis cluster uses a master-slave replication mechanism to ensure data synchronization between different nodes. The core code is as follows:

// RedisClusterClient.c
int RedisClusterClient::clusterReplicaSync(const std::string &master_ip, uint16_t master_port, const std::string &slave_ip, uint16_t slave_port) {
    // ...
    return redisClusterReplicaSync(master_ip.c_str(), master_port, slave_ip.c_str(), slave_port);
}

Failover and load balancing:
Redis cluster achieves failover and load balancing through cooperation between nodes. When a node fails, the hash slots it is responsible for are automatically transferred to other nodes. The core code is as follows:

// RedisClusterClient.c
int RedisClusterClient::clusterSlotMove(uint16_t slot, const std::string & amp;new_node_ip, uint16_t new_node_port) {
    // ...
    return redisClusterSlotMove(slot, new_node_ip.c_str(), new_node_port);
}

Decentralized communication:
Redis cluster uses the decentralized gossipsub protocol for inter-node communication. The core code is as follows:

// RedisClusterClient.c
int RedisClusterClient::clusterNodeAdd(const std::string & amp;ip, uint16_t port, RedisClusterNode *node) {
    // ...
    return redisClusterNodeAdd(ip.c_str(), port, node);
}

Automatic fault detection and recovery:
Redis cluster implements automatic fault detection and recovery through periodic heartbeat detection and failover strategies. The core code is as follows:

// RedisClusterClient.c
int RedisClusterClient::clusterHeartbeat(const std::string & amp;node_ip, uint16_t node_port) {
    // ...
    return redisClusterHeartbeat(node_ip.c_str(), node_port);
}

To sum up, the core principles and core code of Redis cluster involve data sharding, node roles, distributed hash tables, data replication, failover and recovery, decentralized communication, and automatic fault detection and recovery.

2. Redis cluster construction and verification

The establishment of a Redis cluster is divided into the following steps:

Prepare environment
Make sure your system has Redis installed and check if the gcc version is above 5.3. If requirements are not met, upgrade gcc. The following is the command to upgrade gcc:

yum -y install centos-release-sclyum -y install devtoolset-9-gcc devtoolset-9-gcc-c devtoolset-9-binutils
scl enable devtoolset-9 bash

Download Redis source code
Visit the Redis official website (https://redis.io/download) to download the latest version of the source code. Unzip the downloaded file:

tar -xzf redis-5.0.2.tar.gz

Compile Redis
Enter the decompressed directory and compile Redis:

cd redis-5.0.2
make
make install

Configure Redis nodes
Before setting up a Redis cluster, you need to configure a configuration file for each Redis node. In the configuration file (redis.conf) of each node, modify the following parameters:

Turn on cluster mode:

cluster-enabled yes

Set the cluster node address:

cluster-node-timeout 5000

Set the cluster initialization password:

cluster-initialize-password your_password

Start the Redis node
Start the Redis service on each node:

redis-server /path/to/your/redis.conf

Create a Redis cluster
On the first Redis node, create a Redis cluster:

redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 --cluster-replicas 1

At this point, you will see the following output, indicating that the Redis cluster is successfully created:

127.0.0.1:7000 -> 0 nodes
127.0.0.1:7001 -> 1 nodes
127.0.0.1:7002 -> 2 nodes
[OK] All nodes agree about slots configuration.

Verify Redis cluster
Use the following command to verify that the Redis cluster is functioning properly:

redis-cli --cluster check 127.0.0.1:7000

If the following content is output, the cluster is running normally:

127.0.0.1:7000 # node-id
  -node-ip 127.0.0.1
    - slot0-127.0.0.1:7000
    - slot1-127.0.0.1:7001
    - slot2-127.0.0.1:7002
    - slot16384-127.0.0.1:7000
  - pong 0 127.0.0.1:7001 127.0.0.1:7002

At this point, the Redis cluster has been successfully established. You can use the Redis client to connect to the cluster and perform related operations.

3. Redis cluster and Spring Cloud project integration

Integrating Redis cluster with Spring Cloud can improve the scalability, stability and performance of the system. The integration process mainly involves the following aspects:

Configure the Redis cluster:
In the Spring Cloud project, you need to configure the relevant information of the Redis cluster, such as node address, port, password, etc. Configuration file example:

spring:
  redis:
    cluster:
      nodes:
        -ip: 192.168.1.1
          port: 7000
        -ip: 192.168.1.2
          port: 7001
        -ip: 192.168.1.3
          port: 7002
      password: your_password

Integrate Spring Data Redis:
In the Spring Cloud project, the Spring Data Redis component is introduced to facilitate the use of a consistent API to operate the Redis cluster.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>

Create a Redis client bean:
In the Spring Cloud project, create a Redis client bean to encapsulate the connection and operation of the Redis cluster.

@Configuration
public class RedisConfig {<!-- -->
    @Bean
    public RedisTemplate<String, Object> redisTemplate(RedisConnectionFactory redisConnectionFactory) {<!-- -->
        RedisTemplate<String, Object> template = new RedisTemplate<>();
        template.setConnectionFactory(redisConnectionFactory);
        return template;
    }
}

To operate a Redis cluster:
In the Spring Cloud project, use the API provided by Spring Data Redis to operate the Redis cluster.

@Service
public class RedisService {<!-- -->
    @Autowired
    private RedisTemplate<String, Object> redisTemplate;
    public String get(String key) {<!-- -->
        return redisTemplate.opsForValue().get(key);
    }
    public void set(String key, Object value) {<!-- -->
        redisTemplate.opsForValue().set(key, value);
    }
    // Other operations...
}

Using Redis cluster:
Other components in the Spring Cloud project (such as Controller, Service, etc.) use the Redis cluster by injecting the RedisService Bean.

@RestController
public class SomeController {<!-- -->
    @Autowired
    private RedisService redisService;
    @GetMapping("/get")
    public String get(String key) {<!-- -->
        return redisService.get(key);
    }
    @PostMapping("/set")
    public void set(String key, Object value) {<!-- -->
        redisService.set(key, value);
    }
}

In short, during the integration process of Redis cluster and Spring Cloud, you need to configure Redis cluster information, introduce Spring Data Redis components, create Redis client beans, operate Redis cluster, and use Redis cluster in other components. Through these steps, you can make full use of the high performance, scalability and other features of the Redis cluster to improve the overall performance of the Spring Cloud project.

4. Notes on Redis cluster

When building a Redis cluster, you need to pay attention to the following points:

Node selection: Make sure to select nodes with sufficient memory, CPU capabilities, and stable network connections to build the Redis cluster. At the same time, plan the number of nodes based on actual needs so that you can flexibly respond when expanding or adjusting the cluster size in the future.
Configuration file: In the configuration file, set the cluster-require-full-coverage parameter. This parameter is used to control whether the Redis cluster needs to cover all 16384 slots to provide external services. It is recommended to set it to no to avoid the entire cluster being unable to provide services due to individual slot exceptions.
Cluster status monitoring: Regularly check the Redis cluster status and use the redis-cli command to view cluster information, including cluster status, node information, memory, CPU usage, etc. Ensure that the cluster is running normally and identify and resolve potential problems in a timely manner.
Network connectivity: Ensure that the network connectivity between nodes in the Redis cluster is normal to avoid connection failures. This can be checked by testing the connection, viewing network logs, etc.
Data distribution: When building a Redis cluster, attention should be paid to reasonably allocating data to each node. Avoid overloading a node, causing performance bottlenecks or other problems.
Configure master-slave replication: To ensure data security and high availability, a master-slave replication relationship should be configured. When the master node fails, the slave node can replace the master node to ensure that services are not affected.
Load balancing: Use load balancing strategies, such as polling, least connection and other algorithms provided by Redis Cluster, to reasonably distribute client requests to each node to improve the overall performance of the cluster.
Security protection: Strengthen the security measures of the Redis cluster, such as setting passwords, restricting illegal access, monitoring abnormal behaviors, etc.
Monitoring and alarming: Deploy the Redis cluster monitoring system to monitor cluster performance indicators in real time, such as memory usage, CPU usage, etc. When the monitoring indicators exceed the preset threshold, an alarm is automatically triggered to remind operation and maintenance personnel to handle it in a timely manner.
Regular maintenance: Perform regular maintenance on the Redis cluster, including checking node status, optimizing data distribution, upgrading software versions, etc. This helps ensure long-term stable operation of the cluster.
In short, when building and operating a Redis cluster, you should pay attention to the above points to ensure the high availability, performance, and security of the cluster. At the same time, continue to learn and master the new features and technical trends of Redis in order to better apply it to actual projects.

5. Common problems and solutions for Redis clusters

In the actual application process of Redis cluster, you may encounter some common problems. Here are some typical problems and their solutions:

Memory usage is too high:
Problem description: Redis is an in-memory database. When the amount of data is too large, the problem of excessive memory usage is prone to occur.
solution:

Use RDB or AOF persistence mechanism to save some data to disk to reduce the memory used by Redis.
Configure the maximum memory limit of Redis. When Redis memory usage exceeds the limit, old data will be automatically deleted.

Cache avalanche:
Problem description: When the cached data in Redis fails at the same time, causing a large number of requests to fall directly on the back-end database, causing excessive pressure on the database or even crashing, it is called a cache avalanche.
solution:

Use multi-level cache, such as adding local cache, CDN, etc., to reduce Redis cache pressure.
Randomize the expiration time of cached data to prevent a large number of caches from invalidating at the same time.
Increase the number of Redis clusters and distribute cached data on different nodes to reduce the pressure on a single node.

Database pressure is too high:
Problem description: When the number of Redis requests is too large, it will cause excessive pressure on the database.
solution:

Optimize the Redis cluster architecture, such as adding nodes, adjusting data distribution, etc.
Optimize database query performance, such as adjusting indexes, optimizing SQL statements, etc.
Separate reading and writing of the database to reduce the pressure on Redis.

Node failure:
Problem description: A node failure in the Redis cluster may cause service interruption of the entire cluster.
solution:

Configure master-slave replication. When the master node fails, the slave node can replace the master node to ensure that services are not affected.
Regularly check node status to detect and resolve faulty nodes in a timely manner.

Network failure:
Problem description: The network connectivity between nodes in the Redis cluster is poor, which may cause data transmission to be affected.
solution:

Ensure that the network connectivity between nodes in the Redis cluster is good, which can be checked by testing the connection, viewing network logs, etc.
Deploy a network monitoring system to monitor network status in real time and discover and resolve network faults.

Data consistency:
Problem description: In the Redis cluster, data consistency problems may occur, such as the data of a node being out of sync with other nodes.
solution:

Ensure that the master-slave replication relationship in the Redis cluster is normal and check the synchronization progress regularly.
Configure the failover and automatic reconfiguration functions of the Redis cluster to ensure that data can be synchronized in time when a node fails.
The following is a simple Redis cluster configuration example showing how to set up a master-slave replication relationship:

# Configure slave nodes for master node
redis-cli --cluster create 127.0.0.1:7000 --cluster add-node 127.0.0.1:7001 7000
# Configure the slave node as the master node
redis-cli --cluster promote 127.0.0.1:7001

In actual applications, the configuration and strategy of the Redis cluster need to be adjusted according to specific scenarios and needs. At the same time, continue to learn and master the new features and technical trends of Redis in order to better apply it to actual projects.

Directory

1. Core principles of Redis cluster

2. Redis cluster construction and verification

3. Redis cluster and Spring Cloud project integration

4. Notes on Redis cluster

5. Common problems and solutions for Redis clusters