[redis] The ssm project integrates redis, redis annotation caching and application scenarios, redis breakdown, penetration, and avalanche solutions

Table of Contents

1. Integrate redis

1 Introduction

1.1. redis (Remote Dictionary Server)

1.2. MySQL

1.3. Difference

2. Integration

2.1. Configuration

2.2. File configuration

2.3. Key generation rule method

2.4. Attention

2. redis annotation cache

1. @Cacheable annotation

2. @CachePut annotation

3. @CacheEvict annotation

4. Application scenarios

3. Redis breakdown and penetration avalanche

1. Cache Miss

2. Cache Penetration

3. Cache Avalanche

1. Integrate redis

1. Introduction

1.1, redis (Remote Dictionary Server)

  • Redis is a memory-based key-value storage system that stores data in memory, so it is very fast to read and write.
  • Redis supports a variety of data structures, such as strings, hashes, lists, sets, ordered sets, etc., which makes Redis suitable for various application scenarios, such as caching, message queues, counters, etc.
  • Redis has high availability and scalability, and supports master-slave replication and sharding to achieve data backup and load balancing.
  • Redis’s persistence methods include RDB (snapshot) and AOF (log append), which can persist data to disk to ensure data security.

1.2, MySQL

  • MySQL is a relational database management system that uses standard SQL language for data operations.
  • MySQL stores data on disk, so read and write speeds are slower than Redis.
  • MySQL supports transaction processing and complex queries, and is suitable for applications that need to process structured data, such as websites, e-commerce, etc.
  • MySQL has high stability and maturity, supports ACID features (atomicity, consistency, isolation, durability), which can ensure data integrity and consistency.

1.3, difference

  1. redis is a nosql database

  2. MySQL is a sql database

A further understanding:

  • Storage method: Redis stores data in memory, while MySQL stores data on disk.
  • Data structure: Redis supports a variety of data structures, and MySQL uses tables and relationships for data storage.
  • Read and write performance: Since Redis uses memory storage, the read and write speed is faster, while MySQL is slower.
  • Functional features: Redis is suitable for caching and real-time data processing, and MySQL is suitable for structured data storage and complex queries.
  • ACID features: MySQL supports transaction processing and ACID features, while Redis does not support transaction processing by default.
  • Persistence method: Redis can persist data to disk, and MySQL has multiple persistence methods, such as log files and replication.

2. Integration

2.1, configuration

Create an ssm project and add it to the pom file of the configuration file

<redis.version>2.9.0</redis.version>
<redis.spring.version>1.7.1.RELEASE</redis.spring.version>

<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>${redis.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-redis</artifactId>
<version>${redis.spring.version}</version>
</dependency>

2.2. File configuration

Write a configuration file redis.properties in our resources package.

redis.hostName: corresponding IP address
redis.port: corresponding port number
redis.password: Password of the corresponding redis connection

redis.hostName=localhsot
redis.port=6379
redis.password=123456
redis.timeout=10000
redis.maxIdle=300
redis.maxTotal=1000
redis.maxWaitMillis=1000
redis.minEvictableIdleTimeMillis=300000
redis.numTestsPerEvictionRun=1024
redis.timeBetweenEvictionRunsMillis=30000
redis.testOnBorrow=true
redis.testWhileIdle=true
redis.expiration=3600

spring-redis.xml contains

  1. Add registration
  2. Redis connection pool configuration: The corresponding value is the configuration in redis.properties
  3. Redis’s connection factory: Here the connection pool configuration redis is used.
  4. Configure serialization: There are serializers for string, json, and hash
  5. Configuration key generation
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:context="http://www.springframework.org/schema/context"
       xmlns:cache="http://www.springframework.org/schema/cache"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd
       http://www.springframework.org/schema/context
       http://www.springframework.org/schema/context/spring-context.xsd
       http://www.springframework.org/schema/cache
       http://www.springframework.org/schema/cache/spring-cache.xsd">

    <!-- 1. Introduce properties configuration file -->
    <!--<context:property-placeholder location="classpath:redis.properties" />-->

    <!-- 2. redis connection pool configuration -->
    <bean id="poolConfig" class="redis.clients.jedis.JedisPoolConfig">
        <!--Maximum idle number-->
        <property name="maxIdle" value="${redis.maxIdle}"/>
        <!--The maximum number of database connections in the connection pool -->
        <property name="maxTotal" value="${redis.maxTotal}"/>
        <!--Maximum waiting time to establish a connection-->
        <property name="maxWaitMillis" value="${redis.maxWaitMillis}"/>
        <!--Minimum idle time for evicted connections, default 1800000 milliseconds (30 minutes)-->
        <property name="minEvictableIdleTimeMillis" value="${redis.minEvictableIdleTimeMillis}"/>
        <!--The maximum number of evictions during each eviction check. If it is a negative number, it is: 1/abs(n), default 3-->
        <property name="numTestsPerEvictionRun" value="${redis.numTestsPerEvictionRun}"/>
        <!--The time interval for eviction scanning (milliseconds). If it is a negative number, the eviction thread will not run. The default is -1-->
        <property name="timeBetweenEvictionRunsMillis" value="${redis.timeBetweenEvictionRunsMillis}"/>
        <!--Whether to check before taking out the connection from the pool, if the check fails, remove the connection from the pool and try to take out another one-->
        <property name="testOnBorrow" value="${redis.testOnBorrow}"/>
        <!--Check validity when idle, default false -->
        <property name="testWhileIdle" value="${redis.testWhileIdle}"/>
    </bean>

    <!-- 3. redis connection factory -->
    <bean id="connectionFactory" class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory"
          destroy-method="destroy">
        <property name="poolConfig" ref="poolConfig"/>
        <!--IP address -->
        <property name="hostName" value="${redis.hostName}"/>
        <!--Port number -->
        <property name="port" value="${redis.port}"/>
        <!--If Redis is set with a password -->
        <property name="password" value="${redis.password}"/>
        <!--The client timeout unit is milliseconds -->
        <property name="timeout" value="${redis.timeout}"/>
    </bean>

    <!-- 4. redis operation template, use this object to operate redis
        In the hibernate course, hibernatetemplete is equivalent to session and specializes in operating databases.
    -->
    <bean id="redisTemplate" class="org.springframework.data.redis.core.RedisTemplate">
        <property name="connectionFactory" ref="connectionFactory"/>
        <!--If Serializer is not configured, String will be used by default when storing. If the User type is used to store, the error User can't cast to String will be prompted! ! -->
        <property name="keySerializer">
            <bean class="org.springframework.data.redis.serializer.StringRedisSerializer"/>
        </property>
        <property name="valueSerializer">
            <bean class="org.springframework.data.redis.serializer.GenericJackson2JsonRedisSerializer"/>
        </property>
        <property name="hashKeySerializer">
            <bean class="org.springframework.data.redis.serializer.StringRedisSerializer"/>
        </property>
        <property name="hashValueSerializer">
            <bean class="org.springframework.data.redis.serializer.GenericJackson2JsonRedisSerializer"/>
        </property>
        <!--Open transaction -->
        <property name="enableTransactionSupport" value="true"/>
    </bean>

    <!-- 5. Configure cache manager -->
    <bean id="redisCacheManager" class="org.springframework.data.redis.cache.RedisCacheManager">
        <constructor-arg name="redisOperations" ref="redisTemplate"/>
        <!--redis cache data expiration time unit seconds-->
        <property name="defaultExpiration" value="${redis.expiration}"/>
        <!--Whether to use cache prefix, related to cachePrefix-->
        <property name="usePrefix" value="true"/>
        <!--Configure cache prefix name-->
        <property name="cachePrefix">
            <bean class="org.springframework.data.redis.cache.DefaultRedisCachePrefix">
                <constructor-arg index="0" value="-cache-"/>
            </bean>
        </property>
    </bean>

    <!--6. Configure the generation rules for cache generated key names-->
    <bean id="cacheKeyGenerator" class="com.tgq.ssm.redis.CacheKeyGenerator"></bean>

    <!--7. Enable cache annotation function-->
    <cache:annotation-driven cache-manager="redisCacheManager" key-generator="cacheKeyGenerator"/>
</beans>

redis.properties and jdbc.properties will conflict when integrating with Spring; therefore, the place where the configuration file is introduced must be placed in applicationContext.xml

applicationContext.xml is configured as follows

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:context="http://www.springframework.org/schema/context" xmlns:tx="http://www.springframework.org/schema/tx"
       xmlns:aop="http://www.springframework.org/schema/aop"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx. xsd http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop.xsd">
    <!--1. Introduce external multi-file method -->
    <bean id="propertyConfigurer"
          class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
        <property name="ignoreResourceNotFound" value="true" />
        <property name="locations">
            <list>
                <value>classpath:jdbc.properties</value>
                <value>classpath:redis.properties</value>
            </list>
        </property>
    </bean>

<!-- As you continue to learn, you will learn more and more frameworks. You cannot configure all frameworks in the same preparation room, otherwise it will be inconvenient to manage -->
    <import resource="applicationContext-mybatis.xml"></import>
    <import resource="spring-redis.xml"></import>
    <import resource="applicationContext-shiro.xml"></import>
</beans>

2.3. Key generation rule method

The final configuration of spring-redis.xml explains the key configuration rules.

package com.tgq.ssm.redis;

import lombok.extern.slf4j.Slf4j;
import org.springframework.cache.interceptor.KeyGenerator;
import org.springframework.util.ClassUtils;

import java.lang.reflect.Array;
import java.lang.reflect.Method;

@Slf4j
public class CacheKeyGenerator implements KeyGenerator {
    // custom cache key
    public static final int NO_PARAM_KEY = 0;
    public static final int NULL_PARAM_KEY = 53;

    @Override
    public Object generate(Object target, Method method, Object... params) {
        StringBuilder key = new StringBuilder();
        key.append(target.getClass().getSimpleName()).append(".").append(method.getName()).append(":");
        if (params. length == 0) {
            key.append(NO_PARAM_KEY);
        } else {
            int count = 0;
            for (Object param : params) {
                if (0 != count) {//separate parameters with,
                    key.append(',');
                }
                if (param == null) {
                    key.append(NULL_PARAM_KEY);
                } else if (ClassUtils.isPrimitiveArray(param.getClass())) {
                    int length = Array.getLength(param);
                    for (int i = 0; i < length; i + + ) {
                        key.append(Array.get(param, i));
                        key.append(',');
                    }
                } else if (ClassUtils.isPrimitiveOrWrapper(param.getClass()) || param instanceof String) {
                    key.append(param);
                } else {//Java must rewrite hashCode and eqauls
                    key.append(param.hashCode());
                }
                count + + ;
            }
        }

        String finalKey = key.toString();
// IEDA needs to install the lombok plug-in
        log.debug("using cache key={}", finalKey);
        return finalKey;
    }
}

2.4, note

  1. If multiple configurations ending in .properties are registered in applicationContext.xml, then registration cannot be added in spring-*.xml.
  2. The configuration of resources must cover reading files ending in .properties.
  3. For the use of redisTemplate, please refer to jdbcTemplate, amqpTemplate, rabbitMQTemplate, etc.

2. redis annotation cache

1. @Cacheable annotation

Configured on a method or class, the function is: after this method is executed, first go to the cache to see if there is any data. If not, find it from the database, save a copy in the cache, and return the result.
The next time this method is executed, if the cache has not expired, it will first search in the cache. If there is any, it will be returned directly. If not, it will be searched from the database.

value: The name of the cache location, cannot be empty.
key:The cached key, which is empty by default, means using the parameter type and parameter value of the method as the key, and supports SpEL.
condition: Trigger condition. If the condition is met, it will be added to the cache. The default is empty, which means all will be added to the cache. It supports SpEL.

  • @Cacheable Test

But when we call the query method without using the @Cacheable annotation, we can see that the results of our query are two, and the database is queried twice.

When we use the @Cacheable annotation, we only query the database once. If we query again, the sql statement will not appear, and it has been cached in redis.

Test results: If there is data in redis, access redis; if there is no data, access MySQL;

  • Change the original key generation rules

After changing the original rules and running it again, our redis keys will be different.

condition: Cache only occurs when a certain value is greater than or less than a certain value.

2. @CachePut annotation

Similar to the update operation, that is, each time regardless of whether there is a result in the cache, the result is found from the database, the result is updated to the cache, and the result is returned.

value: The name of the cache, defined in the spring configuration file, at least one must be specified.
key: cached key, which can be empty. If specified, it should be written according to SpEL expression. If not specified, it will be combined according to all parameters of the method by default.
condition: cache condition, which can be empty, written in SpEL, returns true or false, and will only be cached if it is true.

When we use it to test, the result is: only deposit but no withdrawal

  • The difference between @Cacheable and @CachePut

The @Cacheable annotation indicates that the return value of the method can be cached. The next time you access the same method, the result will be obtained directly from the cache without executing the logic inside the method. This annotation is marked on the method, specifying the cache name (the default cache name can also be used), or the cache Key can be specified through parameters. If the corresponding Key already exists in the cache, the value in the cache will be returned directly; if the corresponding Key does not exist in the cache, the method will be executed and the return value of the method will be added to the cache.

The @CachePut annotation means to unconditionally execute the logic inside the method and add the method’s return value to the cache. This annotation is also marked on the method and is usually used to update the cache. It differs from the @Cacheable annotation in that every time a method annotated with @CachePut is called, the logic inside the method is executed and the result is added to the cache , overwriting the original cached value.

The @CachePut annotation is often used in scenarios where data in the cache needs to be updated. It can ensure that every time a method is called, the logic inside the method will be executed and the cached results will be updated.

Cacheable: Data will be stored in redis and data will also be read.

CachePut: will write data in redis but will not read data.

3. @CacheEvict annotation

Used to clear cached data used on this method or class (clear where used)

value: a name of the cache location, cannot be empty
key: cached key, empty by default, indicating that the parameter type and parameter value of the method are used as the key, supporting SpEL
condition: Trigger condition. If the condition is met, it will be added to the cache. The default is empty, which means all are added to the cache. SpEL is supported.
allEntries: true means clearing all caches in value, the default is false

Test results: You can configure to delete specified cache data, or you can delete all cache data that meets the rules;

4. Application Scenario

Redis annotation caching is a method that simplifies caching operations by using annotations. It can conveniently automatically read and write the cache before and after method calls. The following are common application scenarios for Redis annotation caching:

  1. Method result caching: For some methods with high computational cost, annotations can be used to cache the results to avoid repeated calculations. When the same method is called next time, the results can be obtained directly from the cache, improving performance and reducing server load.

  2. Data query cache: For frequently read data query operations, you can use annotations to cache the query results. This can avoid repeatedly querying the database and improve the response speed of the system.

  3. Current limiting and circuit breaker: By using annotations to implement current limiting, you can control the number of concurrent requests for certain resources or interfaces and avoid system overload. At the same time, you can set a specific cache time. When the resource or interface is unavailable, the data in the cache will be returned to achieve circuit breaker degradation.

  4. Prevent cache penetration: By using technologies such as annotations and Bloom filters, you can intercept requests before querying and determine whether the corresponding data exists in the cache. If it does not exist, a null value is returned directly to avoid invalid queries to the database.

  5. Asynchronous refresh cache: By using annotations and asynchronous tasks, the corresponding cache can be updated asynchronously at the same time when the data is updated, maintaining the consistency of the cache and the database.

3. Redis breakdown and penetration avalanche

The simple function of redis: It can greatly reduce the access pressure of MySQL.

Most solutions require choosing the appropriate solution based onspecific scenarios and needs. At the same time, reasonable caching strategies, data warm-up and monitoring are the keys to mitigating these problems to ensure system stability and reliability.

1. Breakdown (Cache Miss)

What is cache breakdown?

Cache breakdown is when a hotspot key fails in a centralized high-concurrency access situation, and a large number of requests cannot be obtained in the cache. The cache was instantly penetrated, and all requests were hit directly to the database, just like a hole was penetrated in a barrier.

When the concurrency is high, the key fails, causing the request to directly reach the database;

  • Solution one: Set lock
    • Use a mutex lock or distributed lock to prevent concurrent requests from accessing the database at the same time.
      • Acquire the Redis lock. If not acquired, return to the task queue and continue queuing.
      • Obtain the lock, pull the data from the database and put it into the cache
      • Release the lock and other requests get data from the cache
  • Another way: Current limiting
    • Set a short expiration time in the cache and use an asynchronous update mechanism. That is, when the cache expires, only one request reloads the data into the cache, and other requests wait for the cache data update to complete before reading.
    • Do traffic peak cutting before requesting redis

2. Penetration (Cache Penetration)

Many requests are accessing data that must not exist in the database, causing requests to penetrate both the cache and the database.

The solution is as follows:

  • Rule exclusions:
    1. Use data structures such as Bloom Filters to filter non-existent requests and make judgments before querying the cache to avoid unnecessary queries to the database.
    2. Or you can add some parametric tests. For example, database data IDs are generally incrementing. If a parameter like id = -10 is requested, Redis will inevitably be bypassed. To avoid this situation, you can perform operations such as user authenticity verification.
  • Null Cache:
    1. That is, empty results are cached for a period of time to avoid frequent queries for non-existent data.
    2. It can be understood that when the cache is penetrated, redis stores a null-like value, and the next access will directly return null to the cache. When the value of the data exists in the database, the null value that exists in redis needs to be cleared and the new value loaded. This solution cannot solve frequent random and irregular key requests.

3. Cache Avalanche

Avalanche is similar to breakdown. The difference is that breakdown is when a hotspot key fails at a certain moment, while avalanche is when a large number of hotspot keys fail in an instant.

The plan is as follows:

  • Set an appropriate cache expiration time. You can use random values or add a certain time offset to avoid multiple keys from invalidating at the same time, resulting in excessive database load.
  • Using a multi-level cache architecture, the cache is divided into multiple levels, such as local cache, distributed cache, etc., to improve cache hit rate and availability.
  • Monitor the cache health status in real time. When the cache is abnormal, take corresponding measures, such as automatic downgrade, current limiting, etc., to protect the back-end system.

A faster solution is to set different caching strategies for different hotspot keys.

syntaxbug.com © 2021 All Rights Reserved.