Automatic update of redis cluster topology: exception handling when using Lettuce to connect to Cluster cluster instances

Questions:
Use lettuce to connect to the Cluster cluster instance. After the specification of the instance is changed, when the number of shards changes, some slots (Slots) will be migrated to the new shard. When the client connects to the new shard, the following abnormal problems will occur.

java.lang.IllegalArgumentException: Connection to 100.123.70.194:6379 not allowed. This connection point is not known in the cluster view
exceptionStackTrace
io.lettuce.core.cluster.PooledClusterConnectionProvider.getConnectionAsync(PooledClusterConnectionProvider.java:359)
io.lettuce.core.cluster.ClusterDistributionChannelWriter.write(ClusterDistributionChannelWriter.java:93)
io.lettuce.core.cluster.ClusterCommand.complete(ClusterCommand.java:56)
io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:563)
io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:516)

org.springframework.dao.QueryTimeoutException: Redis command timed out; nested exception is io.lettuce.core.RedisCommandTimeoutException: Command timed out after 5 second(s)
at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:70)

Questions
After redis connection, after a while, the connection is slow
In the Redis Cluster cluster mode, the master is down or the network is jittering, etc. During the master-slave switchover, an error is reported: Redis command timed out, etc.

Connection to X not allowed. This connection point is not known in the cluster view.
official website
refer to

Solution

Enable the cluster topology refresh function of the redis client. Different clients use different processing methods.

• The jedis client automatically supports it by default (because jedis uses its own abnormal feedback to identify reconnection and refresh the cluster information mechanism of the server to ensure its automatic fault recovery)
• The luttuce client is not enabled by default, you need to manually enable the refresh
  • Versions before springboot 1.x use jedis by default, there is no need to manually enable refresh
  • Springboot 2.x, the redis client defaults to Lettuce, which does not support topology refresh by default. Solution:
    • With jedis, there is no need to manually specify to enable refresh
    • To use lettuce, you need to set the refresh node topology policy to be enabled
  • Starting from springboot 2.3.0, the cluster topology refresh function is supported, and the attribute configuration can be enabled

springboot 2.0.X

Method 1

Use letture to connect to redis, you need to set the refresh node topology refresh strategy

Automatically reload partitions using RedisClusterClient.reloadPartitions

#redis cluster config
#RedisCluster cluster node and port information
spring.redis.cluster.nodes=192.168.50.29:6380,192.168.50.29:6381,192.168.50.29:6382,192.168.50.29:6383,192.168.50.29:6384,192.168.50.2 9:6385
#RedisPassword
spring.redis.password=
#Maximum number of redirections to follow when executing commands in the cluster
spring.redis.cluster.max-redirects=6
#The maximum number of connections that the Redis connection pool can allocate at a given time. Unlimited use of negative values
spring.redis.jedis.pool.max-active=1000
# Connection timeout in milliseconds
spring.redis.timeout=2000
# Maximum number of "idle" connections in the pool. Use a negative value for an unlimited number of idle connections
spring.redis.jedis.pool.max-idle=8
#The goal is to keep the minimum number of idle connections in the pool. This setting is only effective when max-idle is set
spring.redis.jedis.pool.min-idle=5
# The maximum amount of time, in milliseconds, that a connection allocation should block before throwing an exception when the pool is exhausted. Use a negative value to block indefinitely
spring.redis.jedis.pool.max-wait=1000
#redis cluster only uses db0
spring.redis.index=0

import io.lettuce.core.ReadFrom;
import io.lettuce.core.cluster.ClusterClientOptions;
import io.lettuce.core.cluster.ClusterTopologyRefreshOptions;
import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.AutoConfigureAfter;
import org.springframework.boot.autoconfigure.data.redis.RedisAutoConfiguration;
import org.springframework.boot.autoconfigure.data.redis.RedisProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.RedisClusterConfiguration;
import org.springframework.data.redis.connection.RedisStandaloneConfiguration;
import org.springframework.data.redis.connection.lettuce.LettuceClientConfiguration;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;
import org.springframework.data.redis.connection.lettuce.LettucePoolingClientConfiguration;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.serializer.GenericJackson2JsonRedisSerializer;
import org.springframework.data.redis.serializer.StringRedisSerializer;

import java.time.Duration;

/**
 * Redis configuration class
 *
 * @author 007
 */
@Configuration
@AutoConfigureAfter(RedisAutoConfiguration. class)
public class RedisConfiguration {<!-- -->
    @Autowired
    private RedisProperties redisProperties;

    @Bean(destroyMethod = "destroy")
    public LettuceConnectionFactory redisConnectionFactory() {<!-- -->
        // redis single node
        if (null == redisProperties.getCluster() || null == redisProperties.getCluster().getNodes()) {<!-- -->
            RedisStandaloneConfiguration configuration = new RedisStandaloneConfiguration(redisProperties. getHost(),
                    redisProperties. getPort());
            configuration.setPassword(redisProperties.getPassword());
            return new LettuceConnectionFactory(configuration);
        }

        // redis cluster
        RedisClusterConfiguration redisClusterConfiguration = new RedisClusterConfiguration(redisProperties. getCluster(). getNodes());
        redisClusterConfiguration.setPassword(redisProperties.getPassword());
        redisClusterConfiguration.setMaxRedirects(redisProperties.getCluster().getMaxRedirects());

        GenericObjectPoolConfig genericObjectPoolConfig = new GenericObjectPoolConfig();
        genericObjectPoolConfig.setMaxTotal(redisProperties.getLettuce().getPool().getMaxActive());
        genericObjectPoolConfig.setMaxIdle(redisProperties.getLettuce().getPool().getMaxIdle());
        genericObjectPoolConfig.setMinIdle(redisProperties.getLettuce().getPool().getMinIdle());
        genericObjectPoolConfig.setMaxWaitMillis(redisProperties.getLettuce().getPool().getMaxWait().getSeconds());

        // Support adaptive cluster topology refresh and dynamic refresh source
        ClusterTopologyRefreshOptions clusterTopologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
                .enableAllAdaptiveRefreshTriggers()
                // enable adaptive refresh
                .enableAdaptiveRefreshTrigger()
                // Turn on timing refresh
                .enablePeriodicRefresh(Duration.ofSeconds(5))
                .build();

        ClusterClientOptions clusterClientOptions = ClusterClientOptions.builder()
                .topologyRefreshOptions(clusterTopologyRefreshOptions).build();
/*
ClusterClientOptions clusterClientOptions = ClusterClientOptions.builder()
.topologyRefreshOptions(clusterTopologyRefreshOptions)//topology refresh
.disconnectedBehavior(ClientOptions.DisconnectedBehavior.REJECT_COMMANDS)
.autoReconnect(true)
.socketOptions(SocketOptions.builder().keepAlive(true).build())
.validateClusterNodeMembership(false)// Cancel the membership of the validation cluster node
.build();
*/

        LettuceClientConfiguration lettuceClientConfiguration = LettucePoolingClientConfiguration.builder()
                .poolConfig(genericObjectPoolConfig)
                .readFrom(ReadFrom.SLAVE_PREFERRED) // Use ReadFrom.SLAVE_PREFERRED to read and write separation
                .clientOptions(clusterClientOptions).build();

        LettuceConnectionFactory lettuceConnectionFactory = new LettuceConnectionFactory(redisClusterConfiguration, lettuceClientConfiguration);
        lettuceConnectionFactory.setShareNativeConnection(false);//Whether to allow multiple thread operations to share the same cache connection, the default is true, each operation will open a new connection when false
        lettuceConnectionFactory.resetConnection();// Reset the underlying shared connection and initialize it in the next visit
        return lettuceConnectionFactory;
    }

 //Customize a RedisTemplate
    @Bean
    @SuppressWarnings("all")
    public RedisTemplate<String, Object> redisTemplate(@Qualifier("lettuceConnectionFactoryUvPv") LettuceConnectionFactory factory) {<!-- -->
        RedisTemplate<String, Object> template = new RedisTemplate<>();
        // Be sure to inject RedisTemplate and redisTemplate.setConnectionFactory(lettuceConnectionFactory);
        template.setConnectionFactory(factory);
 
        //Json serialization configuration
        Jackson2JsonRedisSerializer<Object> jackson2JsonRedisSerializer = new Jackson2JsonRedisSerializer<Object>(Object.class);
        ObjectMapper om = new ObjectMapper();
        om.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY);
        om.activateDefaultTyping(om.getPolymorphicTypeValidator());
        om.enableDefaultTyping(ObjectMapper.DefaultTyping.NON_FINAL);
        // Solve the serialization problem
        om.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
        jackson2JsonRedisSerializer.setObjectMapper(om);
 
        //Serialization of String
        StringRedisSerializer stringRedisSerializer = new StringRedisSerializer();
 
        //key adopts the serialization method of String
        template.setKeySerializer(stringRedisSerializer);
        //The key of hash also adopts the serialization method of String
        template.setHashKeySerializer(stringRedisSerializer);
 
        //value serialization method uses jackson
        template.setValueSerializer(jackson2JsonRedisSerializer);
 
        //The value serialization method of hash adopts jackson
        template.setHashValueSerializer(jackson2JsonRedisSerializer);
        template. afterPropertiesSet();
 
        return template;
    }
}

import io.lettuce.core.ClientOptions;
import io.lettuce.core.cluster.ClusterClientOptions;
import io.lettuce.core.cluster.ClusterTopologyRefreshOptions;
import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.data.redis.RedisProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.data.redis.connection.RedisClusterConfiguration;
import org.springframework.data.redis.connection.RedisNode;
import org.springframework.data.redis.connection.RedisPassword;
import org.springframework.data.redis.connection.lettuce.LettuceClientConfiguration;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;
import org.springframework.data.redis.connection.lettuce.LettucePoolingClientConfiguration;
import org.springframework.stereotype.Component;

import java.time.Duration;
import java.util.HashSet;
import java.util.List;
import java.util.Set;

/**
 * @author : 007
 * @version : 2.0
 * @date : 2020/7/27 10:19
 */
@Component
public class RedisConfig {<!-- -->

    @Autowired
    private RedisProperties redisProperties;

    public GenericObjectPoolConfig<?> genericObjectPoolConfig(RedisProperties.Pool properties) {<!-- -->
        GenericObjectPoolConfig<?> config = new GenericObjectPoolConfig<>();
        config.setMaxTotal(properties.getMaxActive());
        config.setMaxIdle(properties.getMaxIdle());
        config.setMinIdle(properties.getMinIdle());
        if (properties.getTimeBetweenEvictionRuns() != null) {<!-- -->
            config.setTimeBetweenEvictionRunsMillis(properties.getTimeBetweenEvictionRuns().toMillis());
        }
        if (properties.getMaxWait() != null) {<!-- -->
            config.setMaxWaitMillis(properties.getMaxWait().toMillis());
        }
        return config;
    }

    @Bean(destroyMethod = "destroy")
    public LettuceConnectionFactory lettuceConnectionFactory() {<!-- -->

        //Enable adaptive cluster topology refresh and periodic topology refresh
        ClusterTopologyRefreshOptions clusterTopologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
                // Enable all adaptive refresh
                .enableAllAdaptiveRefreshTriggers() // Enable adaptive refresh, if adaptive refresh is not enabled, the connection will be abnormal when the Redis cluster is changed
                // enable adaptive refresh
                //.enableAdaptiveRefreshTrigger()
                // Adaptive refresh timeout (default 30 seconds)
                .adaptiveRefreshTriggersTimeout(Duration.ofSeconds(30)) //The default time is 30 seconds after it is turned off
                // open cycle refresh
                .enablePeriodicRefresh(Duration.ofSeconds(20)) // The default time is 60 seconds after it is turned off. ClusterTopologyRefreshOptions.DEFAULT_REFRESH_PERIOD 60 .enablePeriodicRefresh(Duration.ofSeconds(2)) = .enablePeriodicRefresh().refreshPeriod(Duration.ofSeconds (2))
                .build();

        // https://github.com/lettuce-io/lettuce-core/wiki/Client-Options
        ClientOptions clientOptions = ClusterClientOptions. builder()
                .topologyRefreshOptions(clusterTopologyRefreshOptions)
                .build();

        LettuceClientConfiguration clientConfig = LettucePoolingClientConfiguration.builder()
                .poolConfig(genericObjectPoolConfig(redisProperties.getJedis().getPool()))
                //.readFrom(ReadFrom.MASTER_PREFERRED)
                .clientOptions(clientOptions)
                .commandTimeout(redisProperties.getTimeout()) //Default RedisURI.DEFAULT_TIMEOUT 60
                .build();

        List<String> clusterNodes = redisProperties.getCluster().getNodes();
        Set<RedisNode> nodes = new HashSet<RedisNode>();
        clusterNodes.forEach(address -> nodes.add(new RedisNode(address.split(":")[0].trim(), Integer.valueOf(address.split(":")[1]) )));

        RedisClusterConfiguration clusterConfiguration = new RedisClusterConfiguration();
        clusterConfiguration.setClusterNodes(nodes);
        clusterConfiguration.setPassword(RedisPassword.of(redisProperties.getPassword()));
        clusterConfiguration.setMaxRedirects(redisProperties.getCluster().getMaxRedirects());

        LettuceConnectionFactory lettuceConnectionFactory = new LettuceConnectionFactory(clusterConfiguration, clientConfig);
        // lettuceConnectionFactory.setShareNativeConnection(false); //Whether multiple thread operations are allowed to share the same cache connection, the default is true, and each operation will open a new connection when false
        // lettuceConnectionFactory.resetConnection(); // Reset the underlying shared connection, initialized on subsequent visits
        return lettuceConnectionFactory;
    }
}

Method 2

Use the jedis method to connect to redis, and the service using the jedis client can be restored 15 seconds after the master-slave switch

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
    <exclusions>
        <exclusion>
            <groupId>io.lettuce</groupId>
            <artifactId>lettuce-core</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<dependency>
    <groupId>redis.clients</groupId>
    <artifactId>jedis</artifactId>
</dependency>

Note that the configuration file replaces lettuce.pool with jedis.pool

• lettuce configuration information

  spring.redis.lettuce.pool.max-active=10
  spring.redis.lettuce.pool.max-idle=10
  spring.redis.lettuce.pool.max-wait=10
  spring.redis.lettuce.pool.min-idle=5
  spring.redis.lettuce.shutdown-timeout=1S
  

• jedis configuration information

  spring.redis.jedis.pool.max-active=10
  spring.redis.jedis.pool.max-idle=10
  spring.redis.jedis.pool.max-wait=10
  spring.redis.jedis.pool.min-idle=5
  

  spring:
    redis:
      password: xxx
      host: 172.16.0.x
      port: 6579
      timeout: 5000
      jedis:
        pool:
                #Maximum number of database connections, set 0 as no limit
          max-active: 8
                #The maximum number of waiting connections, set 0 as no limit
          max-idle: 8
                #Maximum connection establishment waiting time. An exception will be received if this time is exceeded. Set to -1 for unlimited.
          max-wait: -1ms
                #The minimum number of waiting connections, set 0 as no limit
          min-idle: 0
      #lettuce:
        #pool:
          #max-active: ${redis.config.maxTotal:1024}
          #max-idle: ${redis.config.maxIdle:50}
          #min-idle: ${redis.config.minIdle:1}
          #max-wait: ${redis.config.maxWaitMillis:5000}
  

springboot 2.3

configuration properties

spring.redis.lettuce.cluster.refresh.adaptive=true
spring.redis.lettuce.cluster.refresh.period=30000

spring:
  redis:
lettuce:
cluster:
refresh:
adaptive: true
period: 30000 # Automatically refresh every 30 seconds