Original/Zhu Jiqian
Background: Recently, I was doing a stress test on a newly developed Springboot system. I found that when I first started the stress test, I could access data from the redis cluster normally. However, after a few minutes of pause, and then when I continued to use jmeter to perform the stress test, I found that redis started to fail. Suddenly, an abnormal message popped up like crazy: Command timed out after 6 second(s)…
Caused by: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 6 second(s) at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51) at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114) at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123) at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80) at com.sun.proxy.$Proxy134.mget(Unknown Source) at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.mGet(LettuceStringCommands.java:119) ... 15 common frames omitted
I hurriedly checked the redis cluster and found that all the nodes in the cluster were normal, and the CPU and memory usage were less than 20%. Looking at all this, I suddenly fell into a long meditation on where the problem was.. …. After searching on Baidu, I found that many people have experienced similar situations. Some people said that setting the timeout to a larger value can solve the problem. I followed this solution and set the timeout value to a larger value, but the timeout problem was still not solved.
Among them, the dependency package for springboot to operate redis is–
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
Cluster configuration–
redis:
timeout: 6000ms
cluster:
nodes:
-xxx.xxx.x.xxx:6379
-xxx.xxx.x.xxx:6379
-xxx.xxx.x.xxx:6379
jedis:
pool:
max-active: 1000
max-idle: 10
min-idle: 5
max-wait: -1
Click on spring-boot-starter-data-redis and find that it contains lettuce dependencies:
I saw some netizens saying that springboot1.x uses jedis by default, and springboot2.x uses lettuce by default. We can simply verify that in the redis driver loading configuration class, output the RedisConnectionFactory information:
@Configuration
@AutoConfigureAfter(RedisAutoConfiguration.class)
public class Configuration {
@Bean
public StringRedisTemplate redisTemplate(RedisConnectionFactory factory) {
log.info("Test print driver type:" + factory);
}
Printout–
Test print driver type: org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory@74ee761e
It can be seen that the lettuce driver connection is used here. My current temporary solution is to replace it with the jedis driver connection that I used more often before. This Command timed out after 6 second( s) problem.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
<exclusions>
<exclusion>
<groupId>io.lettuce</groupId>
<artifactId>lettuce-core</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
</dependency>
So the question is, how does Springboot2.x use lettuce by default? We need to study some of the code inside. We can enter the redis part of the Springboot2.x automatic assembly module, which has a RedisAutoConfiguration class. Its main function is to automatically configure the Springboot connection redis class:
@Configuration(
proxyBeanMethods = false
)
@ConditionalOnClass({RedisOperations.class})
@EnableConfigurationProperties({RedisProperties.class})
@Import({LettuceConnectionConfiguration.class, JedisConnectionConfiguration.class})
public class RedisAutoConfiguration {
public RedisAutoConfiguration() {
}
...omitted
}
Here you only need to pay attention to one line of annotation:
@Import({LettuceConnectionConfiguration.class, JedisConnectionConfiguration.class})
This means that when using the spring-boot-starter-data-redis dependency, both lettuce and jedis drivers can be automatically imported. Logically speaking, there will not be two drivers at the same time, which does not make much sense. Therefore, the order here The order is very important, why do you say that?
Enter LettuceConnectionConfiguration.class and JedisConnectionConfiguration.class respectively, each showing the core code that this article needs to involve:
//LettuceConnectionConfiguration
@ConditionalOnClass({RedisClient.class})
class LettuceConnectionConfiguration extends RedisConnectionConfiguration {
...omitted
@Bean
@ConditionalOnMissingBean({RedisConnectionFactory.class})
LettuceConnectionFactory redisConnectionFactory(ObjectProvider<LettuceClientConfigurationBuilderCustomizer> builderCustomizers, ClientResources clientResources) throws UnknownHostException {
LettuceClientConfiguration clientConfig = this.getLettuceClientConfiguration(builderCustomizers, clientResources, this.getProperties().getLettuce().getPool());
return this.createLettuceConnectionFactory(clientConfig);
}
}
//JedisConnectionConfiguration
@ConditionalOnClass({GenericObjectPool.class, JedisConnection.class, Jedis.class})
class JedisConnectionConfiguration extends RedisConnectionConfiguration {
...omitted
@Bean
@ConditionalOnMissingBean({RedisConnectionFactory.class})
JedisConnectionFactory redisConnectionFactory(ObjectProvider<JedisClientConfigurationBuilderCustomizer> builderCustomizers) throws UnknownHostException {
return this.createJedisConnectionFactory(builderCustomizers);
}
}
<strong>It can be seen that LettuceConnectionConfiguration.class and JedisConnectionConfiguration.class both have the same annotation @ConditionalOnMissingBean({RedisConnectionFactory.class}). This means that if the RedisConnectionFactory bean has been registered in the container, then with it Similar other beans will no longer be loaded and registered. To put it simply, add @ConditionalOnMissingBean({RedisConnectionFactory.class}) annotation to LettuceConnectionConfiguration and JedisConnectionConfiguration respectively. Only one of them can be loaded and registered into the container, and the other one can be loaded and registered. No more loading registrations will be performed. </strong>
So, the question becomes, who will be registered first?
This goes back to the sentence mentioned above. The order in the sentence @Import({LettuceConnectionConfiguration.class, JedisConnectionConfiguration.class}) is critical. LettuceConnectionConfiguration is in front, which means that LettuceConnectionConfiguration will be registered.
It can be seen that Springboot uses lettuce to connect to redis by default.
When we introduce the spring-boot-starter-data-redis dependency package, it is actually equivalent to introducing the lettuce package. At this time, the lettuce driver will be used. If you do not want to use the default lettuce driver, you can directly exclude the lettuce dependency.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
<exclusions>
<exclusion>
<groupId>io.lettuce</groupId>
<artifactId>lettuce-core</artifactId>
</exclusion>
</exclusions>
</dependency>
Then introduce the jedis dependency–
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
</dependency>
In this way, when making the import annotation of RedisAutoConfiguration, because the lettuce dependency is not found, the JedisConnectionConfiguration in the second position of the annotation @Import({LettuceConnectionConfiguration.class, JedisConnectionConfiguration.class}) is valid and can be Registered to the container, it is used as the driver for springboot to operate redis.
What is the difference between lettuce and jedis?
lettuce: The bottom layer is implemented in netty, thread-safe, and has only one instance by default.
jedis: Can be directly connected to the redis server and used with the connection pool to increase physical connections.
Find the method where the error occurred according to the exception prompt. LettuceConverters.toBoolean(this.getConnection().zadd(key, score, value)) in the following code–
public Boolean zAdd(byte[] key, double score, byte[] value) {
Assert.notNull(key, "Key must not be null!");
Assert.notNull(value, "Value must not be null!");
?
try {
if (this.isPipelined()) {
this.pipeline(this.connection.newLettuceResult(this.getAsyncConnection().zadd(key, score, value), LettuceConverters.longToBoolean()));
return null;
} else if (this.isQueueing()) {
this.transaction(this.connection.newLettuceResult(this.getAsyncConnection().zadd(key, score, value), LettuceConverters.longToBoolean()));
return null;
} else {
return LettuceConverters.toBoolean(this.getConnection().zadd(key, score, value));
}
} catch (Exception var6) {
throw this.convertLettuceAccessException(var6);
}
}
LettuceConverters.toBoolean() converts long to Boolean. Under normal circumstances, this.getConnection().zadd(key, score, value) will return 1 if the addition is successful, so LettuceConverters.toBoolean(1) What is obtained is true. On the contrary, if the new addition fails, 0 is returned, that is, LettuceConverters.toBoolean(0). There is also a third case where an exception occurs in this.getConnection().zadd(key, score, value) method. , under what circumstances will an exception occur?
It should be when the connection fails.
This means that during the process of connecting to redis with the lettuce driver, the connection will be disconnected, resulting in the failure to successfully add the new data. If it is not normal after a certain period of time, the connection will time out.
As for what caused the disconnection, I don’t have a good idea for the time being. I’ll leave this problem aside for the time being and wait until I can slowly study it to see if I can find the problem. If anyone has any advice, I’d be grateful.
Finally, I found that the Lettuce client is not recommended for redis client connection on the Alibaba Cloud official website–
