Example explanation of redis using pipeline mode to write performance tuning under Java Springboot

Instance of redis write pipeline mode performance tuning under Springboot

1. Real scene

In the process of producing real projects, it is necessary to write the data of the database to redis synchronously, and encounter the bottleneck of writing to redis during this process. Every time the project is started, the database data must be reloaded to redis, which takes a lot of time.

Second, solution ideas pipelining (pipeline)

Redis pipelined
Redis Pipelined is a way to request redis provided by the client (an operation that prevents the client from blocking). Redis itself has a high throughput, so the biggest test of performance is the network condition. If the network condition applied to redis is not good, each request will be slightly blocked and delayed. This delay is very important for batch requests. Terrible, for example, when we want to insert thousands of data, or get data in batches, we need to use Pipelined.
Pipelined can issue multiple requests without blocking and “package” the request results back in order, which is somewhat similar to concurrent requests and can effectively use the blocking time of waiting for the results.
Note that Pipelined does not guarantee atomicity, that is, the content executed by Pipelined may be “queued” by instructions from other clients or threads. If you want atomic operations, you need to use transactions.

pipelined based on RedisTemplate
Pipelined can be easily implemented using RedisTemplate, and it needs to rely on the native RedisConnection object to implement related operations!
pipelining
Pipeline: The pipeline command of redis, which allows the client to send multiple requests to the server in sequence, without waiting for the reply of the request in the process, and read the results together at the end, which can improve performance.
pipeline is not an atomic operation

Why?
Reduce the number of requests, combine multiple request commands into one request and send it to the redis server through the pipeline, and then receive the results of multiple commands at one time through the callback function, reducing the number of network IOs, which can bring about significant performance improvements in high concurrency situations. It should be noted that the redis server is single-threaded, and multiple commands are synthesized into one request to arrive at the redis server, and they are still executed sequentially one by one, which only reduces the number of IO requests.
How to use?
RedisCallback and SessionCallBack:
1. Function: Let RedisTemplate make callbacks, through which multiple redis commands can be executed at one time in the same connection.
2.SessionCalback provides a good package, use it first.
3. RedisCallback uses the native RedisConnection, which is cumbersome to use and has poor readability, but the native API provides relatively complete functions.

3. Argument performance comparison compared with traditional mode

<dependency>
    <groupId>redis.clients</groupId> #maven import
    <artifactId>jedis</artifactId>
    <version>2.9.0</version>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>

 compile('redis.clients:jedis:2.9.0') #gradle introduction
   compile('org.springframework.data:spring-data-redis:2.0.8.RELEASE')

package pipeline;

import redis.clients.jedis.Jedis;
import redis.clients.jedis.Pipeline;

public class BatchOperSet {<!-- -->

    private static final String HOST = "127.0.0.1";
    private static final int PORT = 6379;

    // Batch insert data into Redis, normal use
    public static void batchSetNotUsePipeline() throws Exception {<!-- -->
        Jedis jedis = new Jedis(HOST, PORT);
        String keyPrefix = "normal";
        long begin = System. currentTimeMillis();
        for (int i = 1; i < 10000; i ++ ) {<!-- -->
            String key = keyPrefix + "_" + i;
            String value = String. valueOf(i);
            jedis.set(key, value);
        }
        jedis. close();
        long end = System. currentTimeMillis();
        System.out.println("not use pipeline batch set total time: " + (end - begin));
    }

    // Batch insert data into Redis, using Pipeline
    public static void batchSetUsePipeline() throws Exception {<!-- -->
        Jedis jedis = new Jedis(HOST, PORT);
        Pipeline pipelined = jedis. pipelined();
        String keyPrefix = "pipeline";
        long begin = System. currentTimeMillis();
        for (int i = 1; i < 10000; i ++ ) {<!-- -->
            String key = keyPrefix + "_" + i;
            String value = String. valueOf(i);
            pipelined.set(key, value);
        }
        pipelined.sync();
        jedis. close();
        long end = System. currentTimeMillis();
        System.out.println("use pipeline batch set total time: " + (end - begin));
    }

    public static void main(String[] args) {<!-- -->
        try {<!-- -->
            batchSetNotUsePipeline();
            batchSetUsePipeline();
        } catch (Exception e) {<!-- -->
            e.printStackTrace();
        }
    }
}

The results of the write performance comparison are as follows:

not use pipeline batch get total time: 2990
use pipeline batch get total time: 41

Conclusion: The write performance of pipeline mode is faster

4. Examples of actual cases

Redis data structure used in real cases
Hash
basic concept:
Redis hashes can store mappings between multiple key-value pairs. Like strings, the values stored in hashes can be both strings and numbers, and users can also perform increment or decrement operations on numeric values stored in hashes. This is very similar to Java’s HashMap, each HashMap has its own name, and can store multiple k/v pairs at the same time.
Underlying implementation:
The encoding of the hash object can be ziplist or hashtable.
If the key and value string lengths of all key-value pairs stored in the hash object are less than 64 bytes and the number of key-value pairs stored is less than 512, ziplist encoding is used; otherwise, hashtable is used;
Application scenario:
Hash is more suitable for storing structured data, such as objects in Java; in fact, objects in Java can also be stored in string, just serialize the object into a json string, but if a certain attribute of the object is updated frequently , then the entire object needs to be serialized and stored again every time, which consumes a lot of overhead. But if hash is used to store each attribute of the object, then only the attribute to be updated needs to be updated each time.
Shopping cart scenario: the user’s id can be used as the key, the product’s id can be used as the stored field, and the quantity of the product can be used as the value of the key-value pair, thus forming the three elements of the shopping cart.

redis 127.0.0.1:6379> hset myhash field1 "zhang" #Set the field for the key whose key value is myhash to field1, and the value is zhang.
(integer) 1
redis 127.0.0.1:6379> hget myhash field1 #Get the key value is myhash, and the field is the value of field1.
"zhang"
redis 127.0.0.1:6379> hget myhash field2 # The field2 field does not exist in the myhash key, so nil is returned.
(nil)
redis 127.0.0.1:6379> hset myhash field2 "san" #Add a new field field2 to myhash, whose value is san.
(integer) 1
redis 127.0.0.1:6379> hlen myhash #hlen command to get the number of fields of myhash key.
(integer) 2
redis 127.0.0.1:6379> hexists myhash field1 #Determine whether there is a field named field1 in the myhash key. Because it exists, the return value is 1.
(integer) 1
redis 127.0.0.1:6379> hdel myhash field1 #Delete the field named field1 in the myhash key, and return 1 if the deletion is successful.
(integer) 1
redis 127.0.0.1:6379> hdel myhash field1 #Delete the field named field1 in the myhash key again, because it has been deleted by the previous command, because it has not been deleted, return 0.
(integer) 0
redis 127.0.0.1:6379> hexists myhash field1 #Determine whether the field1 field exists in the myhash key, because the previous command has deleted it, because it returns 0.
(integer) 0
redis 127.0.0.1:6379> hsetnx myhash field1 zhang #Add a new field field1 to myhash through the hsetnx command, and its value is zhang. Since this field has been deleted, the command is added successfully and returns 1.
(integer) 1
redis 127.0.0.1:6379> hsetnx myhash field1 zhang #Because the field1 field of myhash has been successfully added through the previous command, this command returns 0 after doing nothing.
(integer) 0

We can use the Redis Desktop Manager tool to view the contents of the redis database.

5. Example of Hash structure mode

Hash traditional code mode is as follows:

#Put all map content into redis according to the specified large KEY value
public static <T> void hPutAll(String hashKey, Map<String,T> map, long timeout){<!-- -->
        try{<!-- -->
            if(StringUtils.isBlank(hashKey)){<!-- -->
                return;
            }
            BoundHashOperations<String, String, String> stringObjectObjectBoundHashOperations = redisClient.redisTemplate.boundHashOps(hashKey);

            map.forEach((key,v)->{
                stringObjectObjectBoundHashOperations.put(key,gson.toJson(v));
            });
        }catch (Exception e){<!-- -->
            Logger.error("redis hputall exception", e);
        }
    }

The Hash pipeline mode is as follows:

 public static <T> void hPutAll(String hashKey, Map<String,T> map, long timeout){<!-- -->
        try{<!-- -->
            if(StringUtils.isBlank(hashKey)){<!-- -->
                return;
            }
redisClient.redisTemplate.executePipelined(new SessionCallback<Object>() {<!-- -->
public Object execute(RedisOperations ro) {<!-- -->
BoundHashOperations<String, String, String> hashOps = redisClient.redisTemplate
.boundHashOps(hashKey);

map.forEach((key, v) -> {
hashOps.put(key, gson.toJson(v));

});

//redisClient.redisTemplate.expire(hashKey, defaultTimeOut, TimeUnit.SECONDS);
return null;
}
});

6. Example of SET structure pattern

Pipeline mode for additional set data structures:

@Component
public class RedisTest {<!-- -->
 
    @Autowired
    RedisTemplate<String, Object> redisTemplate;
 
    @PostConstruct
    public void init() {<!-- -->
        test1();
    }
 
    public void test1() {<!-- -->
        List<Object> pipelinedResultList = redisTemplate. executePipelined(new SessionCallback<Object>() {<!-- -->
            @Override
            public <K, V> Object execute(RedisOperations<K, V> operations) throws DataAccessException {<!-- -->
                ValueOperations<String, Object> valueOperations = (ValueOperations<String, Object>) operations. opsForValue();
 
                valueOperations.set("yzh1", "hello world");
                valueOperations.set("yzh2", "redis");
 
                valueOperations.get("yzh1");
                valueOperations.get("yzh2");
 
                // Just return null, because the return value will be overwritten by the return value of the pipeline, and the outer layer cannot get the return value here
                return null;
            }
        });
        System.out.println("pipelinedResultList=" + pipelinedResultList);
    }
}

Pipeline warm-up code
Some systems have high latency requirements, so the first redis pipeline request is very slow, so the pipeline needs to be warmed up when the system starts to ensure low latency for each request after the system starts.

 @PostConstruct
    public void init() {<!-- -->
        long startTime = System. currentTimeMillis();
        redisTemplate.executePipelined(new SessionCallback<Object>() {<!-- -->
            @Override
            public <K, V> Object execute(RedisOperations<K, V> operations) throws DataAccessException {<!-- -->
                operations. hasKey((K) "");
                return null;
            }
        });
        log.info("redis initialization pipeline request end, time-consuming: {}ms", System.currentTimeMillis() - startTime);
    }

7. Example of LIST structure mode

Append LIST data structure code

public void redisPop(List<String> list) {<!-- -->
    List<Object> keys = redisTemplate. executePipelined(new SessionCallback<String>() {<!-- -->
        @Override
        public <K, V> String execute(RedisOperations<K, V> redisOperations) throws DataAccessException {<!-- -->
            for(String str: list){<!-- -->
                for (int i = 0; i < 200; i ++ ) {<!-- -->
                    redisOperations.opsForList().rightPop((K) str);
                }
            }
            return null;
        }
    });
   
   
}