Redis is an efficient in-memory database. It supports the storage of data types including String, List, Set, SortedSet and Hash. In Redis, the value is usually queried based on the key of the data. Redis does not have fuzzy condition queries. In the face of some needs When it comes to paging, sorting, and conditional query scenarios (such as comments, timelines, retrieval, etc.), it is not easy to handle just relying on the functions provided by Redis.
This article does not go into too much detail on the features of Redis. Since we needed to implement conditional query and paging functions based on Redis based on business problems before, we queried many articles on Baidu. Basically, either only the paging function or only the conditional query function was implemented, and there was a lack of a solution that combined the two.
Therefore, this article will provide a technical solution for conditional query + paging based on Redis.
Note: This article only provides implementation ideas and does not provide implementation code
This article will explain it in four parts:
-
Paging implementation
-
Fuzzy condition query implementation
-
Combination implementation of paging and fuzzy condition query
-
Optimization
You can jump directly to the part you need to read.
Redis paging implementation
We are usually used to implementing paging queries in persistent databases such as Mysql and Oracle. However, based on some special business scenarios, our data is not persisted to the database or hot data is loaded for query speed considerations. into the cache database. Therefore, we may need to perform paging queries based on a cache database such as Redis.
The implementation of Redis’ paging query is based on the ZSet data structure provided by Redis. The full name of ZSet is Sorted Set. This structure mainly stores ordered sets. The following is its instruction description and its role in paging implementation:
-
ZADD: SortedSet’s add element instruction ZADD key score member [[score, member]…] will bind a value score for sorting to each added element member, and SortedSet will sort the elements according to the score value. We are usually accustomed to using the time attribute of data as score for sorting. Of course, you can also choose the sorting target according to specific business scenarios.
-
ZREVRANGE: The instruction ZREVRANGE key start stop in SortedSet can return members in the specified range and can be used for paging. In addition, we recommend the public account Java Selection, respond to Java interviews, obtain online interview materials, and support answering questions anytime and anywhere.
-
ZREM: The instruction ZREM key member of SortedSet can remove the specified member based on the key, which can meet the requirements of deleting comments.
So SortedSet is very suitable for paging. The following is a demonstration diagram of paging implementation, including the query situation after inserting new records.
In fact, the List structure in Redis can also implement paging, but List cannot implement automatic sorting, and Zset can also filter data based on score to extract data within the target score range.
Recommended address for fishing:
https://www.yoodb.com/slack-off/home.html
So in terms of implementation, ZSet is often more suitable for us. Of course, if you need to insert repeated data, paging may need to be implemented with the help of List. The specific structure used to implement paging still needs to be selected based on the actual business scenario.
Redis multi-condition fuzzy query implementation
Redis is a key-value type in-memory database. Although it is convenient to directly retrieve data through key, it does not provide SQL conditional query support as convenient as mysql. Therefore, we need to use the structure and functions provided by Redis to implement the fuzzy condition query function ourselves.
In fact, Redis’s fuzzy condition query is implemented based on Hash. We can use certain condition values of the data as the key value of the hash, and store the data itself as the value. Then use the HSCAN instruction provided by Hash to traverse all keys and filter them, and get all the key values that meet our conditions (hscan can perform pattern matching).
For convenience, we usually put all the keys that meet the conditions into a Set or List. In this way, we can retrieve the corresponding data based on the obtained key value. The following is a demonstration diagram of fuzzy query (the naming rule in the field is
Query all users whose gender is female
Query all users with the surname A
Although HSCAN provides us with the pattern matching function, this matching is based on traversal. Each match requires traversing all keys, which is not very efficient. Therefore, this aspect will be supplemented in the following section. This section only talks about how to implement fuzzy matching.
Redis paging + multi-condition fuzzy query combination implementation
The previous sections separately describe how to implement Redis paging and multi-condition query. In actual use, using ZSet alone to implement paging can already show good performance, but there is a problem that the data we page is often accompanied by some dynamic filtering conditions, and ZSet does not provide such a function.
Faced with this situation, we usually have two solutions:
-
If the data has been stored in a persistent database, we can perform conditional queries in the database each time and then put the data into Redis for paging.
-
Implement multi-condition fuzzy query and paging in Redis.
The former solution is actually a good choice, but the disadvantage is that the data is sometimes not necessarily in the persistent database. In some business scenarios, in order to show better concurrency and high response, our data will be placed in the cache database first, and then persisted to the database until a certain time or when certain conditions are met.
In this case, our first solution will not work and we need to use the second solution. Therefore, the following will introduce how to implement paging based on multi-condition fuzzy query.
Implementation ideas
First of all, we can use the method mentioned in the multi-condition fuzzy query chapter to use the condition fields we involve as hash fields, and the content of the data is stored as the corresponding value (generally stored in json format to facilitate deserialization).
We need to implement the agreed query format. Using the example in the previous section, the naming rule in the field is
When we get the matching string, we first go to Redis to find whether there is a ZSet with the matching string as the key. If not, we will traverse all hash fields through the HSCAN provided by Redis, get all the fields that meet the conditions, and put them in A ZSet collection, and set the key of this collection to our conditional matching string. If it already exists, just perform paging query on this ZSet. The method of paging ZSet has been described previously. In this way, we have achieved the simplest paging + multi-condition fuzzy query.
In the above figure, since no matching ZSet collection was found in the cache database, we will generate a new collection based on the matching string for paging.
Performance optimization plan
Although the multi-condition fuzzy query + paging function has been implemented above, during time development, we cannot generate new collections without limit because the matching strings are very diverse, which will put huge pressure on the cache.
Therefore, we can give this collection an expiration time when generating the collection, and the expired collection will be automatically destroyed. Because according to the principle of temporal locality, data that we do not access for a period of time will most likely not be accessed again for a long period of time. And for the hit collection, we will update its expiration time.
At the same time, the real-time nature of our data is also a problem, because our collection is determined by the Hash content when the collection is generated. For newly inserted data into the Hash, the collection cannot be detected, so there are two solutions:
-
The first is to insert it into the Hash and then insert it into other corresponding collections at the same time to ensure that the data is always up to date. This method requires adding a special prefix for identification, otherwise we will not know which collections to insert into.
-
The second method is to update regularly. This method is more labor-saving, but it cannot guarantee the real-time nature of the paging data. Therefore, the specific choice depends on the business scenario.
Summary
This article roughly describes the solution for implementing paging and multi-condition fuzzy query. I hope it can be helpful to everyone.
Copyright statement: This article is an original article by CSDN blogger “It turns out to be someone Xiao” and follows the CC 4.0 BY-SA copyright agreement. Please attach the original source link and this statement when reprinting.
https://blog.csdn.net/qq_33905217/article/details/129211947
If the source of the content published in the public account "Java Selection" is indicated, the copyright belongs to the original source (if the copyright cannot be verified or the source is not indicated, it is from the Internet and is reprinted. The purpose of reprinting is to convey more information. The copyright belongs to the original author. If there is any infringement, please contact us and we will delete it as soon as possible! Recently, many people have asked if there is a reader exchange group! The method to join is very simple, just select the public account Java and reply "Add group" to join the group! Java Selected Interview Questions (WeChat Mini Program): 3000+ interview questions, including Java basics, concurrency, JVM, threads, MQ series, Redis, Spring series, Elasticsearch, Docker, K8s, Flink, Spark, architecture design, etc., online Answer questions at any time! ------ Special recommendation ------ Special recommendation: "Big Notes" is a public account that focuses on sharing the most cutting-edge technology and information, preparing for overtaking in corners, and various open source projects and high-efficiency software. It focuses on digging out good things and is very worthy of everyone's attention. Click on the public account card below to follow. Click "Read the original text" to learn more exciting content! If the article is helpful, please click to read and forward it!