This article has been included in GitHub. Recommended reading: Java Random Notes
WeChat public account: Java Caprice
It is not easy to be original, so pay attention to copyright. Please indicate the original author and original link when reprinting
Article directory
-
- How memory fragmentation occurs
- memory allocator
- How to check if there is memory fragmentation
- The meaning of fragmentation rate
- Clean up memory fragmentation
-
- Redis versions lower than 4.0-RC3
- Redis versions higher than 4.0-RC3
As we explore and optimize Redis performance, ”
“Redis memory fragmentation” is a topic that cannot be ignored.
This article will delve into this seemingly trivial issue, but actually has an important impact on Redis operating efficiency. First, let’s demystify Redis memory fragmentation and understand its nature and why it becomes a challenge we must face.
How memory fragmentation occurs
Redis memory fragmentation is mainly caused by memory management problems in the process of Redis data storage and recycling.
When Redis allocates memory, it will apply for a continuous memory space as needed. However, when Redis deletes or modifies data, the released memory space may not be immediately reused, especially when the sizes of these free memory spaces are inconsistent, which may lead to memory fragmentation.
In order to improve the efficiency of memory usage, Redis uses a memory allocator internally to manage memory application and release. The default memory allocator used by Redis is “jemalloc“.
The memory allocator allocates memory according to a fixed size, not exactly according to the memory size requested by the program.
For example, if the program applies for a 20-byte memory, the memory allocator will allocate a 32-byte memory space. This is done to reduce the number of allocations. Redis will apply for different sizes of memory space to store different types of data for different businesses. Since the memory is allocated according to a fixed size and will be larger than the actual requested memory, memory fragmentation will occur in this process.
Let me give you an example from real life to help you understand:
Suppose you are organizing a library. The library bookshelf is like the memory space where Redis stores data. Each book represents a different size of data. When you first start, you arrange all your books according to their size. The small book is on one side and the large book is on the other side. This way you can effectively use the bookshelf space and make it easier to find books.
However, if you need to remove some books (delete some data) and then add new books (add data), problems may arise. For example, you remove some large books, make their spaces available, and put new smaller books in them. In this way, the space that originally belonged to the big book is now only partially occupied by the small book, and the remaining blank space becomes “memory fragments”.
Or maybe you have a bunch of new big books to put, but there is only room for scattered small books on the bookshelf, which cannot accommodate these big books. At this time, you may need to rearrange the entire bookshelf (similar to Redis’s memory organization) to free up a large continuous space to place these new large books.
To sum up:When data is continuously deleted and added, the vacated location in the memory may not completely match the size of the new data, resulting in unused “fragmented” space, which is memory fragmentation.
Memory allocator
Redis uses a memory allocator to manage the memory resources it needs to use during runtime. Can be libc, jemalloc, tcmalloc. The default is jemalloc.
To specify which memory allocator Redis uses, you need to make a choice when compiling Redis. Usually it can be specified through the MALLOC parameter when executing the make command. For example, if you want to use jemalloc, you can compile Redis like this: make MALLOC=jemalloc.
jemalloc divides the memory space into three ranges: small, large and huge in 64-bit systems. Each range is divided into many small memory block units. When storing data, the memory block with the most appropriate size will be selected for storage.
The memory units divided by jemalloc are shown in the figure below:
That is to say, Redis allocates continuous memory in units of blocks of a specified size, rather than on demand. Redis will allocate space of the corresponding size based on the closest fixed value of the requested memory.
It’s like you have different boxes. In order to put things in, you need to find a box with the closest volume. But after putting it in, you find that there is still space for some small things, so there is no need to look for boxes again.
However, this way of allocating space will bring a certain degree of memory fragmentation. We can think of fixed-size divided spaces as boxes of different volumes, and there will be surplus space in each box to varying degrees. These remaining spaces are memory fragments.
How to check if there are memory fragments
We log in to the Redis server and execute the following command, which will return a text describing the memory usage of Redis.
redis> info memory
We will see information similar to the following:
Here, our main concern is the field named mem_fragmentation_ratio, which shows the ratio of Redis memory fragmentation.
If mem_fragmentation_ratio is greater than 1, it indicates that memory fragmentation exists. The larger this value, the more fragmented the memory will be. If the value is very close to 1 or less than 1, it means there is little or no memory fragmentation.
The calculation formula is:
mem_fragmentation_ratio = used_memory_rss / used_memory
in:
used_memory_rss: represents the total physical memory size occupied by the Redis process (including code area, data area, stack, etc.), the unit is bytes.used_memory: Represents the total amount of memory requested by the Redis allocator, which is the virtual memory space actually used by the process from the perspective of the operating system. The unit is bytes.
The significance of fragmentation rate
Different values of mem_fragmentation_ratio indicate different situations.
- Greater than 1: It indicates that the memory is fragmented, usually between 1 and 1.5 is normal.
- Greater than 1.5: It indicates that the memory fragmentation rate is relatively large, and it is necessary to consider whether to clean up memory fragments and pay attention to it.
- Less than 1: It means that the swap memory has been used, that is, the hard disk is used. The normal memory is not enough. You need to consider whether to expand the memory. Using swap will greatly affect the performance.
Clean up memory fragments
Redis version lower than 4.0-RC3
Versions prior to Redis 4.0-RC3 did not have built-in memory defragmentation tools. If you want to clean up memory fragments, you can restart it.
When Redis restarts, it will store the data to the disk through the RDB persistence function, and then load the data from the disk to the memory. This process can effectively clean up the memory fragmentation. However, this approach can cause temporary interruptions in service.
Redis version higher than 4.0-RC3
Starting from the Redis 4.0-RC3 version, the active-defrag feature was introduced. It can automatically clean up fragments without restarting.
The enable configuration is as follows. The default value for this option is off. Activating defragmentation may take up some CPU time.
redis> config set activedefrag yes
Note: The function of automatically cleaning up memory fragments can be enabled only when the Redis memory allocator is jemalloc.
After enabling, automatic cleanup will be triggered only when the setting conditions of the following two parameters are met at the same time.
active-defrag-ignore-bytes 100mb # The default is 100MB, which means when the memory fragmentation space reaches 100MB active-defrag-threshold-lower 10 # The default is 10, which means when the proportion of memory fragmentation space to the physical memory space allocated to redis by the OS reaches 10%
Redis is a single-process model, and the automatic cleaning of memory fragments is operated through the main thread, which also consumes a certain amount of CPU resources. In order to avoid automatic cleaning from reducing the processing performance of Redis, the following two parameters can control the upper and lower limits of the proportion of CPU time consumed by cleaning actions:
active-defrag-cycle-min 5 # The default is 5, which means that the proportion of CPU time used in the automatic cleaning process is not less than 5%, ensuring that the cleaning can be carried out normally; active-defrag-cycle-max 75 # The default is 75, which means that the proportion of CPU time used in the automatic cleaning process is not higher than 75%. Once it exceeds, the cleaning will stop, thereby avoiding a large number of memory copies blocking Redis during cleaning, resulting in response delays. rise.
If you are not satisfied with the effect of automatic cleaning, you can use the following command to directly clean up the fragments manually:
redis > memory purge
It should be noted that this command will block the main process, and currently only the memory statistics of jemalloc as the memory allocator are implemented, and other allocators are not supported yet.
This article ends here. After we have an in-depth study of Redis memory fragmentation management and optimization strategies, we can make it clear: understanding and properly handling memory fragmentation is crucial to ensuring the performance and stability of Redis.
Whether it is adjusting the memory allocation strategy or using appropriate data structures, it is an optimization of Redis memory management.
At the same time, regular monitoring and review are also essential steps. I hope this article can provide you with some valuable inspiration in dealing with Redis memory fragmentation issues. Remember, every good engineer should take pride in understanding the tools they use. Let us continue to focus on and optimize Redis so that it can better serve our projects and promote business development.