Click above to follow “Terminal R&D Department“
Set it as a "star" to master more database knowledge with you
Article source: https://c1n.cn/mtr19
Directory
-
background
-
Diary Slimming Methodology
-
Optimization case
-
Summarize
Background
In daily development, usually for the convenience of debugging and troubleshooting, a lot of INFO level logs are printed.
As the number of visits increases, accidentally, the size of a certain log file in one day exceeds a certain threshold (such as 5G), so you receive an alarm for optimizing the log size, and feedback to your supervisor if you do not optimize within a certain period of time ,Oops…
Excessively large logs can easily cause some operation and maintenance operations to consume machine performance, such as log file retrieval, data collection, and disk cleanup.
So, what are the common ideas for log slimming? This article is based on a specific case to talk about my views.
Diary Slimming Methodology
| Only print necessary logs
Sometimes for the convenience of testing, a lot of INFO level logs are temporarily printed. For this kind of log, you can delete unnecessary logs or adjust them to DEBUG level before the project goes online.
However, in some scenarios, some logs can be printed as DEBUG or INFO. Printing at the INFO level takes up space, and printing at the DEBUG level is needed when checking problems online. What should I do?
We can modify the log tool class to support the passing of a certain switch in the context (the normal call does not have this switch, and the company’s Tracer or RPC context is passed), and the DEBUG log can be temporarily raised to the INFO level.
The pseudocode is as follows:
if(log.isDebugEnable()){
log.debug(xxx);
}else if(TracerUtils. openDebug2Info()){
log.info("【debug2info】" + xxx);
}
In this way, some logs that are tangled whether to print as INFO logs can be printed as DEBUG level, and automatically upgraded to INFO logs when checking problems.
In order to avoid misunderstanding, distinguish between DEBUG promotion INFO logs and ordinary INFO logs, and add a log prefix similar to [debug2info].
Of course, you can also engage in some other show operations, here is just an example, please infer other cases from one instance.
| Combined printing
Some logs that can be merged can be considered to be merged.
If the INFO log is printed before and after the same method:
INFO [64-bit traceId] XXXService before execution size =10 INFO [64-bit traceId] XXXService after execution size =4
can be combined into one:
INFO [64-bit traceId] XXXService before execution size=10 after execution size=4
| Simplify & amp; Abbreviate & amp; Compress
A certain log is very necessary, but the printed object is a bit large. If the troubleshooting requirements can be met, we can:
-
Choose to print only its ID
-
Create a log-specific object that only retains key fields, convert it into a log-specific object, and then print.
-
Abbreviations can be used, such as write is simplified to w, read is simplified to r, execute is simplified to e, etc.; for example, there are 20 core beans in the pipeline, and different numbers can be used to replace the full bean name when printing logs, such as S1 and S2, although not so Intuitive, but it can check problems and reduce the amount of logs.
Optimization Case
|Scene description
A business scenario involves many beans. In order to reuse some common logic, these beans inherit from an abstract class.
In the abstract class, some common logic before and after executing the bean is defined, such as printing the number of items in the current pipeline before and after execution. The last bean needs to print out the result after performing the result transformation.
| Optimization Analysis
①Only print necessary logs
Only print necessary logs:
-
Since the execution of the current bean is equivalent to the execution of the previous bean, only the log after execution can be printed, and the INFO log before execution can be deleted or changed to DEBUG (only necessary logs are printed)
-
Usually the problem only occurs when the size before and after execution is inconsistent, so you can add a judgment before printing the log after execution. If the size before and after execution is the same, it will not be printed.
The pseudocode is as follows:
if(sizeBefore != sizeAfter){
log.info("service:{}, before size:{}, after size:{}", getName(), sizeBefore, sizeAfter)
}
The effect of this trick is obvious, because most beans have the same size before and after execution, so this log will not be printed. And assuming that there were 20 before, this log needs to be printed 20 times, and it may only need to be printed 2-3 times after improvement.
②Log merge
In order to facilitate troubleshooting, it is also necessary to print the size before execution, then record the size before execution in the memory, and print the size before execution when printing the log after execution.
The pseudocode is as follows:
log.info("service:{}, size:{}", getName(), sizeBefore)
log.info("service:{}, size after execution:{}", getName(), sizeBefore, sizeAfter)
After the merge:
log.info("service:{}, before size:{}, after size:{}", getName(), sizeBefore, sizeAfter)
③Log simplification
For the final result, convert the result object (such as XXDTO) into a log object (XXSimpleLogDTO) that only includes key information, such as id, title, and then print it after converting it into a log object.
log.info("resultId:{}",result.getId());
or:
log.info("result:{}",toSimpleLog(result));
| Performance Evaluation
The log generates about 5G a day, about 80% of which is the size before and after printing execution, about 10% is the final result of printing, and some other logs.
After optimization by the above method, the daily log volume is less than 1G.
There is a trade-off between meeting the needs of troubleshooting and achieving log slimming.
Summary
Log thinning needs to be balanced, and the necessary logs for troubleshooting are kept as streamlined as possible. It can be optimized by deleting unnecessary logs, merging logs, and simplifying logs.
We can also perform some show operations, and support online DEBUG to temporarily increase INFO (of course, arthas can also be used) to assist us in troubleshooting.
What other good journaling tips do you have for weight loss? Welcome to comment and communicate with me at the end of the article.
Written at the end
If you have anything you don’t understand after reading it, you can leave a message below to discuss it, or you can private message me to ask me. I will usually reply after seeing it. Finally, if you think the article is helpful to you, remember to give it a thumbs up, and pay attention to it so you don’t get lost
@Terminal R&D Department
There are fresh dry goods to share every day!
Reply [idea activation] to get the activation method of idea Reply [Java] Get java-related video tutorials and materials Reply [SpringCloud] Get more learning materials related to SpringCloud Reply [python] Get a full set of 0 basic Python knowledge manual Reply [2020] Get 2020java-related interview questions tutorial Reply [Join group] You can join the technical exchange group related to the terminal R&D department read more Before using Spring's BeanUtils, it is recommended that you first understand these pitfalls! lazy-mock, a tool for lazy people to generate backend mock data Early adopters on Huawei Hongmeng OS, my first "hello world", take off! ByteDance side: Is i++ thread-safe? An accident caused by SQL, the colleague was fired directly! ! Too heartbroken! Investigate that the CPU of Alibaba Cloud ECS reaches 100% A powerful swagger-ui written by vue, a bit showy (with open source address) Believe in yourself, there is nothing you can't do, only unexpected, you can get more than technology here! Click "Looking" to support Brother Xiao, thank you