High-performance log desensitization component: already supports log4j2 and logback plug-ins

Project introduction

Log desensitization is a common security requirement. The ordinary method based on tool class methods is too intrusive to the code and is particularly troublesome to write.

Sensitive provides an annotation-based approach and has built-in common desensitization methods to facilitate development.

It also supports common log desensitization plug-ins such as logback and log4j2.

Extended reading

How to elegantly desensitize financial user sensitive data?

What should I do if I cannot quickly locate the information based on the log desensitization?

log4j2 plug-in unified desensitization

Description

The above method is very suitable for new projects and is promoted according to the corresponding specifications.

However, many financial companies have many historical legacy projects, or use irregularities, such as using maps, etc., which causes the above method to consume a lot of time during desensitization technology modification, and the cost of backtracking is very high.

Is there any way to handle it directly in the log layer?

log4j2 Rewrite

We can use the desensitization strategy uniformly based on log4j2 RewritePolicy.

Note: If you use the slf4j interface, it is also supported when implemented as log4j2.

Getting started

maven introduction

Introducing core desensitization package.

<dependency>
    <groupId>com.github.houbb</groupId>
    <artifactId>sensitive-log4j2</artifactId>
    <version>1.6.1</version>
</dependency>

There are also other general projects, such as log4j2 package:

<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-api</artifactId>
    <version>${log4j2.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>${log4j2.version}</version>
</dependency>

log4j2.xml configuration

Examples are as follows:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN" packages = "com.github.houbb.sensitive.log4j2.layout">

    <Properties>
        <Property name="DEFAULT_PATTERN">%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n</Property>
        <Property name="DEFAULT_CHARSET">UTF-8</Property>
    </Properties>

    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <SensitivePatternLayout/>
        </Console>
    </Appenders>

    <Loggers>
        <Root level="DEBUG">
            <AppenderRef ref="Console"/>
        </Root>
    </Loggers>

</Configuration>

A few steps:

  1. Specify package as packages = "com.github.houbb.sensitive.log4j2.layout"

  2. According to the log4j2 layout specification, specify the Layout strategy as SensitivePatternLayout

Test

Normal log printing:

private static final String TEST_LOG = "mobile:13088887777; bankCard:6217004470007335024, email:[email protected], amount:123.00, " +
        "IdNo:340110199801016666, name1:Li Ming, name2:Li Xiaoming, name3:Li Zetoming, name4:Shandong Xiaoli Shun" +
        ", birthday:20220517, GPS:120.882222, IPV4:127.0.0.1, address: No. 888, Xuhui District, Shanghai, China;";

logger.info(TEST_LOG);

The automatic desensitization effect is as follows:

01:37:28.010 [main] INFO com.github.houbb.sensitive.test.log4j2.Log4j2AndSlf4jLayoutTest - mobile:130****7777|9FC4D36D63D2B6DC5AE1297544FBC5A2; bankCard:6217************ *5024|444F49289B30944AB8C6C856AEA21180, email:mahu*****@qq.com|897915594C94D981BA86C9E83ADD449C, amount:123.00, IdNo:340110199801016666, name1:Li Ming, name2:Li Xiaoming , name3:Li Zetomorrow, name4:Shandong Oguri Shun, birthday :20220517, GPS:120.882222, IPV4:127.0.0.1, address: Xu******, Shanghai, China|821A601949B1BD18DCBAAE27F2E27147;

ps: This is to demonstrate various effects. The actual default corresponding strategies are 1, 2, 3, 4, and 9.

log4j2 configuration customization

In order to meet various user scenarios, the configurability of the SensitivePatternLayout strategy was introduced in V1.6.0.

Users can specify it through the chars-scan-config.properties configuration file under application resources.

Default configuration

In the log4j2 configuration, the SensitivePatternLayout configuration defaults to:

chars.scan.prefix=::,,'"'"=| + ()()
chars.scan.scanList=1,2,3,4,9
chars.scan.replaceList=1,2,3,4,9
chars.scan.defaultReplace=12
chars.scan.replaceHash=md5
chars.scan.whiteList=""

Attribute description

Property description for the SensitivePatternLayout strategy.

Property Description Default value Remarks
prefix Requires desensitization Matching prefix of information ::,,'"'"= + ()() and English vertical bar Reduce the false positive rate
replaceHash Hash strategy mode md5 Supports two modes of md5/none
scanList Sensitive scan policy list 1,2,3,4 1~10 10 built-in sensitive information scanning strategies, separate multiple with commas
replaceList Sensitive replacement strategy list 1,2,3,4 1~ 10 Built-in 10 sensitive information replacement strategies, separate multiple with commas
defaultReplace Sensitive replacement Default policy 12 1~13 There are 13 built-in sensitive information replacement strategies, specify one. When there is no match in the list, this is used by default
whiteList whitelist “ Whitelist information that you want to skip processing

The built-in strategies for 1-13 are described as follows:

Strategy Identity Description
1 Mobile phone number
2 ID card
3 Bank card
4 Mailbox
5 Chinese name
6 Date of birth
7 GPS
8 IPV4
9 Address
10 Passport
11 Match any unmasked
12 Match any half-masked
13 Match any full mask

Shortcomings

Compared with the plug-in of log4j2, the policy customization here is indeed not powerful, but it can satisfy 99% of desensitization scenarios.

In the future, we will have time to consider the idea of plugins similar to log4j2 to implement more flexible custom strategies.

logback desensitization plug-in

Description

For user convenience, v1.6.0 begins to support logback plug-in mode.

Getting started

maven introduction

Introduce logback dependency package

<dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-classic</artifactId>
    <version>${logback.version}</version>
</dependency>

Specify logback.xml configuration

<configuration>
    <!-- Based on converter -->
    <conversionRule conversionWord="sensitive" converterClass="com.github.houbb.sensitive.logback.converter.SensitiveLogbackConverter" />
    <!-- Use converter -->
    <appender name="STDOUTConverter" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %sensitive%n</pattern>
        </encoder>
    </appender>

    <!-- Use layout -->
    <appender name="STDOUTLayout" class="ch.qos.logback.core.ConsoleAppender">
        <layout class="com.github.houbb.sensitive.logback.layout.SensitiveLogbackLayout">
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </layout>
    </appender>

    <!-- Set the root log level to DEBUG and output the log to the console -->
    <root level="DEBUG">
        <appender-ref ref="STDOUTConverter"/>
        <appender-ref ref="STDOUTLayout"/>
    </root>
</configuration>

There are two modes supported here: Converter and Layout. You can choose either one.

It is recommended to use SensitiveLogbackConverter to desensitize log content.

Log effect

The decryption effect is similar to log4j2, as follows:

01:42:32.579 [main] INFO c.g.h.sensitive.test2.LogbackMain - mobile:130****7777|9FC4D36D63D2B6DC5AE1297544FBC5A2; bankCard:6217************5024|444F49289B30944AB8C6C856AEA21180, email : mahu*****@qq.com|897915594C94D981BA86C9E83ADD449C, amount:123.00, " + "IdNo:340110199801016666, name1:Li Ming, name2:Li Xiaoming, name3:Li Zetoming, name4:Shandong Oguri Shun" + ", birthday: 20220517, GPS:120.882222, IPV4:127.0.0.1, address: Xu******, Shanghai, China|821A601949B1BD18DCBAAE27F2E27147;

Configuration properties

Same as log4j2, no details will be given here.

Summary

The method based on the log plug-in is more convenient, easy to promote and use the project, and has greater practicality.

The open source address of the project

https://github.com/houbb/sensitive