A set of universal asynchronous processing solutions

  • Preface

  • Purpose

  • advantage

  • principle

  • components

  • Design Patterns

  • flow chart

  • database script

  • Asynchronous strategy

  • Security Level

  • Execution status

  • flow chart

  • apollo configuration

  • usage

  • Notice

  • Show results

  • github address

1Foreword

Good system design must adhere to the opening and closing principle. As the business continues to be iteratively updated, the core code will be constantly modified, and the probability of errors will greatly increase. However, most of the added functions are expanding the original functions. To ensure both performance and quality, we often use asynchronous thread pools for processing, but this adds a lot of uncertainties.

From this, I designed a set of general asynchronous processing SDK, which can easily implement various asynchronous processing

2Purpose

Asynchronous processing not only ensures that the method can be effectively executed without affecting the main process

More importantly, various cover-up methods ensure that data is not lost, thereby achieving ultimate consistency.

3 Advantages

Non-intrusive design, independent database, independent scheduled tasks, independent message queue, independent manual execution interface (unified login authentication)

Using the spring transaction event mechanism, even if the asynchronous policy resolution fails, the business will not be affected.

If your method is running a transaction, it will wait until the transaction is committed or rolled back before processing the event.

Even if the transaction is submitted and the asynchronous policy parsing fails, we still have a back-up plan to execute (unless there is a problem with the database, message queue, or method bug)

4 principles

After the container initialization bean is completed, all methods are traversed and the methods annotated with @AsyncExec are cached.

Publish events through the AOP aspect when the method is running

Transaction event listening and processing asynchronous execution strategy

@TransactionalEventListener(fallbackExecution = true, phase = TransactionPhase.AFTER_COMPLETION)

  • fallbackExecution=true No transaction is running, the event is still processed

  • TransactionPhase.AFTER_COMPLETION handles events both after the transaction is committed and after the transaction is rolled back

5 components

  • kafka message queue

  • xxl job scheduled task

  • mysql database

  • spring aspect

  • vue interface

6 Design Patterns

  • Strategy

  • template method

  • dynamic proxy

7 flow chart

f5c733338b18e16ed088eba995ed379c.png

8Database Script

CREATE TABLE `async_req` (
  `id` bigint NOT NULL AUTO_INCREMENT COMMENT 'Primary key ID',
  `application_name` varchar(100) NOT NULL DEFAULT '' COMMENT 'Application name',
  `sign` varchar(50) NOT NULL DEFAULT '' COMMENT 'Method signature',
  `class_name` varchar(200) NOT NULL DEFAULT '' COMMENT 'Full path class name',
  `method_name` varchar(100) NOT NULL DEFAULT '' COMMENT 'Method name',
  `async_type` varchar(50) NOT NULL DEFAULT '' COMMENT 'Asynchronous strategy type',
  `exec_status` tinyint NOT NULL DEFAULT '0' COMMENT 'Execution status 0: Initialization 1: Execution failed 2: Execution successful',
  `exec_count` int NOT NULL DEFAULT '0' COMMENT 'Number of executions',
  `param_json` longtext COMMENT 'Request parameters',
  `remark` varchar(200) NOT NULL DEFAULT '' COMMENT 'Business description',
  `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Creation time',
  `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'update time',
  PRIMARY KEY (`id`) USING BTREE,
  KEY `idx_applocation_name` (`application_name`) USING BTREE,
  KEY `idx_exec_status` (`exec_status`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='Asynchronous processing of requests';

CREATE TABLE `async_log` (
  `id` bigint NOT NULL AUTO_INCREMENT COMMENT 'Primary key ID',
  `async_id` bigint NOT NULL DEFAULT '0' COMMENT 'Asynchronous request ID',
  `error_data` longtext COMMENT 'Execution error message',
  `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Creation time',
  PRIMARY KEY (`id`) USING BTREE,
  KEY `idx_async_id` (`async_id`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='Asynchronous processing log';

9Asynchronous Strategy

fe1bf0492376e57e19bca6977c02f07b.png

10 security level

4411af7ee0e0d272bfdfba2348b090e3.png

11 execution status

dc12906b518dfb2abc5e2cb825308bf8.png

12 flow chart

f8f72daadf7b07dbaecfc97aa2771c36.png
b96bd9389eff2e3378790296d0c38a3f.png
bcdddb8bfd4d2aff6d75365c2c220936.png

13apollo configuration

# switch: off by default
async.enabled=true

# Application Name
spring.application.name=xxx

# data source druid
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://127.0.0.1:3306/fc_async?useUnicode=true & amp;characterEncoding=utf-8 & amp;zeroDateTimeBehavior=convertToNull & amp;useSSL=false & amp;allowMultiQueries=true & amp;rewriteBatchedStatements=true
spring.datasource.username=user
spring.datasource.password=xxxx
spring.datasource.filters=config
spring.datasource.connectionProperties=config.decrypt=true;config.decrypt.key=yyy
#static address
spring.resources.add-mappings=true
spring.resources.static-locations=classpath:/static/


# The following configurations have default values
# Number of core threads
async.executor.thread.corePoolSize=10
#Maximum number of threads
async.executor.thread.maxPoolSize=50
# Queue capacity
async.executor.thread.queueCapacity=10000
# Active time
async.executor.thread.keepAliveSeconds=600

#Whether the record is deleted if the execution is successful: deleted by default
async.exec.deleted=true
 
# Custom queue name prefix: default application name
async.topic=${spring.application.name}
 
#Number of retry executions: default 5 times
async.exec.count=5
 
#Maximum number of queries to retry
async.retry.limit=100

# Compensate for the maximum number of queries
async.comp.limit=100

# Login interception: default false
async.login=false

14 Usage

1, Asynchronous switch

scm.async.enabled=true

2. Add annotations to methods that require asynchronous execution (must be spring proxy methods)

@AsyncExec(type = AsyncExecEnum.SAVE_ASYNC, remark = "Data Dictionary")

3. Manual address processing

http://localhost:8004/async/index.html

15 Note

1, Application name

spring.application.name

2, queue name

${async.topic:${spring.application.name}}_async_queue

Custom topic: async.topic=xxx

3. Make your business idempotent

4. An application shares a queue

self-produced and consumed

5. Scheduled tasks

  • Asynchronous retry of scheduled tasks (retry once every 2 minutes, configurable number of retries)

  • Asynchronous compensation scheduled tasks (compensated once an hour, created one hour ago)

16 effect display

c1761160ffd1ceb88c805aa4f81837b4.png

17github address

https://github.com/xiongyanokok/fc-async.git

Source: juejin.cn/post/7266087843239084090

a1c530bfbb87798941899e327cc7ac3d.gif



Stop using field injection in SpringBoot!


Interviewer: Why is Nacos so strong! Tell me about the implementation principle? I was confused. .


Since the project used the interface request merge, the efficiency has been doubled!


Stop encapsulating various Util tool classes, this god-level framework is worth having!


Why does IDEA not recommend using the @Autowired annotation?










I recently interviewed BAT and compiled an interview material "Java Interview BAT Clearance Manual", which covers Java core technology, JVM, Java concurrency, SSM, microservices, databases, data structures, etc.
How to get it: Click "Watching", follow the official account and reply 666 to receive it. More content will be provided one after another. 


See you tomorrow (ω