io.seata.core.exception.RmTransactionException: RPC Timeout

Project scenario: Using seata for microservice distributed transaction management Problem description A large number of Seata transaction exceptions occurred online, io.seata.core.exception.RmTransactionException: RPC Timeout java.sql.SQLException: io.seata.core.exception.RmTransactionException: RPC Timeout at io.seata.rm.datasource.ConnectionProxy.recognizeLockKeyConflictException(ConnectionProxy.java:161) at io.seata.rm.datasource.ConnectionProxy.processGlobalTransactionCommit(ConnectionProxy.java:252) at io.seata.rm.datasource.ConnectionProxy.doCommit(ConnectionProxy.java:230) at io.seata.rm.datasource.ConnectionProxy.lambda$commit$0(ConnectionProxy.java:188) at io.seata.rm.datasource.ConnectionProxy$LockRetryPolicy.execute(ConnectionProxy.java:344) at io.seata.rm.datasource.ConnectionProxy.commit(ConnectionProxy.java:187) at io.seata.rm.datasource.exec.AbstractDMLBaseExecutor.lambda$executeAutoCommitTrue$2(AbstractDMLBaseExecutor.java:138) at io.seata.rm.datasource.ConnectionProxy$LockRetryPolicy.doRetryOnLockConflict(ConnectionProxy.java:356) at io.seata.rm.datasource.exec.AbstractDMLBaseExecutor$LockRetryPolicy.execute(AbstractDMLBaseExecutor.java:180) at io.seata.rm.datasource.exec.AbstractDMLBaseExecutor.executeAutoCommitTrue(AbstractDMLBaseExecutor.java:136) at io.seata.rm.datasource.exec.AbstractDMLBaseExecutor.doExecute(AbstractDMLBaseExecutor.java:82) at io.seata.rm.datasource.exec.BaseTransactionalExecutor.execute(BaseTransactionalExecutor.java:125) at io.seata.rm.datasource.exec.ExecuteTemplate.execute(ExecuteTemplate.java:137) at io.seata.rm.datasource.exec.ExecuteTemplate.execute(ExecuteTemplate.java:56) […]

Actual combat! Use Arthas to locate the timeout problem of the Spring Boot interface and let the application take off ~

Helloeveryone,IamBucaiChen~ Background Thecompanyhasachannelsystemthatisspecificallyusedtointerfacewiththird-partychannels.Ithasnobusinesslogicandmainlydoesworksuchasmessageconversionandparameterverification,whichservesasalinkbetweenthepreviousandthefollowing. Recently,Ihavebeenoptimizingtheresponsetimeoftheinterfaceandoptimizingthecode,butthetimestilldoesnotmeettherequirements;thereisastrangetime-consumingproblemofabout100ms.Aftertherequestprocessingtimeisprintedintheinterface,itisstilldifferentfromthecaller’sresponsetime.Thedifferenceisabout100ms.Forexample,theprogramrecords150ms,butthecaller’swaitingtimeisabout250ms. Thedetailedpositioning&solutionprocessatthattimeisrecordedbelow(infact,thesolutionisverysimple,thekeyliesinhowtopositionandfindasolutiontotheproblem) Locationprocess Analyzingcode Thechannelsystemisacommonspring-bootwebproject,usingintegratedtomcat.Afteranalyzingthecode,Ifoundthattherewasnothingspecial,nospecialfiltersorinterceptors,soIinitiallyruledoutthatitwasabusinesscodeproblem. Analyzethecallingprocess Afterthisproblemoccurred,wefirstconfirmedthecallingprocessoftheinterface.Sinceitisaninternaltest,therearefewercallprocesses. Nginx-reverseproxy->channelsystem Thecompanyisacloudserver,andthenetworkisalsothecloud’sintranet.Sincethecauseoftheproblemisnotclear,weusethetroubleshootingmethodtofirstconfirmwhetherthereisaproblemwiththeservernetwork. FirstconfirmwhetherthereisaproblemwiththesendingendtoNginxHost: [jboss@VM_0_139_centos~]$ping10.0.0.139 PING10.0.0.139(10.0.0.139)56(84)bytesofdata. 64bytesfrom10.0.0.139:icmp_seq=1ttl=64time=0.029ms 64bytesfrom10.0.0.139:icmp_seq=2ttl=64time=0.041ms 64bytesfrom10.0.0.139:icmp_seq=3ttl=64time=0.040ms 64bytesfrom10.0.0.139:icmp_seq=4ttl=64time=0.040ms Judgingfromthepingresults,thereisnoproblemwiththedelayfromthesendingendtotheNginxhost.Next,checkthenetworkfromNginxtothechannelsystem. #Sincethelogisok,Iwillcopytheabovelogdirectlyhere. [jboss@VM_0_139_centos~]$ping10.0.0.139 PING10.0.0.139(10.0.0.139)56(84)bytesofdata. 64bytesfrom10.0.0.139:icmp_seq=1ttl=64time=0.029ms 64bytesfrom10.0.0.139:icmp_seq=2ttl=64time=0.041ms 64bytesfrom10.0.0.139:icmp_seq=3ttl=64time=0.040ms 64bytesfrom10.0.0.139:icmp_seq=4ttl=64time=0.040ms Judgingfromthepingresults,thereisnoproblemwiththenetworkdelayfromNginxtothechannelsystemserver. Sincethenetworkseemstobefine,youcancontinuethetroubleshootingmethod,cutoffNginx,andconnecttheclientdirectlytotheserverofthechannelsystemthroughtheloopbackaddress(localhost)toavoidgoingthroughthenetworkcard/dns,narrowthescopeoftheproblemandseeifitcanberepaired.Now(thisapplicationandaddressweresimulatedbymelater,andthetestwasanemptyinterface): [jboss@VM_10_91_centostmp]$curl-w”@curl-time.txt”http://127.0.0.1:7744/send success http:200 dns:0.001s redirect:0.000s time_connect:0.001s time_appconnect:0.000s time_pretransfer:0.001s time_starttransfer:0.073s size_download:7bytes speed_download:95.000B/s ———- time_total:0.073stotalrequesttime Judgingfromthecurllog,callinganemptyinterfacethroughtheloopbackaddressalsotakes73ms.Thisisstrange.Itskipsalltheintermediatecallingnodes(includingfilters&interceptors,etc.)anddirectlyrequeststheapplicationofanemptyinterface,whichtakes73ms.Pleaserequestagainandsee: [jboss@VM_10_91_centostmp]$curl-w”@curl-time.txt”http://127.0.0.1:7744/send success http:200 dns:0.001s redirect:0.000s time_connect:0.001s time_appconnect:0.000s time_pretransfer:0.001s time_starttransfer:0.003s size_download:7bytes speed_download:2611.000B/s ———- […]

keil-flash timeout problem

1Problem description This new board basically adopts the same design as the previous board, and the resistors and capacitors have basically remained unchanged. However, after welding the board and using the program of the previous board, a flash timeou problem occurred. How to solve this? Phenomena 1 Online prompts say that the chip may be […]

Dubbo Practical Chapter: dubbo timeout and retry

Dubbo timeout retry overview Timeout can be set on both the consumer and provider sides. Timeout priority: consumer method level>provider method level>consumer interface level>provider interface level>consumer level>provider level When the consumer call times out, a retry call will be triggered. The configuration attribute corresponding to retries is retries. The default number of retries is 2. […]

HttpClient timeout settings setConnectionTimeout and setSoTimeout

An http request must have three stages, one: establishment of connection; two: data transmission; three, disconnection. HTTP three-way handshake and four-way wave http communicates based on TCP/IP. TCP establishes a connection through 3 handshakes and finally terminates the communication with 4 waves. Zhihu explains the three-way handshake and four-way wave as follows: Author: Zhihu user […]

Looking for the other half in the Kingdom of Mathematics: finding the sum of true factors of an integer, algorithm optimization solution to the timeout problem

The Kingdom of Mathematics looks for the other half, finds the sum of the true factors of an integer, and optimizes the algorithm to solve the timeout problem. (This note is suitable for coders who are familiar with python strings and lists) [The details of learning are a joyful process] Python official website: python cutting […]

Full analysis of all timeout exceptions in zookeeper

ZooKeeper’s timeout exceptions include two types: 1) The client’s readTimeout causes the connection to be lost. 2) The server session timeout sessionTimeout causes the client connection to fail. The client’s readTimeout causes the connection to be lost The readTimeout of the ZooKeeper client cannot be displayed and is calculated based on the session timeout: 1. […]

Java multi-threaded programming-Wait timeout control

Foreword: This article is based on my personal understanding of Chapter 5 of “Java Multi-Threaded Programming Practical Guide”. Because Chapter 5 contains a lot of content, it has been split into many articles. The source code is excerpted from the author’s source code, and my own understanding will be added to the source code. The […]

Macro tasks, micro tasks, event loop event loop and promise, setTimeout, async, nextTick [Super detailed example explanation]

Table of Contents js single thread Macro tasks: tasks queued for execution on the main thread, executed sequentially Macro task macrotask: setTimeout, setInterval timing events, Ajax, DOM events, script execution, I/O operations, UI rendering, etc. Microtask: a task that does not enter the main thread but enters the “microtask list” Microtask (asynchronous): Promise, async/await, alert […]