sentinel and sentinel-golang make your service rock solid

First of all, let’s talk about Sentinel. This is a flow control software that has been used internally by Alibaba for many years and evolved. It has withstood the test of Double Eleven for many years. It was first used for the Java language and the Sentinel-golang version was launched in 2020. Official documentation: https://sentinelguard.io/zh-cn/docs/introduction.html […]

yolov5-v7.0 detect.py annotation

# YOLOv5 by Ultralytics, GPL-3.0 license import argparse import os import platform importsys from pathlib import Path import torch FILE = Path(__file__).resolve() ROOT = FILE.parents[0] # YOLOv5 root directory if str(ROOT) not in sys.path: sys.path.append(str(ROOT)) # add ROOT to PATH ROOT = Path(os.path.relpath(ROOT, Path.cwd())) # relative #Import modules under relative paths from models.common import DetectMultiBackend […]

rsync+inotify real-time data synchronization

1. Related introduction 1. rsync (remote synchronize) rsync is a remote data synchronization tool under Liunx/Unix. It can quickly synchronize files and directories between multiple hosts through LAN/WAN. There are generally two ways to synchronize files between Linux, namely rsync and scp. scp is equivalent to copying and pasting. If the file does not exist, […]

Pytest’s use case tagging and test execution

Pytest can pass data into the test function through tags, and can also filter the executed use cases through tags. 1. Built-in tags in pytest To use pytest marks, you need to use pytest.mark.marks. There are many built-in marks in pytest to cope with various test scenarios. 1.1, pytest.mark.parametrize: mark for use case parameterization Parametrize […]

Micro front-end qiankun use cases and source code

Qiankun 1. Install Qiankun: 2. Create the main application: 3. Create a sub-application: 4. Sub-application adapter configuration: 5. Application communication: 1Global status management: 2 The sub-application calls the main application method: 6. Build and deploy: 1 Main application build: 2 sub-app builds: 3Deploy the main application and sub-applications: 4 Ensure communication between the main application […]

[Python] Pandas data analysis

? Article directory ? Rename column name rename Deduplication function drop_duplicates() Sorting sort_values() merge merge() Rank rank() Group by() transpose transform() String matching contains() capitalize() Conditional judgment where() Conditional filtering query() Insert data insert() Accumulate cumsum() Random sampling sample() Rename column name rename When working with data using pandas, sometimes you need to rename the […]

[Vue3 source code analysis] computed

export function computed<T>( getter: ComputedGetter<T>, debugOptions?: DebuggerOptions ): ComputedRef<T> export function computed<T>( options: WritableComputedOptions<T>, debugOptions?: DebuggerOptions ): WritableComputedRef<T> export function computed<T>( getterOrOptions: ComputedGetter<T> | WritableComputedOptions<T>, debugOptions?: DebuggerOptions, isSSR = false ) {<!– –> //Formatting parameters let getter: ComputedGetter<T> let setter: ComputedSetter<T> // If what is passed is a function (that is, functional writing), then it […]

Mail function-SMTP protocol mail sending in python

Article directory 1. SMTP protocol email preparation 2. smtplib module 1. Use smtplib to encapsulate a mail class 2.Send email Replenish 1. SMTP protocol email preparation Need an smtp server 2. smtplib module The smtplib module is a module that comes with python 1. Use smtplib to encapsulate a mail class import smtplib import logging […]

Service log performance tuning, huge pit caused by log

Onlythosewhohavebeenseverelybeatenbyonlineserviceproblemsunderstandhowimportantlogsare! Whoisinfavorandwhoisagainst?Ifyoufeelthesameway,congratulationsonbecomingasocialperson:) Theimportanceoflogstotheprogramisself-evident.Itislightweight,simple,andrequiresnobraineffort.Itcanbefoundeverywhereintheprogramcodeandhelpsustroubleshootandlocateproblems.However,theseeminglyinconspicuouslogshidevarious”pits”.Ifusedimproperly,notonlywilltheynothelpus,buttheywillbecomeservice”killers”. Thisarticlemainlyintroducesthe”pits”causedbyimproperuseofproductionenvironmentlogsandhowtoavoidthem,whichisespeciallyobviousinhigh-concurrencysystems.Atthesametime,asetofimplementationsolutionsareprovidedtoallowprogramsandlogsto”coexistharmoniously”. Avoidpitfallsandpointnorth Inthischapter,Iwillintroducethelogproblemsencounteredonlineinthepast,andanalyzetherootcausesoftheproblemsonebyone. Irregularlogwritingformat //Format1 log.debug(“getuser”+uid+”fromDBisEmpty!”); //Format2 if(log.isdebugEnable()){ log.debug(“getuser”+uid+”fromDBisEmpty!”); } //Format3 log.debug(“getuser{}fromDBisEmpty!”,uid); Ibelievethateveryonehasseentheabovethreewritingmethodsmoreorlessintheprojectcode,sowhatarethedifferencesbetweenthembefore,andwhatimpactwilltheyhaveonperformance? IfyouturnofftheDEBUGloglevelatthistime,thedifferenceappears: Format1stillneedstoperformstringconcatenation,evenifitdoesnotoutputlogs,itisawaste. Thedisadvantageofformat2isthatadditionaljudgmentlogicneedstobeadded,whichaddswastecodeandisnotelegantatall. Therefore,format3isrecommended.Itwillbedynamicallysplicedonlyduringexecution.Afterturningoffthecorrespondingloglevel,therewillbenoperformanceloss. Productionprintingalargenumberoflogsconsumesperformance Havingasmanylogsaspossiblecanstringtogetheruserrequests,makingiteasiertodeterminethelocationoftheproblematiccode.Duetothecurrentdistributedsystemandcomplexbusiness,thelackofanylogisahugeobstacleforprogrammerstolocateproblems.Therefore,programmerswhohavesufferedfromproductionproblemsmustlogasmuchaspossibleduringthecodedevelopmentprocess. Inordertolocateandfixproblemsassoonaspossibleiftheyoccuronlineinthefuture,programmerswilllogasmanykeylogsaspossibleduringtheprogrammingimplementationphase.Aftergoingonline,theproblemcanbequicklylocated,butthentherewillbenewchallenges:withtherapiddevelopmentofthebusiness,uservisitscontinuetoincrease,andthesystempressureisincreasing.Atthistime,therearealargenumberofINFOlogsonline,especiallyinDuringpeakperiods,alargenumberoflogdiskwritesconsumeserviceperformance. Thenthisbecomesagametheory.Iftherearemorelogs,itwillbeeasiertotroubleshootproblems,butserviceperformancewillbe”eaten”.Iftherearefewerlogs,servicestabilitywillhavenoimpact,buttroubleshootingwillbedifficult,andprogrammerswillsuffer. Question:WhydoestheperformancesufferiftherearetoomanyINFOlogs(CPUusageisveryhighatthistime)? Rootcause1:SynchronousprintinglogdiskI/Ohasbecomeabottleneck,resultinginalargenumberofthreadblocks Itisconceivablethatifthelogsarealloutputtothesamelogfileandmultiplethreadsarewritingtothefile,itwillbechaotic.Thesolutionistoaddlockstoensurethatthelogfileoutputwillnotbeconfused.Ifitisduringpeakperiods,lockcontentionwillundoubtedlyconsumethemostperformance.Whenonethreadgrabsthelock,otherthreadscanonlyblockandwait,whichseriouslydragsdowntheuserthread.Theperformanceisthattheupstreamcalltimesoutandtheuserfeelsstuck. Thefollowingisthestackwhenthethreadisstuckwritingafile: StackTraceis: java.lang.Thread.State:BLOCKED(onobjectmonitor) atorg.apache.logging.log4j.core.appender.OutputStreamManager.writeBytes(OutputStreamManager.java:352) -waitingtolock<0x000000063d668298>(aorg.apache.logging.log4j.core.appender.rolling.RollingFileManager) atorg.apache.logging.log4j.core.layout.TextEncoderHelper.writeEncodedText(TextEncoderHelper.java:96) atorg.apache.logging.log4j.core.layout.TextEncoderHelper.encodeText(TextEncoderHelper.java:65) atorg.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:68) atorg.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:32) atorg.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:228) ….. SoisitokaytoreducetheINFOlogonline?Similarly,theamountofERRORlogscannotbeunderestimated.Ifalargeamountofabnormaldataappearsonlineoralargenumberofdownstreamtimeoutsoccur,alargenumberofERRORlogswillbegeneratedinstantly.Atthistime,thediskI/Owillstillbefull,causinguserthreadstoblock. Question:Assumingyoudon’tcareaboutINFOtroubleshooting,istherenoperformanceproblemifyouonlyprintERRORlogsinproduction? Rootcause2:ThreadBlockcausedbylogprintingexceptionstackunderhighconcurrency Onceuponatime,alargenumberoftimeoutsoccurredonlineanddownstream,andtheexceptionswereallcaughtbyourservice.Fortunately,thedisasterrecoverydesignanticipatedthatthisproblemwouldoccur,andimplementedabottom-linelogic.Fortunately,therewasnoimpact,buttheserverstarted”Teachpeoplehowtobeagoodperson.”Theonlinemonitoringstartedtoalarm.TheCPUusageincreasedtoofast,andtheCPUincreaseddirectlyto90%+.Atthistime,weurgentlyexpandedthecapacitytostoptheloss,andfoundamachinetopulldownthetrafficandpullthestack. Aftercheckingthedumpedthreadstack,combinedwiththeflameregressionanalysis,mostoftheready-madethreadsarestuckatthefollowingstacklocations: StackTraceis: java.lang.Thread.State:BLOCKED(onobjectmonitor) atjava.lang.ClassLoader.loadClass(ClassLoader.java:404) -waitingtolock<0x000000064c514c88>(ajava.lang.Object) atsun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) atjava.lang.ClassLoader.loadClass(ClassLoader.java:357) atorg.apache.logging.log4j.core.impl.ThrowableProxyHelper.loadClass(ThrowableProxyHelper.java:205) atorg.apache.logging.log4j.core.impl.ThrowableProxyHelper.toExtendedStackTrace(ThrowableProxyHelper.java:112) atorg.apache.logging.log4j.core.impl.ThrowableProxy.(ThrowableProxy.java:112) atorg.apache.logging.log4j.core.impl.ThrowableProxy.(ThrowableProxy.java:96) atorg.apache.logging.log4j.core.impl.Log4jLogEvent.getThrownProxy(Log4jLogEvent.java:629) … […]