Questions
If the user reports that the online service request is unresponsive, follow the steps below to troubleshoot.
1. Confirm server memory usage
Execute the free
command to check whether the server memory is normal.
total used free shared buff/cache available Mem: 16256764 7583240 2177924 2901216 6495600 5437168 Swap: 2097148 1280 2095868
It looks like the server memory is normal.
2. Confirm server disk usage
Execute the df -h
command to check the disk usage.
File system capacity used available used% mount point xxxxxxxx 7.8G 0 7.8G 5% /xx ...
Disk space also appears to be adequate.
Third, check which process takes up more memory
Enter the top
command, and then press shift p
, the one at the top is the one that takes up the most memory.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME + COMMAND 32297 xxxxxx 20 0 109.7g 4.7g 4.0g S 46.2 30.2 160554:47 java
Here you can get the pid that occupies the most memory.
Fourth, view the stack information of the thread
Execute jstack
, the status of the thread will be displayed.
Full thread dump OpenJDK 64-Bit Server VM (17 + 35-2724 mixed mode, sharing): Threads class SMR info: _java_thread_list=0x00007f24080015b0, length=41, elements={ 0x00007f266c0899e0, 0x00007f266c08b050, 0x00007f266c091d60, 0x00007f266c093280, 0x00007f266c0948a0, 0x00007f266c0964a0, 0x00007f266c097c00, 0x00007f266c0a12a0, 0x00007f266c0befe0, 0x00007f266c0fa7d0, 0x00007f266c23c6b0, 0x00007f266c26dab0, 0x00007f266c425430, 0x00007f266c5ce320, 0x00007f266c6a3ca0, 0x00007f266c6e9790, 0x00007f266c9297f0, 0x00007f266c92d740, 0x00007f266c95d4a0, 0x00007f266c027640, 0x00007f23b8005600, 0x00007f23dc55f2c0, 0x00007f23dc5705c0, 0x00007f23bc0b8bc0, 0x00007f23bc15ebf0, 0x00007f23bc1e0cd0, 0x00007f23b800b850, 0x00007f23b800c2e0, 0x00007f23b800d370, 0x00007f23b800e9d0, 0x00007f23b800fbd0, 0x00007f23b8010ee0, 0x00007f23b8012220, 0x00007f23c4439b80, 0x00007f23c4006500, 0x00007f23c46ae980, 0x00007f23544eeb00, 0x00007f237c406d00, 0x00007f23a829ba30, 0x00007f23c46b2560, 0x00007f2408000a90 } "Reference Handler" #2 daemon prio=10 os_prio=0 cpu=1463269.24ms elapsed=23238405.12s tid=0x00007f266c0899e0 nid=0x7e38 waiting on condition [0x00007f26542be000] java.lang.Thread.State: RUNNABLE at java.lang.ref.Reference.waitForReferencePendingList(java.base@17/Native Method) at java.lang.ref.Reference.processPendingReferences(java.base@17/Reference.java:253) at java.lang.ref.Reference$ReferenceHandler.run(java.base@17/Reference.java:215)
If there is a deadlock, there will be the following:
java.lang.Thread.State: BLOCKED at xxxxxx1 - waiting to lock <0xxxxxx> - locked <1xxxxx> java.lang.Thread.State: BLOCKED at xxxxxx2 - waiting to lock <1xxxxxx> - locked <0xxxxx>
5. View heap memory usage
Execute the jmap -heap
command to see the usage of the heap memory.
Attaching to process ID 5064, please wait... Debugger attached successfully. Server compiler detected. JVM version is 25.271-b09 using thread-local object allocation. Parallel GC with 8 thread(s) Heap Configuration: MinHeapFreeRatio = 0 MaxHeapFreeRatio = 100 MaxHeapSize = 536870912 (512.0MB) NewSize = 268435456 (256.0MB) MaxNewSize = 268435456 (256.0MB) OldSize = 268435456 (256.0MB) NewRatio = 2 SurvivorRatio = 8 MetaspaceSize = 134217728 (128.0MB) CompressedClassSpaceSize = 260046848 (248.0MB) MaxMetaspaceSize = 268435456 (256.0MB) G1HeapRegionSize = 0 (0.0MB) Heap Usage: PS Young Generation Eden Space: capacity = 241172480 (230.0MB) used = 94437072 (90.06221008300781MB) free = 146735408 (139.9377899169922MB) 39.15748264478601% used From Space: capacity = 13107200 (12.5MB) used = 3045992 (2.9048843383789062MB) free = 10061208 (9.595115661621094MB) 23.23907470703125% used To Space: capacity = 12582912 (12.0MB) used = 0 (0.0MB) free = 12582912 (12.0MB) 0.0% used PS Old Generation capacity = 268435456 (256.0MB) used = 84914376 (80.98065948486328MB) free = 183521080 (175.01934051513672MB) 31.63307011127472% used 46153 interned Strings occupying 5019920 bytes.
Here you can see the memory usage of the Eden area, the from area, the to area, and the old generation. If the old generation reaches 99.99%, it means that the heap memory has overflowed.
6. View the top 10 object information in the heap memory
jmap -histo:live
Execute this command, you can see the top 10 object information of memory usage.
num #instances #bytes class name ---------------------------------------------- 1: 165329 18150792 [C 2: 163258 3918192 java.lang.String 3: 40481 3562328 java.lang.reflect.Method 4: 28014 2844936 [Ljava.lang.Object; 5: 67703 2166496 java.util.concurrent.ConcurrentHashMap$Node 6: 7919 2106384 [B 7: 17131 1934896java.lang.Class
If you see a class object written by yourself here, you may be able to find the problem.
Seven, analyze the problem of memory overflow
After determining which node has the problem, first cut off the traffic of the node. If the sixth step does not analyze what caused the memory overflow, you can troubleshoot according to the following steps.
1. Export the dump file jmap -dump:format=b,live,file=
Execute this command, you can export the file named The dump file of fileName.hprof
2. Analyzing the dump file Here is an introduction to using the mat tool to analyze the dump file.
- First click
window ---> preferences ---> memory Analyzer
of the mat tool, and checkkeep unreachable objects
, which will keep unreachable objects; - Click
file ---> open heap dump
, select the dump file just now, wait a few minutes, and the mat tool will generate a default report; - Problems will be listed in the default report, click
details
to see the details of the problem, usually the object with the problem will be listed; - Select the problematic object, right click
Merge Shortest Paths to GC Roots ---> exclude weak references
; - Then
Java Basics ---> Thread Details
, so that you can see the thread details, there will be error information in the thread details, you can locate which line of the code reported the error;
8. If the default report cannot locate the specific code according to the above method, you can try the following steps:
- Click on histogram on the overview page, and then sort by retained heap;
- Then select the object that is ranked first by default, right click
merge shortest path to GC Roots
; - Then select the thread,
Java Basics ---> Thread Details
, and look at the details one by one. Generally, you can locate the problem by looking at the first few.