Online Service Request No Response – Troubleshooting Process

Questions

If the user reports that the online service request is unresponsive, follow the steps below to troubleshoot.

1. Confirm server memory usage

Execute the free command to check whether the server memory is normal.

 total used free shared buff/cache available
 Mem: 16256764 7583240 2177924 2901216 6495600 5437168
 Swap: 2097148 1280 2095868

It looks like the server memory is normal.

2. Confirm server disk usage

Execute the df -h command to check the disk usage.

File system capacity used available used% mount point
xxxxxxxx 7.8G 0 7.8G 5% /xx
...

Disk space also appears to be adequate.

Third, check which process takes up more memory

Enter the top command, and then press shift p, the one at the top is the one that takes up the most memory.

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME + COMMAND
32297 xxxxxx 20 0 109.7g 4.7g 4.0g S 46.2 30.2 160554:47 java

Here you can get the pid that occupies the most memory.

Fourth, view the stack information of the thread

Execute jstack , the status of the thread will be displayed.

Full thread dump OpenJDK 64-Bit Server VM (17 + 35-2724 mixed mode, sharing):

Threads class SMR info:
_java_thread_list=0x00007f24080015b0, length=41, elements={
0x00007f266c0899e0, 0x00007f266c08b050, 0x00007f266c091d60, 0x00007f266c093280,
0x00007f266c0948a0, 0x00007f266c0964a0, 0x00007f266c097c00, 0x00007f266c0a12a0,
0x00007f266c0befe0, 0x00007f266c0fa7d0, 0x00007f266c23c6b0, 0x00007f266c26dab0,
0x00007f266c425430, 0x00007f266c5ce320, 0x00007f266c6a3ca0, 0x00007f266c6e9790,
0x00007f266c9297f0, 0x00007f266c92d740, 0x00007f266c95d4a0, 0x00007f266c027640,
0x00007f23b8005600, 0x00007f23dc55f2c0, 0x00007f23dc5705c0, 0x00007f23bc0b8bc0,
0x00007f23bc15ebf0, 0x00007f23bc1e0cd0, 0x00007f23b800b850, 0x00007f23b800c2e0,
0x00007f23b800d370, 0x00007f23b800e9d0, 0x00007f23b800fbd0, 0x00007f23b8010ee0,
0x00007f23b8012220, 0x00007f23c4439b80, 0x00007f23c4006500, 0x00007f23c46ae980,
0x00007f23544eeb00, 0x00007f237c406d00, 0x00007f23a829ba30, 0x00007f23c46b2560,
0x00007f2408000a90
}

"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=1463269.24ms elapsed=23238405.12s tid=0x00007f266c0899e0 nid=0x7e38 waiting on condition [0x00007f26542be000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.ref.Reference.waitForReferencePendingList(java.base@17/Native Method)
        at java.lang.ref.Reference.processPendingReferences(java.base@17/Reference.java:253)
        at java.lang.ref.Reference$ReferenceHandler.run(java.base@17/Reference.java:215)

If there is a deadlock, there will be the following:

java.lang.Thread.State: BLOCKED
     at xxxxxx1
     - waiting to lock <0xxxxxx>
     - locked <1xxxxx>
java.lang.Thread.State: BLOCKED
     at xxxxxx2
     - waiting to lock <1xxxxxx>
     - locked <0xxxxx>

5. View heap memory usage

Execute the jmap -heap command to see the usage of the heap memory.

Attaching to process ID 5064, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.271-b09

using thread-local object allocation.
Parallel GC with 8 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 0
   MaxHeapFreeRatio = 100
   MaxHeapSize = 536870912 (512.0MB)
   NewSize = 268435456 (256.0MB)
   MaxNewSize = 268435456 (256.0MB)
   OldSize = 268435456 (256.0MB)
   NewRatio = 2
   SurvivorRatio = 8
   MetaspaceSize = 134217728 (128.0MB)
   CompressedClassSpaceSize = 260046848 (248.0MB)
   MaxMetaspaceSize = 268435456 (256.0MB)
   G1HeapRegionSize = 0 (0.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 241172480 (230.0MB)
   used = 94437072 (90.06221008300781MB)
   free = 146735408 (139.9377899169922MB)
   39.15748264478601% used
From Space:
   capacity = 13107200 (12.5MB)
   used = 3045992 (2.9048843383789062MB)
   free = 10061208 (9.595115661621094MB)
   23.23907470703125% used
To Space:
   capacity = 12582912 (12.0MB)
   used = 0 (0.0MB)
   free = 12582912 (12.0MB)
   0.0% used
PS Old Generation
   capacity = 268435456 (256.0MB)
   used = 84914376 (80.98065948486328MB)
   free = 183521080 (175.01934051513672MB)
   31.63307011127472% used

46153 interned Strings occupying 5019920 bytes.

Here you can see the memory usage of the Eden area, the from area, the to area, and the old generation. If the old generation reaches 99.99%, it means that the heap memory has overflowed.

6. View the top 10 object information in the heap memory

jmap -histo:live | head -n 10 Execute this command, you can see the top 10 object information of memory usage.

num #instances #bytes class name
----------------------------------------------
   1: 165329 18150792 [C
   2: 163258 3918192 java.lang.String
   3: 40481 3562328 java.lang.reflect.Method
   4: 28014 2844936 [Ljava.lang.Object;
   5: 67703 2166496 java.util.concurrent.ConcurrentHashMap$Node
   6: 7919 2106384 [B
   7: 17131 1934896java.lang.Class

If you see a class object written by yourself here, you may be able to find the problem.

Seven, analyze the problem of memory overflow

After determining which node has the problem, first cut off the traffic of the node. If the sixth step does not analyze what caused the memory overflow, you can troubleshoot according to the following steps.

1. Export the dump file jmap -dump:format=b,live,file=.hprof Execute this command, you can export the file named The dump file of fileName.hprof

2. Analyzing the dump file Here is an introduction to using the mat tool to analyze the dump file.

  • First click window ---> preferences ---> memory Analyzer of the mat tool, and check keep unreachable objects, which will keep unreachable objects;
  • Click file ---> open heap dump, select the dump file just now, wait a few minutes, and the mat tool will generate a default report;
  • Problems will be listed in the default report, click details to see the details of the problem, usually the object with the problem will be listed;
  • Select the problematic object, right click Merge Shortest Paths to GC Roots ---> exclude weak references;
  • Then Java Basics ---> Thread Details, so that you can see the thread details, there will be error information in the thread details, you can locate which line of the code reported the error;

8. If the default report cannot locate the specific code according to the above method, you can try the following steps:

  • Click on histogram on the overview page, and then sort by retained heap;
  • Then select the object that is ranked first by default, right click merge shortest path to GC Roots;
  • Then select the thread, Java Basics ---> Thread Details, and look at the details one by one. Generally, you can locate the problem by looking at the first few.