Let’s start with the lock. There is still a big difference between the built-in lock synchronized and jucl (java.util.concurrent.locks) lock. A very important difference is to use jstack to export the thread dump. Using the synchronized command, you can easily see that the lock is locked. Which thread holds it, but the jucl lock is indispensable. In the above, I used httpclient to write a connection pool, used a socket setting, setSoLinger(60), and then restarted the remote service in a test, which caused the system to be stuck. Check the connection of the machine, and found that many connections are In the close_wait state. Considering the restart of the remote service, all existing connections must be disconnected, and four waved hands are required to disconnect, so what is the state of close_wait?
Looking at the picture above, it is obvious that the remote server needs to restart, actively close the connection, and send a FIN. The local machine should immediately respond with an ACK when it receives it. Then, the local machine should immediately send a FIN, but it does not, and stays in the CLOSE_WAIT state. Reminiscent of the SocketConfig setSoLinger(60) used, it may be this setting that causes this problem. When hanging, use jstack to view the thread dump, as follows:
"Thread-2158" #2171 prio=5 os_prio=0 tid=0x00000000193fe800 nid=0xa3c80 waiting on condition [0x000000003e70e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000081eab2a8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.apache.http.pool.AbstractConnPool.getTotalStats(AbstractConnPool.java:509) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.formatStats(PoolingHttpClientConnectionManager.java:227) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:265) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at http.connectionPool.TestConnectionPool.lambda$main$1(TestConnectionPool.java:85) at http.connectionPool.TestConnectionPool$$Lambda$3/670035812.run(Unknown Source) at java.lang.Thread.run(Thread.java:748)
Many threads are waiting to enter the lock, so just check who holds 0x0000000081eab2a8, haha. But I checked the entire thread dump file, but I couldn’t find which thread held the lock, because ReentrantLock was used, which was different from synchronized. The jstack command can add parameters, -l print lock details -e print thread details, try jstack -l pid there is such a sentence
Locked ownable synchronizers:
Search again, you can use regular expressions to exclude most of the records
^.*(?<!wait for )<0x0000000081eb5388>.*$ Query lines that are not wait for
result:
"Thread-2623" #2636 prio=5 os_prio=0 tid=0x000000001aa7f000 nid=0xba4c8 runnable [0x000000005daed000] java.lang.Thread.State: RUNNABLE at java.net.DualStackPlainSocketImpl.close0(Native Method) at java.net.DualStackPlainSocketImpl.socketClose0(DualStackPlainSocketImpl.java:167) at java.net.AbstractPlainSocketImpl.socketPreClose(AbstractPlainSocketImpl.java:693) at java.net.AbstractPlainSocketImpl.close(AbstractPlainSocketImpl.java:530) - locked <0x00000000839b0280> (a java.lang.Object) at java.net.PlainSocketImpl.close(PlainSocketImpl.java:237) at java.net.SocksSocketImpl.close(SocksSocketImpl.java:1075) at java.net.Socket.close(Socket.java:1495) - locked <0x00000000839b00e0> (a java.lang.Object) - locked <0x00000000839b00c0> (a java.net.Socket) at sun.security.ssl.BaseSSLSocketImpl.close(BaseSSLSocketImpl.java:624) - locked <0x00000000839aff68> (a sun.security.ssl.SSLSocketImpl) at sun.security.ssl.SSLSocketImpl.closeSocket(SSLSocketImpl.java:1585) at sun.security.ssl.SSLSocketImpl.closeInternal(SSLSocketImpl.java:1723) at sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:1612) at org.apache.http.impl.BHttpConnectionBase.close(BHttpConnectionBase.java:334) at org.apache.http.impl.conn.LoggingManagedHttpClientConnection.close(LoggingManagedHttpClientConnection.java:81) at org.apache.http.impl.conn.CPoolEntry.closeConnection(CPoolEntry.java:70) at org.apache.http.impl.conn.CPoolEntry.close(CPoolEntry.java:96) at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:318) at org.apache.http.pool.AbstractConnPool.access$200(AbstractConnPool.java:69) at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:246) - locked <0x00000000d814f998> (a org.apache.http.pool.AbstractConnPool$2) at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:193) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:303) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at http.connectionPool.TestConnectionPool.lambda$main$1(TestConnectionPool.java:90) at http.connectionPool.TestConnectionPool$$Lambda$3/670035812.run(Unknown Source) at java.lang.Thread.run(Thread.java:748) Locked ownable synchronizers: - <0x0000000081eb5388> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
Found a thread! ! , you can see that it is closing and holds the lock of the connection pool. In this case, because setSoLinger is set, it will wait for 60s, so the threads requesting connections are piled up.