zun.common.exception.ZunException: Unexpected error: UnixHTTPConnectionPool(host=localhost’, port=None): Read timed out. (read timeout=60)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server [req-6ff62c08-fd25-4df6-8a81-d144956cbbd9 7db25ffff6314194822f5ada52569de5 ce76b778ff0b 453aa7b486f9934f694e default – -] Exception during message handling: zun.common.exception.ZunException: Unexpected error : UnixHTTPConnectionPool(host=’localhost’, port=None): Read timed out. (read timeout=60)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 445, in _make_request
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server six.raise_from(e, None)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “”, line 3, in raise_from
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 440, in _make_request
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server httplib_response = conn.getresponse()
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/lib64/python3.6/http/client.py”, line 1346, in getresponse
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server response.begin()
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/lib64/python3.6/http/client.py”, line 307, in begin
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server version, status, reason = self._read_status()
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/lib64/python3.6/http/client.py”, line 268, in _read_status
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server line = str(self.fp.readline(_MAXLINE + 1), “iso-8859-1”)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/lib64/python3.6/socket.py”, line 586, in readinto
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server return self._sock.recv_into(b)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/eventlet/greenio/base.py”, line 371, in recv_into
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server return self._recv_loop(self.fd.recv_into, 0, buffer, nbytes, flags)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/eventlet/greenio/base.py”, line 359, in _recv_loop
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server self._read_trampoline()
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/eventlet/greenio/base.py”, line 331, in _read_trampoline
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server timeout_exc=socket_timeout(timed out’))
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/eventlet/greenio/base.py”, line 210, in _trampoline
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server mark_as_closed=self._mark_as_closed)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/eventlet/hubs/init.py”, line 159, in trampoline
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server return hub.switch()
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py”, line 313, in switch
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server return self.greenlet.switch()
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server socket.timeout: timed out
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/requests/adapters.py”, line 449, in send
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server timeout=timeout
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 756, in urlopen
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py”, line 532, in increment
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server raise six.reraise(type(error), error, _stacktrace)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py”, line 735, in reraise
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server raise value
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 706, in urlopen
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server chunked=chunked,
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 447, in _make_request
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 337, in _raise_timeout
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server self, url, “Read timed out. (read timeout=%s)” % timeout_value
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host=localhost’, port=None): Read timed out. (read timeout=60)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/zun/common/utils.py”, line 226, in decorated_function
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/zun/compute/manager.py”, line 577, in container_show
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server container = self.driver.show(context, container)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/zun/container/docker/driver.py”, line 616, in show
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server response = docker.inspect_container(container.container_id)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/docker/utils/decorators.py”, line 19, in wrapped
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server return f(self, resource_id, *args, **kwargs)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/docker/api/container.py”, line 774, in inspect_container
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server self._get(self._url(“/containers/{0}/json”, container)), True
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/docker/utils/decorators.py”, line 46, in inner
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/docker/api/client.py”, line 237, in _get
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server return self.get(url, **self._set_request_timeout(kwargs))
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/requests/sessions.py”, line 555, in get
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server return self.request(GET’, url, **kwargs)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/requests/sessions.py”, line 542, in request
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server resp = self.send(prep, **send_kwargs)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/requests/sessions.py”, line 655, in send
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server r = adapter.send(request, **kwargs)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/requests/adapters.py”, line 529, in send
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server raise ReadTimeout(e, request=request)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host=localhost’, port=None): Read timed out. (read timeout=60)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/oslo_messaging/rpc/server.py”, line 165, in _process_incoming
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py”, line 309, in dispatch
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py”, line 229, in _do_dispatch
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/zun/common/utils.py”, line 232, in decorated_function
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server raise e
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server zun.common.exception.ZunException: Unexpected error: UnixHTTPConnectionPool(host=’localhost’, port=None): Read timed out. (read timeout=60 )
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server
Solution:
View the number of mq connections:
rabbitmqctl list_connections | wc -l rabbitmqctl list_connections
If there are not enough handles, modify the number of related handles:
- In /etc/security/limits.conf add:
mysql hard nofile 65535 mysql soft nofile 65535 * soft nofile 655350 *hard nofile 655350 * soft nproc 655350 *hard nproc 655350 * soft memlock unlimited *hard memlock unlimited
- Create the folder: mkdir -p /etc/systemd/system/mariadb.service.d/
Add the file vim /etc/systemd/system/mariadb.service.d/limits.conf with the following content:
[Service] LimitNOFILE=65535
- Modify vim /etc/my.cnf and add the max_connections item as follows:
[mysqld] max_connections=30000
-
Refresh configuration: systemctl daemon-reload;
-
Restart the service: systemctl restart mariadb
2.Nova computing node discovery
nova-manage cell_v2 discover_hosts –verbose
3.rabbitmq handle is full
- Edit the vim /usr/lib/systemd/system/rabbitmq-server.service file,
Add LimitNOFILE=300000 under [Service]; as follows:
[Service] LimitNOFILE=300000
-
Refresh configuration: systemctl daemon-reload;
-
Restart the service: systemctl restart rabbitmq-server;
Re-create the keystone authentication user on the control node:
source ~/keystonerc_admin
openstack user del zun
openstack user create –domain default –password zun zun
openstack role add –project services –user zun admin
openstack user del kuryr
openstack user create –domain default –password kuryr kuryr
openstack role add –project services –user kuryr admin
Restart the zun service of the control node:
systemctl restart zun-api zun-wsproxy
Restart the zun service of the computing node:
systemctl restart kuryr-libnetwork
systemctl restart zun-compute
Use the rabbitmqctl list_connections command in RabbitMQ to view the statistics of TCP/IP connections. The command format is:
rabbitmqctl list_connections [connectioninfoitem...]
Note that the connectioninfoitem parameter is used to indicate which information items will be included in the result set, and the column order of the result set will match the order of the parameters. The optional values for connectioninfoitem are as follows:
pid: Erlang process ID associated with the connection
name: The name of the connection.
port: server port
host: Returns the server host name, or IP address, obtained by reverse DNS, or is not enabled.
peer_port: Server peer port. When a client connects to the server, the client’s port is peer_port.
peer_host: Returns the peer host name or IP address obtained by reverse DNS, or is not enabled.
ssl: Whether to enable SSL
ssl_protocol: SSL protocol, such as tlsvl
ssl_key_exchange: SSL key exchange algorithm, such as: rsa
ssl_cipher: SSL encryption algorithm, such as: aes_256_cbc
ssl_hash: SSL hash algorithm, such as: sha
peer_cert_subject: The subject of the peer’s SSL security certificate, based on the form of RFC4514
peer_cert_issuer: The issuer of the peer SSL security certificate, based on the form of RFC4514
peer_cert_validity: Validity period of the peer SSL security certificate.
state: connection status, including starting, tuning, opening, running, flow, blocking, blocked, closing and closed.
channels: The number of channels in this connection.
protocol: The version of the AMQP protocol used, currently {0,9,1} or {0,8,0}. Note that if the client requests a connection for AMQP 0-9, RabbitMQ will also treat it as 0-9-1.
auth_mechanism: SASL authentication mechanism used, such as PLAIN, AMQPLAIN, EXTERNAL, RABBIT-CR-DEMO, etc.
user: The username associated with the connection.
vhost: The name of the vhost associated with the connection.
timeout: connection timeout/negotiated heartbeat interval, in seconds.
frame_max: The size of the maximum transmission frame, in B
channel_max: The maximum number of channels on this connection. If the value is 0, it means there is no upper limit, but the client will generally convert 0 to 65535
client_properties: Information properties sent by the client during connection establishment.
recv_oct: Number of bytes received.
recv_cnt: Number of data packets received.
send_oct: Number of bytes sent.
send_cnt: Number of data packets sent.
send_pend: send queue size.
connected_at: timestamp of connection establishment.
Example
(1) If connectioninfoitem information is not specified, user, peer_host, peer_port and state information will be displayed by default.
D:\server\rabbitmq_server-3.9.11\sbin> rabbitmqctl list_connections Listing connections... user peer_host peer_port state root 127.0.0.1 9365 running
(2) Manually specify connectioninfoitem information, for example:
D:\server\rabbitmq_server-3.9.11\sbin> rabbitmqctl list_connections user peer_host peer_port state Listing connections... user peer_host peer_port state root 127.0.0.1 9365 running