Connection Pool (Connection Pool) stores network connections as objects in memory during system initialization. When the user needs to connect, an established empty connection object is taken from the connection pool and the connection is not closed after use. , put the connection back into the connection pool without having to frequently create and close connections, which can reduce connection overhead and improve system response speed.
How connection pooling works:
- Creation and initialization of connections: In the initial stage, parameters such as the minimum number of connections, the maximum number of connections, and the connection time are set, and then a certain number of database connections are created. These connections are often referred to as the “idle connections” of the connection pool.
- Connection acquisition: When the application needs to interact with the database, it can look for an idle connection in the connection pool. If there is, use it. If not, create a connection or wait for other clients to release the connection.
- Usage of connections: Applications can use database connections to perform query, insert, update, or delete operations. Once the operation is complete, the connection can remain open for reuse at a later time.
- Return of connection: When an application no longer needs a connection, it returns the connection to the connection pool so that other applications can use it again.
- Management of connections: Connection pools typically track the status and availability of connections. If a connection becomes invalid (for example, due to a disconnection or an error), the connection pool automatically replaces it.
The advantages of connection pooling include:
- Reducing connection overhead: The connection pool avoids frequent creation and closing of connections, thereby reducing connection overhead and improving the performance of database operations.
- Improving response speed: Since some connections have been pre-created in the connection pool, when there is a database operation request, an idle connection can be obtained directly from the connection pool, avoiding the time of waiting for a connection to be created, thus Improved response speed.
- Resource management: The connection pool can dynamically adjust the number and status of connections through configuration parameters, so as to better adapt to different business scenarios and load conditions and avoid excessive database connections from burdening the database server. .
- Connection reuse: Connection pooling allows multiple applications to share connections to achieve connection reuse, which is very useful for high-load systems.
Connection pooling in Python library
redis third-party library redis connection pool
# python3.7.9 redis 5.0.1
# redis/connection.py
class ConnectionPool(object):
def __init__(self, connection_class=Connection, max_connections=None,
**connection_kwargs):
#The maximum number of connections in the connection pool, 2**31 can be created by default
max_connections = max_connections or 2 ** 31
if not isinstance(max_connections, (int, long)) or max_connections < 0:
raise ValueError('"max_connections" must be a positive integer')
self.connection_class = connection_class
self.connection_kwargs = connection_kwargs
self.max_connections = max_connections
# Protect_checkpid() critical section
self._fork_lock = threading.Lock()
self.reset()
def reset(self):
self._lock = threading.Lock()
# Connection pool size
self._created_connections = 0
# Available connections
self._available_connections = []
# The connection in use
self._in_use_connections = set()
# Process ID of the connection pool
self.pid = os.getpid()
#Create a connection and add it to the connection pool
def make_connection(self):
"Create a new connection"
if self._created_connections >= self.max_connections:
raise ConnectionError("Too many connections")
self._created_connections + = 1
return self.connection_class(**self.connection_kwargs)
\t
# Get a connection from the connection pool
def get_connection(self, command_name, *keys, **options):
"Get a connection from the pool"
self._checkpid()
with self._lock:
try:
# Pop a connection from the available connection pool
connection = self._available_connections.pop()
except IndexError:
# If no connection is available, create one
connection = self.make_connection()
# Then add it to the used list
self._in_use_connections.add(connection)
try:
# ensure this connection is connected to Redis
connection.connect()
try:
if connection.can_read():
raise ConnectionError('Connection has data')
except ConnectionError:
connection.disconnect()
connection.connect()
if connection.can_read():
raise ConnectionError('Connection not ready')
except BaseException:
# release the connection back to the pool so that we don't
# leak it
self.release(connection)
raise
return connection
# Release a connection
def release(self, connection):
"Releases the connection back to the pool"
self._checkpid()
with self._lock:
try:
# Remove from used list
self._in_use_connections.remove(connection)
exceptKeyError:
# Gracefully fail when a connection is returned to this pool
# that the pool doesn't actually own
pass
if self.owns_connection(connection):
# If it is a connection in the connection pool, add it to the available connection pool.
self._available_connections.append(connection)
else:
# If it is a normal connection, disconnect
self._created_connections -= 1
connection.disconnect()
return
Connection pooling in the built-in module http
# urllib3/connectionpool.py
class HTTPConnectionPool(ConnectionPool, RequestMethods):
def __init__(self):
#Initialize the connection pool
self.pool = self.QueueCls(maxsize)
def _new_conn(self):
"""Create a new connection"""
conn = self.ConnectionCls(
host=self.host,
port=self.port,
timeout=self.timeout.connect_timeout,
strict=self.strict,
**self.conn_kw
)
return conn
def _get_conn(self, timeout=None):
"""Get a connection"""
conn=None
try:
conn = self.pool.get(block=self.block, timeout=timeout)
except AttributeError: # self.pool is None
raise ClosedPoolError(self, "Pool is closed.")
"""
.........
"""
# Get a connection, create a new connection if it does not exist
return conn or self._new_conn()
def _put_conn(self, conn):
"""After the connection is used, put it back into the connection pool"""
try:
self.pool.put(conn, block=False)
return # Everything is dandy, done.
exceptAttributeError:
# self.pool is None.
pass
def close(self):
"""Close all connections"""
if self.pool is None:
return
# Disable access to the pool
old_pool, self.pool = self.pool, None
try:
while True:
conn = old_pool.get(block=False)
if conn:
conn.close()
except queue.Empty:
pass # Done.