What is the producer consumer pattern
In the process of software development, we often encounter such scenarios:
Some modules are responsible for producing data, which are processed by other modules (the modules here may be: functions, threads, processes, etc.). The module that generates data is called a producer, and the module that processes data is called a consumer. The buffer zone between producers and consumers is called a warehouse. Producers are responsible for transporting goods to the warehouse, and consumers are responsible for taking out goods from the warehouse. This constitutes the producer-consumer model.
The structure diagram is as follows:
To make it easier for everyone to understand, let’s give an example of sending a letter. Suppose you want to send a letter. The general process is as follows:
-
You write the letter well – it is equivalent to the producer producing data
-
You put the letter into the mailbox – it is equivalent to the producer putting the data into the buffer
-
The postman takes the letter out of the mailbox and processes it accordingly – equivalent to the consumer taking the data out of the buffer and processing the data
Advantages of the producer-consumer model
- Decoupling
Assume that the producer and consumer are two threads respectively. If the producer directly calls a method of the consumer, the producer will be dependent on the consumer (that is, coupled). If the consumer’s code changes in the future, it may affect the producer’s code. And if both depend on a certain buffer, there is no direct dependence between the two, and the coupling will be reduced accordingly.
For example, if we go to the post office to deliver a letter, if we do not use a mailbox (that is, a buffer zone), you must hand the letter directly to the postman. Some students may say, isn’t it simple to just give it to the postman? It’s not simple. You have to know who the postman is before you can give him the letter. This creates a dependency between you and Postman (equivalent to strong coupling between producers and consumers). In case the postman changes one day, you have to re-understand it (which is equivalent to modifying the producer code due to changes in the consumer). The mailbox is relatively fixed, and the cost of relying on it is relatively low (equivalent to weak coupling with the buffer).
- Concurrency
Since the producer and the consumer are two independent concurrent bodies, they communicate using buffers. The producer only needs to throw data into the buffer and can continue to produce the next data, while the consumer only needs to retrieve data from the buffer. Just get the data in different zones, so that they won’t be blocked due to each other’s processing speed.
Continuing the above example, if we don’t use the mailbox, we have to wait for the postman at the post office until he comes back and hand the letter to him. During this period, we can’t do anything (that is, producer blocking). Or the postman has to go door to door and ask who wants to send a letter (equivalent to consumer polling).
- Supports uneven busyness
When the producer produces data quickly, the consumer has no time to process it. The unprocessed data can be temporarily stored in the buffer and processed slowly. This will not cause data loss or affect producer production due to consumer performance.
Let’s take the example of sending letters again. Assume that the postman can only take away 1,000 letters at a time. If it happens to be Valentine’s Day (or Christmas) to send greeting cards, more than 1,000 letters need to be sent. At this time, the buffer zone of the mailbox It comes in handy. The postman temporarily stores the letters that are too late to take away in the mailbox until he comes next time to take them away.
Through the above introduction, you should already understand the producer-consumer model.
Multithreaded programming in Python
Before implementing the producer-consumer model, we first learn multi-threaded programming in Python.
Threads are execution units directly supported by the operating system. High-level languages usually have built-in multi-threading support, and Python is no exception. And Python’s threads are real Posix Threads, not simulated threads.
Python’s standard library provides two modules: _thread and threading. _thread is a low-level module, and threading is a high-level module that encapsulates _thread. In most cases, we only need to use the advanced module threading.
Let’s first look at a piece of code that implements multi-threading in Python.
import time,threading
#Thread code
class TaskThread(threading.Thread):
def __init__(self,name):
threading.Thread.__init__(self,name=name)
def run(self):
print('thread %s is running...' % self.getName())
for i in range(6):
print('thread %s >>> %s' % (self.getName(), i))
time.sleep(1)
print('thread %s finished.' % self.getName())
taskthread = TaskThread('TaskThread')
taskthread.start()
taskthread.join()
The following is the execution result of the program:
thread TaskThread is running... thread TaskThread >>> 0 thread TaskThread >>> 1 thread TaskThread >>> 2 thread TaskThread >>> 3 thread TaskThread >>> 4 thread TaskThread >>> 5 thread TaskThread finished.
The TaskThread class inherits from the Thread thread class in the threading module. The name parameter of the constructor specifies the name of the thread, and specific tasks are implemented by overloading the base class run function.
After getting briefly familiar with Python’s threads, let’s implement a producer-consumer model.
from Queue import Queue
import random,threading,time
#Producer class
class Producer(threading.Thread):
def __init__(self, name,queue):
threading.Thread.__init__(self, name=name)
self.data=queue
def run(self):
for i in range(5):
print("%s is producing %d to the queue!" % (self.getName(), i))
self.data.put(i)
time.sleep(random.randrange(10)/5)
print("%s finished!" % self.getName())
#consumercategory
class Consumer(threading.Thread):
def __init__(self,name,queue):
threading.Thread.__init__(self,name=name)
self.data=queue
def run(self):
for i in range(5):
val = self.data.get()
print("%s is consuming. %d in the queue is consumed!" % (self.getName(),val))
time.sleep(random.randrange(10))
print("%s finished!" % self.getName())
def main():
queue = Queue()
producer = Producer('Producer',queue)
consumer = Consumer('Consumer',queue)
producer.start()
consumer.start()
producer.join()
consumer.join()
print 'All threads finished!'
if __name__ == '__main__':
main()
The execution results may be as follows:
Producer is producing 0 to the queue! Consumer is consuming. 0 in the queue is consumed! Producer is producing 1 to the queue! Producer is producing 2 to the queue! Consumer is consuming. 1 in the queue is consumed! Consumer is consuming. 2 in the queue is consumed! Producer is producing 3 to the queue! Producer is producing 4 to the queue! Producer finished! Consumer is consuming. 3 in the queue is consumed! Consumer is consuming. 4 in the queue is consumed! Consumer finished! All threads finished!
Because multi-threading is executed preemptively, the printed running results may not be exactly the same as the above.
Summary
This example implements a simple producer-consumer model through Python. The Queue module in Python already provides support for thread synchronization, so this article does not cover multi-threading issues such as locks, synchronization, and deadlocks.
Did you gain something from reading this article? Please forward and share it with more people
Finally:
Python learning materials
If you want to learn Python to help you automate your office, or are preparing to learn Python or are currently learning it, you should be able to use the following and get it if you need it.
① Python learning roadmap for all directions, knowing what to learn in each direction ② More than 100 Python course videos, covering essential basics, crawlers and data analysis ③ More than 100 Python practical cases, learning is no longer just theory ④ Huawei’s exclusive Python comic tutorial, you can also learn it on your mobile phone ⑤Real Python interview questions from Internet companies over the years, very convenient for review
There are ways to get it at the end of the article
1. Learning routes in all directions of Python
The Python all-direction route is to organize the commonly used technical points of Python to form a summary of knowledge points in various fields. Its usefulness is that you can find corresponding learning resources according to the above knowledge points to ensure that you learn more comprehensively.
2. Python course video
When we watch videos to learn, we cannot just move our eyes and brains but not our hands. The more scientific learning method is to use them after understanding. At this time, hands-on projects are very suitable.
3. Python practical cases
Optical theory is useless. You must learn to follow along and practice it in order to apply what you have learned into practice. At this time, you can learn from some practical cases.
Four Python Comics Tutorial
Use easy-to-understand comics to teach you to learn Python, making it easier for you to remember and not boring.
5. Internet company interview questions
We must learn Python to find a high-paying job. The following interview questions are the latest interview materials from first-tier Internet companies such as Alibaba, Tencent, Byte, etc., and Alibaba bosses have given authoritative answers. After finishing this set I believe everyone can find a satisfactory job based on the interview information.
This complete version of the complete set of Python learning materials has been uploaded to CSDN. If friends need it, you can also scan the official QR code of csdn below or click on the WeChat card at the bottom of the homepage and article to get the method. [Guaranteed 100% free]
