Explosive performance! Python multi-process mode implements multi-core CPU parallel computing

Article directory

  • Preface
  • 1. Multi-process mode in Python
  • 2. Methods to improve program execution efficiency
    • 1. Multiple processes execute tasks concurrently
    • 2. Process pool
  • 3.Message queue
    • 4. Shared memory
    • 5.Asynchronous IO
  • Summarize
      • About Python technical reserves
        • 1. Learning routes in all directions of Python
        • 2. Python basic learning video
        • 3. Excellent Python learning books
        • 4. Python toolkit + project source code collection
        • ①Python toolkit
        • ②Python practical case
        • ③Python mini game source code
        • 5. Interview materials
        • 6. Python part-time channels

Foreword

With the continuous development of computer hardware, multi-core CPUs have become popular hardware devices. Taking advantage of the advantages of multi-core CPUs can effectively improve program execution efficiency.

The multi-process mode can realize parallel computing of multi-core CPU. As a high-level programming language, Python provides multiple methods such as multi-process and multi-thread to implement parallel computing.

In this article, we will focus on how to use multi-process mode to improve program execution efficiency in Python.

1. Multi-process mode in Python

In Python, you can use the multiprocessing module to implement multiple processes. multiprocessing is a module in the Python standard library that manages the creation and communication of multiple processes.

In multiprocessing, you can use the Process class to create a process. The constructor of the Process class can accept a function as a parameter.

This function will be executed in the child process. Here’s a simple example:

import multiprocessing
def worker():
    print("Worker process started")
if __name__ == '__main__':
    p = multiprocessing.Process(target=worker)
    p.start()
    p.join()

In the above example, we first defined a worker function, then created a process using the Process class and passed the worker function as a parameter to the constructor of the Process class.

Finally, we call the start method of the Process class to start the process, and call the join method of the Process class to wait for the process to end.

2. Methods to improve program execution efficiency

Using multi-process mode in Python to improve program execution efficiency can be achieved in the following ways:

1. Multiple processes execute tasks concurrently

In multi-process mode, tasks can be assigned to multiple processes for parallel execution, thereby taking advantage of multi-core CPUs.

In Python, you can use the multiprocessing module to implement concurrent execution of tasks by multiple processes.

Here’s a simple example:

import multiprocessing
def worker(name):
    print("Worker %s started" % name)
if __name__ == '__main__':
    for i in range(5):
        p = multiprocessing.Process(target=worker, args=(i,))
        p.start()

In the above example, we define a worker function that accepts a parameter name and prints out the information about Worker name started in the function body.

Then we used a for loop to create 5 processes and passed the worker function and corresponding parameters to the constructor of the Process class.

Finally, we call the start method of the Process class to start the process.

2. Process pool

For a large number of repeated tasks, you can use a process pool to maintain a certain number of processes. Each process returns the result after executing a task, and then the process pool allocates the next task.

This can avoid frequent creation and destruction of processes and improve efficiency. In Python, you can use the Pool class of the multiprocessing module to implement a process pool.

Here’s a simple example:

import multiprocessing
def worker(name):
    print("Worker %s started" % name)
if __name__ == '__main__':
    with multiprocessing.Pool(processes=4) as pool:
        pool.map(worker, range(10))

In the above example, we define a worker function that accepts a parameter name and prints out the information about Worker name started in the function body.

Then we use the with statement to create a process pool and specify the number of processes in the process pool to be 4.

Finally, we use the map method of the Pool class to pass the worker function and corresponding parameters to the process pool, and the process pool will automatically assign tasks to different processes for execution.

3. Message queue

In multi-process mode, different processes need to communicate, and message queues can be used to achieve inter-process communication.

You can use the Queue module in Python to implement message queues. Here’s a simple example:

import multiprocessing
def producer(queue):
    for i in range(10):
        queue.put(i)
def consumer(queue):
    while not queue.empty():
        print(queue.get())
if __name__ == '__main__':
    queue = multiprocessing.Queue()
    p1 = multiprocessing.Process(target=producer, args=(queue,))
    p2 = multiprocessing.Process(target=consumer, args=(queue,))
    p1.start()
    p2.start()
    p1.join()
    p2.join()

In the above example, we defined a producer function and a consumer function. The producer function puts numbers from 0 to 9 into the message queue, and the consumer function takes the numbers from the message queue and prints them out.

Then we used the Queue class of the multiprocessing module to create a message queue, and used the Process class to create two processes to execute the producer function and consumer function respectively.

4. Shared memory

For data that needs to be shared by multiple processes, shared memory can be used to avoid the overhead of data copying and inter-process communication.

In Python, shared memory can be implemented using the Value and Array classes of the multiprocessing module.

Here’s a simple example:

import multiprocessing
def worker(counter):
    counter.value + = 1
if __name__ == '__main__':
    counter = multiprocessing.Value('i', 0)
    processes = []
    for i in range(5):
        p = multiprocessing.Process(target=worker, args=(counter,))
        processes.append(p)
        p.start()
    for p in processes:
        p.join()
    print(counter.value)

In the above example, we defined a worker function that accepts a parameter counter and increments the value of counter by 1 each time it is executed.

Then we used the Value class of the multiprocessing module to create an integer variable counter, and used the Process class to create 5 processes to execute the worker functions respectively.

Finally, we print out the value of counter.

5. Asynchronous IO

For I/O-intensive tasks, asynchronous IO can be used to improve efficiency. In Python, you can use the asyncio module to implement asynchronous IO.

Here’s a simple example:

import asyncio
async def worker():
    await asyncio.sleep(1)
    print("Worker process started")
loop = asyncio.get_event_loop()
loop.run_until_complete(worker())

In the above example, we defined a worker function that uses the asynchronous IO feature of the asyncio library.

In the function body, the asyncio.sleep function is used to simulate a long I/O operation and print a message after the operation is completed.

Then we created an event loop using the get_event_loop function of the asyncio library, and started the worker function using the run_until_complete function. During program execution, the event loop is responsible for scheduling and executing asynchronous IO operations.

Summary

In Python, multi-process mode can be used to achieve parallel computing on multi-core CPUs, thereby improving program execution efficiency.

In this article, we introduce how to use Python’s multiprocessing module to implement multi-process concurrent execution of tasks, process pools, message queues, shared memory, asynchronous IO and other methods to improve program execution efficiency.

In practical applications, it is necessary to choose an appropriate parallel computing method according to specific scenarios, and pay attention to avoiding common problems such as deadlocks.

About Python technical reserves

Learning Python well is good whether you are getting a job or doing a side job to make money, but you still need to have a learning plan to learn Python. Finally, we share a complete set of Python learning materials to give some help to those who want to learn Python!

CSDN gift package: “Python introductory information & amp; practical source code & amp; installation tools] free of charge (Safe link, click with confidence)

1. Learning routes in all directions of Python

The technical points in all directions of Python are organized to form a summary of knowledge points in various fields. Its usefulness is that you can find corresponding learning resources according to the above knowledge points to ensure that you learn more comprehensively.

2. Python basic learning video

② Route corresponding learning video

There are also many learning videos suitable for beginners. With these videos, you can easily get started with Python ~ Insert picture description here

③Practice questions

After each video lesson, there are corresponding exercises to test your learning results haha!

Due to limited space, only part of the information is shown

3. High-quality Python learning books

When I learn a certain basic and have my own understanding ability, I will read some books or handwritten notes compiled by my seniors. These notes record their understanding of some technical points in detail. These understandings are relatively unique and can be learned. to a different way of thinking.

4. Python toolkit + project source code collection
①Python toolkit

The commonly used development software for learning Python is here! Each one has a detailed installation tutorial to ensure you can install it successfully!

②Python practical case

Optical theory is useless. You must learn to type code along with it and practice it in order to apply what you have learned to practice. At this time, you can learn from some practical cases. 100+ practical case source codes are waiting for you!

③Python mini game source code

If you feel that the practical cases above are a bit boring, you can try writing your own mini-game in Python to add a little fun to your learning process!

5. Interview materials

We must learn Python to find a high-paying job. The following interview questions are the latest interview materials from first-tier Internet companies such as Alibaba, Tencent, Byte, etc., and Alibaba bosses have given authoritative answers. After finishing this set I believe everyone can find a satisfactory job based on the interview information.

6. Python part-time channels

Moreover, after learning Python, you can also take orders and make money on major part-time platforms. I have compiled various part-time channels + part-time precautions + how to communicate with customers into documents.


This complete version of the complete set of Python learning materials has been uploaded to CSDN. If you need it, friends can scan the CSDN official certification QR code below on WeChat to get it for free [Guaranteed 100% Free]