1. GIL
Anyone familiar with python knows that there is a global interpreter lock in the python interpreter written in C language. Due to the existence of the global interpreter lock, the python interpreter can only run the code of one thread at the same time, which greatly affects python. Multi-threaded performance. Due to historical reasons, this interpreter lock is almost impossible to eliminate now.
The reason why python GIL affects the performance of multi-threading is because in the case of multi-threading, the code of the thread can only run when the thread obtains a global lock, and there is only one global lock, so use python multi-threading , only one thread is running at the same time, so even in the case of multi-core, it can only exert the performance of a single core.
2. Multi-threaded processing of IO-intensive tasks
IO-intensive tasks refer to the fact that the system’s CPU performance is much better than that of the hard disk and memory. At this time, when the system is running, most of the conditions are when the CPU is waiting for I/O (hard disk/memory) read/write operations. At this time, CPU Loading Not high. Tasks involving network and disk IO are IO-intensive tasks. When a thread performs IO-intensive tasks, the CPU is idle, so the GIL will be released to other threads, thereby shortening the overall waiting time.
from concurrent.futures import ThreadPoolExecutor from time import sleep, time # Number of Workers N = 4 # Create thread pool pool = ThreadPoolExecutor(max_workers=N)
2.1 Define an IO-intensive function
The function “sleeps” for x seconds.
def io_bound_func(x):
sleep(x)
print("Sleep for %d seconds." % x)
2.2 Use serial processing
Traverse all elements of a list and execute the func function.
def process_array(arr):
for x in arr:
io_bound_func(x)
2.3 Use multi-threading
Through the map method of the thread pool, the same function can be applied to all elements in the list.
def fast_process_array(arr):
for x in pool.map(io_bound_func, arr):
pass
2.4 Calculate function running time
-
Run time for serial version = 1 + 2 + 3 = 6 seconds
-
Running time of multi-threaded version = max(1, 2, 3) = 3 seconds
def time_it(fn, *args):
start = time()
fn(*args)
print("The running time of the %s version is %.5f seconds!" % (fn.__name__, time() - start))
time_it(process_array, [1, 2, 3])
Sleep for 1 seconds. Sleep for 2 seconds. Sleep for 3 seconds. The process_array version ran in 6.00883 seconds!
time_it(fast_process_array, [1, 2, 3])
Sleep for 1 seconds. Sleep for 2 seconds. Sleep for 3 seconds. The fast_process_array version runs in 3.00300 seconds!
3. Multi-threaded CPU-intensive tasks
CPU-intensive tasks are characterized by requiring a large amount of calculations and consuming CPU resources, such as calculating pi, high-definition decoding of videos, etc., all relying on the computing power of the CPU. When a thread performs a CPU-intensive task, the CPU is busy. After running 1000 bytecodes, the GIL will be released to other threads. In addition, the time to switch threads may be slower than serial code.
3.1 Define a CPU-intensive function
This function sums integers between [1, x].
def cpu_bound_func(x):
tot = 0
a = 1
while a <= x:
tot + = x
a + = 1
print("Finish sum from 1 to %d!" % x)
return tot
3.2 Use serial processing
Traverse all elements of a list and execute the func function.
def process_array(arr):
for x in arr:
cpu_bound_func(x)
3.3 Using multi-threading
Through the map method of the thread pool, the same function can be applied to all elements in the list.
def fast_process_array(arr):
for x in pool.map(cpu_bound_func, arr):
pass
3.4 Calculate function running time
-
Serial version runs in 2.1 seconds
-
The multi-threaded version runs in 2.2 seconds
def time_it(fn, *args):
start = time()
fn(*args)
print("The running time of the %s version is %.5f seconds!" % (fn.__name__, time() - start))
time_it(process_array, [10**7, 10**7, 10**7])
Finish sum from 1 to 10000000! Finish sum from 1 to 10000000! Finish sum from 1 to 10000000! The process_array version runs in 2.10489 seconds!
time_it(fast_process_array, [10**7, 10**7, 10**7])
Finish sum from 1 to 10000000! Finish sum from 1 to 10000000! Finish sum from 1 to 10000000! The fast_process_array version runs in 2.20897 seconds!
About Python’s technical reserves
If you are planning to learn Python or are currently learning it, you should be able to use the following:
① A roadmap for learning Python in all directions, knowing what to learn in each direction
② More than 100 Python course videos, covering essential basics, crawlers and data analysis
③ More than 100 Python practical cases, learning is no longer just theory
④ Huawei’s exclusive Python comic tutorial, you can also learn it on your mobile phone
⑤Real Python interview questions from Internet companies over the years, very convenient for review
There are ways to get it at the end of the article
1. Learning routes in all directions of Python
The Python all-direction route is to organize the commonly used technical points of Python to form a summary of knowledge points in various fields. Its usefulness is that you can find corresponding learning resources according to the above knowledge points to ensure that you learn more comprehensively.
2. Python course video
When we watch videos and learn, we can’t just move our eyes and brain but not our hands. The more scientific learning method is to use them after understanding. At this time, hands-on projects are very suitable.
3. Python practical cases
Optical theory is useless. You must learn to follow along and practice it in order to apply what you have learned into practice. At this time, you can learn from some practical cases.
4. Python comic tutorial
Use easy-to-understand comics to teach you to learn Python, making it easier for you to remember and not boring.
5. Internet company interview questions
We must learn Python to find a high-paying job. The following interview questions are the latest interview materials from first-tier Internet companies such as Alibaba, Tencent, Byte, etc., and Alibaba bosses have given authoritative answers. After finishing this set I believe everyone can find a satisfactory job based on the interview information.
This complete version of the complete set of Python learning materials has been uploaded to CSDN. If friends need it, you can also scan the official QR code of csdn below or click on the WeChat card at the bottom of the homepage and article to get the method. [Guaranteed 100% free]
