A multi-threaded approach to faster OpenCV video streaming in Python

Overview

In this article, we will see two Python code examples without multithreading for reading video frames from a camera. We’ll see the difference in FPS obtained with/without multithreading.

What is multithreading?

A thread is an execution unit in a process. Multithreading refers to the concurrent execution of multiple threads by quickly switching control of the CPU between threads (called context switching). In our example, we will see that multi-threading enables faster real-time video processing by increasing FPS (frames per second).

Thread basics in Python

The following code snippet shows how to create a thread using the threading module in python:

# importing the threading module
import threading
  
# importing the time module
import time
  
# Function to print "Hello", however, the function sleeps
# for 2 seconds at the 11th iteration
def print_hello():
  for i in range(20):
    if i == 10:
      time.sleep(2)
    print("Hello")
  
# Function to print numbers till a given number
def print_numbers(num):
  for i in range(num + 1):
    print(i)
  
# Creating the threads. Target is set to the name of the
# function that needs to be executed inside the thread and
# args are the arguments to be supplied to the function that
  
# needs to be executed.
print("Greetings from the main thread.")
thread1 = threading.Thread(target = print_hello, args = ())
thread2 = threading.Thread(target = print_numbers, args = (10,))
  
# Starting the two threads
thread1.start()
thread2.start()
print("It's the main thread again!")

Let’s try to understand the output by tracing the execution of the code:

Main thread execution. Print “Greetings from the main thread”, create thread1 and thread2 and start the threads.
A context switch occurs and execution of thread1 begins.
After the first ten iterations, thread1 goes to sleep and thread2 begins executing, completing before the next context switch.
Now, the main thread takes control of the CPU and prints “It’s the main thread again!”
Another context switch occurs and thread2 resumes execution and completes.
Since the main thread has no more instructions to execute, the program terminates.

Use thread.join()

What if I need to block the main thread until thread1 and thread2 finish executing?

thread.join() comes in handy because it blocks the calling thread until the thread calling its join() method terminates:

# importing the threading module
import threading
  
# importing the time module
import time
# Function to print "Hello", however, the function sleeps
  
# for 2 seconds at the 11th iteration
def print_hello():
  for i in range(20):
    if i == 10:
      time.sleep(2)
    print("Hello")
  
# Function to print numbers till a given number
def print_numbers(num):
  for i in range(num + 1):
    print(i)
  
# Creating the threads. Target is set to the name of the
# function that needs to be executed inside the thread and
# args are the arguments to be supplied to the function that
# needs to be executed.
print("Greetings from the main thread.")
thread1 = threading.Thread(target = print_hello, args = ())
thread2 = threading.Thread(target = print_numbers, args = (10,))
  
# Starting the two threads
thread1.start()
thread2.start()
thread1.join()
thread2.join()
print("It's the main thread again!")
print("Threads 1 and 2 have finished executing.")

Reasons Multi-threading helps faster processing

The video processing code is divided into two parts: reading the next available frame from the camera and performing video processing on the frame, such as running a deep learning model for face recognition, etc.

Read the next frame and process it sequentially in a program without multithreading. The program waits for the next frame to be available before doing the necessary processing on it. The time required to read a frame is primarily related to the time required to request, wait, and transfer the next video frame from the camera to memory. The time spent performing computations on video frames, whether on the CPU or GPU, accounts for the majority of the time spent in video processing.

In a program with multiple threads, reading the next frame and processing it does not need to be sequential. While one thread performs the task of reading the next frame, the main thread can use the CPU or GPU to process the last read frame. This way, by overlapping the two tasks, the total time to read and process frames can be reduced.

OpenCV code – no multithreading

# importing required libraries
import cv2
import time
  
# opening video capture stream
vcap = cv2.VideoCapture(0)
if vcap.isOpened() is False :
    print("[Exiting]: Error accessing webcam stream.")
    exit(0)
fps_input_stream = int(vcap.get(5))
print("FPS of webcam hardware/input stream: {}".format(fps_input_stream))
grabbed, frame = vcap.read() # reading single frame for initialization/ hardware warm-up
  
# processing frames in input stream
num_frames_processed = 0
start = time.time()
while True :
    grabbed, frame = vcap.read()
    if grabbed is False :
        print('[Exiting] No more frames to read')
        break
  
# adding a delay for simulating time taken for processing a frame
    delay = 0.03 # delay value in seconds. so, delay=1 is equivalent to 1 second
    time.sleep(delay)
    num_frames_processed + = 1
cv2.imshow('frame' , frame)
    key = cv2.waitKey(1)
    if key == ord('q'):
        break
end = time.time()
  
# printing time elapsed and fps
elapsed=end-start
fps = num_frames_processed/elapsed
print("FPS: {} , Elapsed Time: {} , Frames Processed: {}".format(fps, elapsed, num_frames_processed))
  
# releasing input stream, closing all windows
vcap.release()
cv2.destroyAllWindows()

OpenCV code – multi-threading

# importing required libraries
import cv2
import time
from threading import Thread # library for implementing multi-threaded processing
  
# defining a helper class for implementing multi-threaded processing
classWebcamStream:
    def __init__(self, stream_id=0):
        self.stream_id = stream_id # default is 0 for primary camera
          
        # opening video capture stream
        self.vcap = cv2.VideoCapture(self.stream_id)
        if self.vcap.isOpened() is False :
            print("[Exiting]: Error accessing webcam stream.")
            exit(0)
        fps_input_stream = int(self.vcap.get(5))
        print("FPS of webcam hardware/input stream: {}".format(fps_input_stream))
              
        # reading a single frame from vcap stream for initializing
        self.grabbed, self.frame = self.vcap.read()
        if self.grabbed is False :
            print('[Exiting] No more frames to read')
            exit(0)
  
# self.stopped is set to False when frames are being read from self.vcap stream
        self.stopped = True
  
# reference to the thread for reading next available frame from input stream
        self.t = Thread(target=self.update, args=())
        self.t.daemon = True # daemon threads keep running in the background while the program is executing
          
    # method for starting the thread for grabbing next available frame in input stream
    def start(self):
        self.stopped = False
        self.t.start()
  
# method for reading next frame
    def update(self):
        while True :
            if self.stopped is True :
                break
            self.grabbed, self.frame = self.vcap.read()
            if self.grabbed is False :
                print('[Exiting] No more frames to read')
                self.stopped = True
                break
        self.vcap.release()
  
# method for returning latest read frame
    def read(self):
        return self.frame
  
# method called to stop reading frames
    def stop(self):
        self.stopped = True
  
# initializing and starting multi-threaded webcam capture input stream
webcam_stream = WebcamStream(stream_id=0) # stream_id = 0 is for primary camera
webcam_stream.start()
  
# processing frames in input stream
num_frames_processed = 0
start = time.time()
while True :
    if webcam_stream.stopped is True :
        break
    else :
        frame = webcam_stream.read()
  
# adding a delay for simulating time taken for processing a frame
    delay = 0.03 # delay value in seconds. so, delay=1 is equivalent to 1 second
    time.sleep(delay)
    num_frames_processed + = 1
cv2.imshow('frame' , frame)
    key = cv2.waitKey(1)
    if key == ord('q'):
        break
end = time.time()
webcam_stream.stop() # stop the webcam stream
  
# printing time elapsed and fps
elapsed=end-start
fps = num_frames_processed/elapsed
print("FPS: {} , Elapsed Time: {} , Frames Processed: {}".format(fps, elapsed, num_frames_processed))
  
# closing all windows
cv2.destroyAllWindows()

[I compiled a lot of Python learning materials when I taught myself Python before, but I can’t use them now. I have uploaded it to the CSDN official website. Friends in need can scan the QR code below to obtain it]

A multi-threaded approach to faster OpenCV video streaming in Python

Overview

What is multithreading?

Thread basics in Python

Use thread.join()

Reasons Multi-threading helps faster processing

OpenCV code – no multithreading

OpenCV code – multi-threading

1. Study Outline

2. Development tools

3. Python basic materials

4. Practical information