Python implements dual-targeting, distortion correction, and stereoscopic correction

1. The role of dual-targeting, distortion correction, and stereoscopic correction

  1. Binocular Targeting:

    • 3D reconstruction and ranging: Through binocular targeting, you can determine the relative position and orientation between the two cameras, so that you can calculate the depth of the object based on the parallax information and perform 3D reconstruction and ranging.
    • Pose Estimation: Binocular cameras can be used to estimate the pose of an object or camera, which is very important for applications such as virtual reality, augmented reality, and robot navigation.

  2. Distortion correction:

    • Image quality improvement: Lens distortion will introduce image distortion, and distortion correction can improve image quality and make the shape of objects more accurate.
    • Feature matching: In image feature matching, distortion correction can improve the accuracy of matching and help correctly match feature points between different views.

  3. Stereocorrection:

    • Stereo Vision: Stereo correction is a key step for binocular or multi-eye vision systems, which can make the pixel corresponding points of two cameras on the same horizontal line, thus simplifying the disparity calculation in stereo vision.
    • Depth Perception: After stereo correction, you can use the disparity map to estimate the depth of the object, which is very important for achieving depth perception and 3D reconstruction.

Second, binocular camera code

#coding:utf-8
import cv2
import time
import time

left_camera = cv2.VideoCapture(0)
left_camera.set(cv2.CAP_PROP_FRAME_WIDTH,640)
left_camera.set(cv2.CAP_PROP_FRAME_HEIGHT,480)

right_camera = cv2.VideoCapture(1)
right_camera.set(cv2.CAP_PROP_FRAME_WIDTH,640)
right_camera.set(cv2.CAP_PROP_FRAME_HEIGHT,480)

path="/home/song/pic/" #Picture storage path

AUTO =False # True to automatically take a photo, False to manually press the s key to take a photo
INTERVAL = 0.0000005 # Adjust the automatic photo interval

cv2.namedWindow("left")
cv2.namedWindow("right")
cv2.moveWindow("left", 0, 0)

counter = 0
utc = time.time()
folder = "/home/song/pic/" # Photo storage path

def shot(pos, frame):
    global counter
    timestr = datetime.datetime.now()
    path = folder + pos + "_" + str(counter) + ".jpg"
    cv2.imwrite(path, frame)
    print("snapshot saved into: " + path)

while True:
    ret, left_frame = left_camera.read()
    ret, right_frame = right_camera.read()

    cv2.imshow("left", left_frame)
    cv2.imshow("right", right_frame)

    now = time.time()
    if AUTO and now - utc >= INTERVAL:
        shot("left", left_frame)
        shot("right", right_frame)
        counter + = 1
        utc=now

    key = cv2.waitKey(1)
    if key == ord("q"):
        break
    elif key == ord("s"):
        shot("left", left_frame)
        shot("right", right_frame)
        counter + = 1
        
left_camera.release()
right_camera.release()
cv2.destroyWindow("left")
cv2.destroyWindow("right")

Third, calibrate the left and right targets respectively

Before calibrating the binocular camera, you need to perform single-object calibration on the left and right monocular cameras. This is because the calibration of binocular cameras requires knowing the internal parameters of each camera (such as camera matrix, distortion coefficient) and the external parameters between cameras (relative position and orientation). These parameters are obtained through single-object specification.

Single-target targeting usually involves the following steps:

  1. Camera intrinsic parameter calibration: By taking an image containing a calibration plate of known size, and then using the camera calibration algorithm to estimate the camera intrinsic parameters, such as focal length, principal point coordinates and distortion coefficient.

  2. Camera extrinsic parameter calibration: By placing the camera in different positions or taking images in different directions, use the extrinsic parameter calibration algorithm to estimate the position and orientation of the camera. This step usually requires multiple images to determine the camera’s position and attitude in three-dimensional space.

  3. Distortion correction: Use the distortion coefficient obtained by internal parameter calibration to correct the image to remove lens distortion.

  4. Generate calibration file: Save the internal and external parameters and distortion coefficients in the calibration file for subsequent dual-object calibration.

Once the internal parameters, external parameters and distortion coefficients of the monocular camera are known, binocular calibration can be performed to determine the relative position and orientation between the binocular cameras, as well as the calibration parameters in stereo vision.

#-*- coding:utf-8 -*-
import numpy as np
import cv2
import glob

#Set iteration termination condition
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)

# Set object points in the form of (0,0,0), (1,0,0), (2,0,0) ...., (6,5,0)
objp = np.zeros((6*7,3), np.float32) #I use a 6×7 checkerboard. You can modify the relevant parameters according to your own checkerboard.
objp[:,:2] = np.mgrid[0:7,0:6].T.reshape(-1,2)

# Use arrays to store object points and image points of all pictures
objpoints = [] # 3d point in real world space
imgpoints = [] # 2d points in image plane.

#Use glob to match all pictures with file names containing ".jpg" in the folder /home/song/pic_1/right/
images = glob.glob(r"/home/song/pic/right/*.jpg")

for fname in images:
    img = cv2.imread(fname)
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    # Find checkerboard corner points
    ret, corners = cv2.findChessboardCorners(gray, (7,6), None)
    # If found, add object points, image points
    if ret == True:
        objpoints.append(objp)
        corners2=cv2.cornerSubPix(gray,corners, (11,11), (-1,-1), criteria)
        imgpoints.append(corners)
        # Draw lines connecting diagonal points to display them
        cv2.drawChessboardCorners(img, (7,6), corners2, ret)
        cv2.imshow('img', img)
        cv2.waitKey(500)
cv2.destroyAllWindows()

# Calibration
ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, gray.shape[::-1], None, None)
print(mtx, dist)

#De-distort all images. There are two methods to achieve this: undistort() and remap()
images = glob.glob(r"/home/song/pic/right/*.jpg")
for fname in images:
    prefix=fname.split('/')[5]
    img = cv2.imread(fname)
    h, w = img.shape[:2]
    newcameramtx, roi=cv2.getOptimalNewCameraMatrix(mtx, dist, (w,h), 1, (w,h))

    # # Use cv.undistort() for distortion correction
    # dst = cv2.undistort(img, mtx, dist, None, newcameramtx)
    # # Crop the effective area of the image
    # # x, y, w, h = roi
    # # dst = dst[y:y + h, x:x + w]
    # cv2.imwrite('/home/song/pic_1/undistort/' + prefix, dst)

    # Use the remap() function for correction
    mapx, mapy = cv2.initUndistortRectifyMap(mtx, dist, None, newcameramtx, (w, h), 5)
    dst = cv2.remap(img, mapx, mapy, cv2.INTER_LINEAR)
    #Crop the effective area of the image
    x, y, w, h = roi
    dst = dst[y:y + h, x:x + w]
    cv2.imwrite('/home/song/pic/undistort/' + prefix, dst)

#Reprojection error calculation
mean_error = 0
for i in range(len(objpoints)):
    imgpoints2, _ = cv2.projectPoints(objpoints[i], rvecs[i], tvecs[i], mtx, dist)
    error = cv2.norm(imgpoints[i],imgpoints2, cv2.NORM_L2)/len(imgpoints2)
    mean_error + = error

print("total error: ", mean_error/len(objpoints))

4. Binary target determination and stereo correction

#coding:utf-8
import numpy as np
import cv2
import matplotlib.pyplot as plt
from PIL import Image

#Set iteration termination condition
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
criteria_stereo = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)

# Set object points in the form of (0,0,0), (1,0,0), (2,0,0) ...., (6,5,0)
objp = np.zeros((6 * 7, 3), np.float32) #I use a 6×7 checkerboard. You can modify the relevant parameters according to your own checkerboard.
objp[:, :2] = np.mgrid[0:7, 0:6].T.reshape(-1, 2)

# Use arrays to store object points and image points of all pictures
objpoints = [] # 3d points in real world space
imgpointsR = [] # 2d points in image plane
imgpointsL = []

# In this experimental collection, a total of 30 groups of images to be calibrated are read in sequentially and the following operations are performed
for i in range(0,30):
    t = str(i)
    ChessImaR = cv2.imread('/home/song/pic/right_' + t + '.jpg', 0) # Right view
    ChessImaL = cv2.imread('/home/song/pic/left_' + t + '.jpg', 0) # Left view
    retR, cornersR = cv2.findChessboardCorners(ChessImaR,(7, 6), None) #Extract the corner points of each picture on the right
    retL, cornersL = cv2.findChessboardCorners(ChessImaL,(7, 6), None) # # Extract the corner points of each picture on the left
    if (True == retR) & amp; (True == retL):
        objpoints.append(objp)
        cv2.cornerSubPix(ChessImaR, cornersR, (11, 11), (-1, -1), criteria) # Sub-pixel precision, precision of rough extracted corner points
        cv2.cornerSubPix(ChessImaL, cornersL, (11, 11), (-1, -1), criteria) # Sub-pixel precision, precision of rough extracted corner points
        imgpointsR.append(cornersR)
        imgpointsL.append(cornersL)

# Camera single and double target setting and correction
# Calibrate the right camera separately
retR, mtxR, distR, rvecsR, tvecsR = cv2.calibrateCamera(objpoints,imgpointsR,ChessImaR.shape[::-1], None, None)

# Obtain the new camera matrix and then pass it to initUndistortRectifyMap to use remap to generate the mapping relationship
hR, wR = ChessImaR.shape[:2]
OmtxR, roiR = cv2.getOptimalNewCameraMatrix(mtxR, distR,(wR, hR), 1, (wR, hR))

# Calibrate the left camera separately
retL, mtxL, distL, rvecsL, tvecsL = cv2.calibrateCamera(objpoints,imgpointsL,ChessImaL.shape[::-1], None, None)

# Obtain the new camera matrix and then pass it to initUndistortRectifyMap to use remap to generate the mapping relationship
hL, wL = ChessImaL.shape[:2]
OmtxL, roiL = cv2.getOptimalNewCameraMatrix(mtxL, distL, (wL, hL), 1, (wL, hL))

# Calibration of binocular camera
# Set the flag bit to cv2.CALIB_FIX_INTRINSIC, which will fix the input cameraMatrix and distCoeffs unchanged, and only solve ,,,
flags = 0
flags |= cv2.CALIB_FIX_INTRINSIC

retS, MLS, dLS, MRS, dRS, R, T, E, F = cv2.stereoCalibrate(objpoints,imgpointsL,imgpointsR,OmtxL,distL,OmtxR,distR,
                                                           ChessImaR.shape[::-1], criteria_stereo,flags)


#Use stereoRectify() to calculate the mapping matrix for stereo correction
rectify_scale= 1 # If set to 0, the image will be cropped. If set to 1, all original image pixels will be retained.
RL, RR, PL, PR, Q, roiL, roiR= cv2.stereoRectify(MLS, dLS, MRS, dRS,
                                                 ChessImaR.shape[::-1], R, T,
                                                 rectify_scale,(0,0))
# Use the initUndistortRectifyMap function to calculate the mapping transformation for distortion correction and stereoscopic correction to achieve epipolar alignment.
Left_Stereo_Map= cv2.initUndistortRectifyMap(MLS, dLS, RL, PL,
                                             ChessImaR.shape[::-1], cv2.CV_16SC2)

Right_Stereo_Map= cv2.initUndistortRectifyMap(MRS, dRS, RR, PR,
                                              ChessImaR.shape[::-1], cv2.CV_16SC2)

#Stereoscopic correction effect display
for i in range(0,1): # Take the first pair of pictures as an example
    t = str(i)
    frameR = cv2.imread('/home/song/pic/right_' + t + '.jpg', 0)
    frameL = cv2.imread('/home/song/pic/left_' + t + '.jpg', 0)
    
    Left_rectified= cv2.remap(frameL,Left_Stereo_Map[0],Left_Stereo_Map[1], cv2.INTER_LANCZOS4, cv2.BORDER_CONSTANT, 0) # Use the remap function to complete the mapping
    im_L=Image.fromarray(Left_rectified) #numpy to image class
   
    Right_rectified= cv2.remap(frameR,Right_Stereo_Map[0],Right_Stereo_Map[1], cv2.INTER_LANCZOS4, cv2.BORDER_CONSTANT, 0)
    im_R=Image.fromarray(Right_rectified) #numpy to image class

#Create an area where two pictures can be placed side by side at the same time, and then paste the two pictures in order
    width = im_L.size[0]*2
    height = im_L.size[1]

    img_compare = Image.new('RGBA',(width, height))
    img_compare.paste(im_L,box=(0,0))
    img_compare.paste(im_R,box=(640,0))
    
    #Draw lines evenly on the polar-aligned picture
    for i in range(1,20):
        len=480/20
        plt.axhline(y=i*len, color='r', linestyle='-')
    plt.imshow(img_compare)
    plt.show()

5. Specific applications of binoculars

Binocular vision refers to simulating the human binocular vision system through images captured by two cameras (or cameras). It can provide more depth information and three-dimensionality, so it has specific applications in many fields.

Here are some specific applications of binocular vision:

  1. 3D reconstruction: From images captured by binocular cameras, stereo vision algorithms can be used to reconstruct the 3D structure of the scene. This is useful for areas such as computer-aided design, virtual reality, augmented reality, and more.

  2. Object detection and tracking: Binocular vision can provide more depth information, making object detection and tracking more accurate and stable. For example, in autonomous driving, binocular vision can be used to detect and track other vehicles, pedestrians, etc.

  3. Depth perception: The depth information of objects in the scene can be obtained through binocular vision, allowing depth perception and distance measurement. This is very useful in fields such as robot navigation, indoor positioning, and drone obstacle avoidance.

  4. Pose Estimation: Binocular vision can be used to estimate the pose and motion of an object or human body. For example, in human-computer interaction, binocular vision can be used for gesture recognition and tracking.

  5. Stereo matching: Stereo matching can be performed through binocular vision, that is, corresponding pixels in two images are matched. This is an important problem in computer vision and can be used for image registration, target recognition, etc.

  6. Visual SLAM: Binocular vision can be used in conjunction with the Simultaneous Localization and Mapping (SLAM) algorithm to achieve simultaneous positioning and map construction. This is very important in fields such as autonomous vehicles and drones.

The knowledge points of the article match the official knowledge files, and you can further learn related knowledge. OpenCV skill treeHomepageOverview 23607 people are learning the system