0 Preface
Autonomous driving technology is a cutting-edge field dominated by machine learning. Various algorithms of machine learning can be seen everywhere in the field of autonomous driving. Today, I will introduce to you lane line detection in autonomous driving technology.
1 Lane detection
Every task in the field of autonomous driving is quite complex and seems impossible to start. So when faced with such an extremely complex problem, the way we solve the problem is to first try to simplify the problem, and then try to solve the problem step by step from simplicity to difficulty. Lane line detection should be considered a relatively simple task in driverless driving. It relies on some related computer vision technologies and reads
Analyze the image data passed in by the camera and identify the lane line position. I think this is useful for lidar
It may be that there is nothing you can do. So today we will start with the simplest task and see what technologies can help us detect lane lines.
Let’s simplify the problem first. The so-called simplified problem is to use some conditional restrictions to reduce the problem of lane line detection. Let’s look at the data first, that is, the input algorithm is an image of a vehicle driving, and the output lane line position is.
More often than not, how do we deal with a more difficult task? Sometimes we don’t have any ideas when we get the task. Don’t be anxious and don’t think too much. Let’s start doing it step by step, starting with the simplest one and then go on. You will have ideas as you do it, and some problems will also be exposed. Let’s first find a video. I got this video from a project about lane line detection on the Internet, and I also used his ideas to do this. Well to start doing this now, the simplest thing is to read the video first and then display it on the screen for easy debugging.
2 Goals
Detect the position of lane lines in the image and provide lane line information for path planning.
3 Detection ideas
- Image grayscale processing
- Image Gaussian smoothing
- canny edge detection
- Area Mask
- Hough transform
- Draw lane lines
4 Code Implementation
4.1 Video image loading
?
import cv2
?import numpy as np
?import sys
?
import pygame
from pygame.locals import *
class Display(object):
def __init__(self,Width,Height):
pygame.init()
pygame.display.set_caption('Drive Video')
self.screen = pygame.display.set_mode((Width,Height),0,32)
def paint(self,draw):
self.screen.fill([0,0,0])
draw = cv2.transpose(draw)
draw = pygame.surfarray.make_surface(draw)
self.screen.blit(draw,(0,0))
pygame.display.update()
?
?
? if __name__ == "__main__":
? solid_white_right_video_path = "test_videos/Dancheng Senior Lane Line Detection.mp4"
? cap = cv2.VideoCapture(solid_white_right_video_path)
? Width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
? Height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
?
display = Display(Width,Height)
while True:
ret, draw = cap.read()
draw = cv2.cvtColor(draw,cv2.COLOR_BGR2RGB)
if ret == False:
break
display.paint(draw)
for event in pygame.event.get():
if event.type == QUIT:
sys.exit()
I won’t go into details about the above code. By default, everyone knows something about Python. There are also many code examples on the Internet about how to use opencv to read images. Everyone can understand it at a glance. Here because I am using mac
Sometimes there may be some problems displaying video images, so we use pygame to display opencv read images. Let everyone decide this based on their actual situation. It is worth mentioning that opencv
The read image is in BGR format. To display the image correctly in pygame, you need to convert the BGR to RGB format.
4.2 Lane marking area
Now this area is drawn based on the observation image,
?
def color_select(img,red_threshold=200,green_threshold=200,blue_threshold=200):
ysize,xsize = img.shape[:2]
color_select = np.copy(img)
rgb_threshold = [red_threshold, green_threshold, blue_threshold]
thresholds = (img[:,:,0] < rgb_threshold[0]) \
| (img[:,:,1] < rgb_threshold[1]) \
| (img[:,:,2] < rgb_threshold[2])
color_select[thresholds] = [0,0,0]
return color_select
The effect is as follows:
4.3 Region
The position of the lane line we want to detect is relatively fixed, usually in front of the car, so we draw it, that is, only detect the area we care about. Create a mask to filter out the areas you don’t care about and keep the areas you care about.
4.4 canny edge detection
Also related to edge detection is computer vision. First, use gradient changes to detect edges in the image. How to identify gradient changes in the image? The answer is the convolution kernel. The convolution kernel is to find the location where the gradient changes greatly on discontinuous pixels. we know
The sobal kernel can detect edges very well, so canny is optimized for sobal kernel detection.
? # Sample code, author Dan Cheng: Q746876041
? def canny_edge_detect(img):
? gray = cv2.cvtColor(img,cv2.COLOR_RGB2GRAY)
? kernel_size = 5
? blur_gray = cv2.GaussianBlur(gray,(kernel_size, kernel_size),0)
?
low_threshold = 180
high_threshold = 240
edges = cv2.Canny(blur_gray, low_threshold, high_threshold)
return edges
4.5 Hough transform
The Hough transform maps the lines in the x and y coordinate systems to points (m, b) in the Hough space. So the Hough transform is actually an operation from complex to simple (similar to dimensionality reduction). When using canny
After edge detection, the image can be submitted to Hough transform for recognition of simple graphics (lines, circles), etc. Here, Hough transform is used to find straight lines in canny edge detection results.
ignore_mask_color = 255
# Get image size
imshape = img.shape
# Define mask vertices
vertices = np.array([[(0,imshape[0]),(450, 290), (490, 290), (imshape[1],imshape[0])]], dtype=np.int32)
# Use fillpoly to draw the mask
cv2.fillPoly(mask, vertices, ignore_mask_color)
masked_edges = cv2.bitwise_and(edges, mask)
# Define parameters of Hough transform
rho=1
theta = np.pi/180
threshold = 2
min_line_length = 4 # Minimum number of pixels to form a line
max_line_gap = 5 # Maximum pixel spacing between connectable line segments
# Create an image for drawing lane lines
line_image = np.copy(img)*0
# Apply Hough transform to canny edge detection results
# The output "line" is an array containing the endpoints of the detected line segments
lines = cv2.HoughLinesP(masked_edges, rho, theta, threshold, np.array([]),
min_line_length, max_line_gap)
# Traverse the array of "lines" to draw on line_image
for line in lines:
for x1,y1,x2,y2 in line:
cv2.line(line_image,(x1,y1),(x2,y2),(255,0,0),10)
color_edges = np.dstack((edges, edges, edges))
import math
import cv2
import numpy as np
"""
Gray Scale
Gaussian Smoothing
Canny Edge Detection
Region Masking
Hough Transform
Draw Lines [Mark Lane Lines with different Color]
"""
class SimpleLaneLineDetector(object):
def __init__(self):
pass
def detect(self,img):
# Image grayscale processing
gray_img = self.grayscale(img)
print(gray_img)
#Image Gaussian smoothing
smoothed_img = self.gaussian_blur(img = gray_img, kernel_size = 5)
#canny edge detection
canny_img = self.canny(img = smoothed_img, low_threshold = 180, high_threshold = 240)
#area Mask
masked_img = self.region_of_interest(img = canny_img, vertices = self.get_vertices(img))
#hough transform
houghed_lines = self.hough_lines(img = masked_img, rho = 1, theta = np.pi/180, threshold = 20, min_line_len = 20, max_line_gap = 180)
# Draw lane lines
output = self.weighted_img(img = houghed_lines, initial_img = img, alpha=0.8, beta=1., gamma=0.)
return output
def grayscale(self,img):
return cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
def canny(self,img, low_threshold, high_threshold):
return cv2.Canny(img, low_threshold, high_threshold)
def gaussian_blur(self,img, kernel_size):
return cv2.GaussianBlur(img, (kernel_size, kernel_size), 0)
def region_of_interest(self,img, vertices):
mask = np.zeros_like(img)
if len(img.shape) > 2:
channel_count = img.shape[2]
ignore_mask_color = (255,) * channel_count
else:
ignore_mask_color = 255
cv2.fillPoly(mask, vertices, ignore_mask_color)
masked_image = cv2.bitwise_and(img, mask)
return masked_image
def draw_lines(self,img, lines, color=[255, 0, 0], thickness=10):
for line in lines:
for x1,y1,x2,y2 in line:
cv2.line(img, (x1, y1), (x2, y2), color, thickness)
def slope_lines(self,image,lines):
img = image.copy()
poly_vertices = []
order = [0,1,3,2]
left_lines = []
right_lines = []
for line in lines:
for x1,y1,x2,y2 in line:
if x1 == x2:
pass
else:
m = (y2 - y1) / (x2 - x1)
c = y1 - m * x1
if m < 0:
left_lines.append((m,c))
elif m >= 0:
right_lines.append((m,c))
left_line = np.mean(left_lines, axis=0)
right_line = np.mean(right_lines, axis=0)
?
? for slope, intercept in [left_line, right_line]:
?
rows, cols = image.shape[:2]
y1= int(rows)
y2= int(rows*0.6)
x1=int((y1-intercept)/slope)
x2=int((y2-intercept)/slope)
poly_vertices.append((x1, y1))
poly_vertices.append((x2, y2))
self.draw_lines(img, np.array([[[x1,y1,x2,y2]]]))
poly_vertices = [poly_vertices[i] for i in order]
cv2.fillPoly(img, pts = np.array([poly_vertices],'int32'), color = (0,255,0))
return cv2.addWeighted(image,0.7,img,0.4,0.)
def hough_lines(self,img, rho, theta, threshold, min_line_len, max_line_gap):
lines = cv2.HoughLinesP(img, rho, theta, threshold, np.array([]), minLineLength=min_line_len, maxLineGap=max_line_gap)
line_img = np.zeros((img.shape[0], img.shape[1], 3), dtype=np.uint8)
line_img = self.slope_lines(line_img,lines)
return line_img
def weighted_img(self,img, initial_img, alpha=0.1, beta=1., gamma=0.):
lines_edges = cv2.addWeighted(initial_img, alpha, img, beta, gamma)
return lines_edges
def get_vertices(self,image):
rows, cols = image.shape[:2]
bottom_left = [cols*0.15, rows]
top_left = [cols*0.45, rows*0.6]
bottom_right = [cols*0.95, rows]
top_right = [cols*0.55, rows*0.6]
ver = np.array([[bottom_left, top_left, top_right, bottom_right]], dtype=np.int32)
return ver
4.6 HoughLinesP detection principle
Next, entering the code section, the senior will explain in detail the meaning of the HoughLinesP parameter and how to use it.
?
? lines = cv2.HoughLinesP(cropped_image,2,np.pi/180,100,np.array([]),minLineLength=40,maxLineGap=5)
?
- The first parameter is the image we want to check the Hough accumulator array
- The second and third parameters are used to define how our Hough coordinates are divided into bins, that is, the accuracy of the small grid. We vote by passing a curve through the bin grid, and we decide the values of p and theta based on the number of votes. 2 means the width of our cell is in pixels.
We can divide the cells through the figure below. As long as the curve passes through the cells, the cells will be voted. We record the number of votes and the one with the most records is used as a parameter.
- If the defined size is too large, the accuracy will be lost. If the defined grid size is too small, although the accuracy will be improved, it will also increase the calculation time.
- The next parameter 100 means that the line we voted for above 100 is the line we are looking for that meets the requirements. That is to say, there need to be more than 100 lines intersecting here in the bin small grid. This is the parameter we are looking for.
- minLineLength gives 40 which means we check that the line length cannot be less than 40 pixels
- maxLineGap=5 as line break cannot be larger than 5 pixels
4.6.1 Define the method of displaying lane lines
?
? def disply_lines(image,lines):
?pass
Display the found lane lines by defining a function.
?
? line_image = disply_lines(lane_image,lines)
?
4.6.2 View the detection lane line data structure
?
? def disply_lines(image,lines):
? line_image = np.zeros_like(image)
? if lines is not None:
? for line in lines:
? print(line)
First define a matrix with the same size as the original image to draw and find the lane lines. We first determine whether the lane lines have been found. The return value of lines should not be None.
It is a matrix. We can simply print it to see the effect.
?
? [[704 418 927 641]]
? [[704 426 791 516]]
? [[320 703 445 494]]
? [[585 301 663 381]]
? [[630 341 670 383]]
4.6.3 Detect lane lines
Looking at the two-dimensional array of the data structure [[x1,y1,x2,y2]], this requires us to convert it into one-dimensional data [x1,y1,x2,y2]
? def disply_lines(image,lines):
? line_image = np.zeros_like(image)
? if lines is not None:
? for line in lines:
? x1,y1,x2,y2 = line.reshape(4)
? cv2.line(line_image,(x1,y1),(x2,y2),(255,0,0),10)
? return line_image
?
line_image = disply_lines(lane_image,lines)
cv2.imshow('result',line_image)
4.6.4 Synthesis
Regarding synthetic images, we superimpose and synthesize two images by giving them a certain weight.
4.6.5 Optimization
The detected lane lines are still not smooth enough. We need to optimize. The basic idea is to average the slopes and intercepts of these straight lines and then draw all detected points onto a straight line.
?
def average_slope_intercept(image,lines):
left_fit = []
right_fit = []
for line in lines:
x1, y1, x2, y2 = line.reshape(4)
parameters = np.polyfit((x1,x2),(y1,y2),1)
print(parameters)
Here the senior defines two arrays left_fit and right_fit to store the points of the lane lines on the left and right sides respectively. Let’s print the slope and intercept of the lines and use numpy
Provide the polyfit method to input two points and we can get the slope and intercept of the straight line passing through these points.
?
? [1. -286.]
?[1.03448276-302.27586207]
?[-1.672 1238.04]
?[1.02564103-299.
?
?
?[1.02564103-299.
?
def average_slope_intercept(image,lines):
left_fit = []
right_fit = []
for line in lines:
x1, y1, x2, y2 = line.reshape(4)
parameters = np.polyfit((x1,x2),(y1,y2),1)
# print(parameters)
slope = parameters[0]
intercept = parameters[1]
if slope < 0:
left_fit.append((slope,intercept))
else:
right_fit.append((slope,intercept))
print(left_fit)
print(right_fit)
?
Let’s output the size of the picture. Our picture starts with its upper left corner as the origin 0, 0, so we draw a straight line from the bottom of the picture more than 700 upwards. We don’t need to draw the whole thing, we can just intercept part of it.
?
def make_coordinates(image, line_parameters):
slope, intercept = line_parameters
y1 = image.shape[0]
y2 = int(y1*(3/5))
x1 = int((y1 - intercept)/slope)
x2 = int((y2 - intercept)/slope)
# print(image.shape)
return np.array([x1,y1,x2,y2])
So the straight line starts and ends and we give y1,y2 and then we find x in terms of y through the slope and intercept of the equation.
? averaged_lines = average_slope_intercept(lane_image,lines);
? line_image = disply_lines(lane_image,averaged_lines)
? combo_image = cv2.addWeighted(lane_image,0.8, line_image, 1, 1,1)
?
cv2.imshow('result',combo_image)
5 Finally
This project is relatively new and suitable as a competition topic. It is highly recommended by senior students!
More information, project sharing:
https://gitee.com/dancheng-senior/postgraduate