1. Introduction:
Version: opencv-python 4.8.1.78 python 3.11 (csdn also has similar examples, but the version is relatively old, so I implemented it myself)
This small functional demo mainly uses the matchTemplate function of cv’s template matching function. For some text and pictures with fixed templates, you can refer to the following code to use templates. (ps: Although using torch to train the model can provide more versatile functions, for images with fixed text formats such as credit cards, it is more convenient to use template matching)
2. Implementation ideas
First of all, the picture below is a visa credit card. This is our input. What we want to get is the number in the red box.
So our first step is to think about how to get this red box. In CV, generally we can use boundary detection to get the boundary information. The best estimate is to remove the gray from the background, leaving white and gold text. Information, at this time, the boundaries of the text will be close to each other. We can use the expansion and erosion, that is, the closing operation, to obtain the adjacent boundary blocks. Then filter out the text part we need according to the size of the boundary block. Finally, template numbers are used to realize text recognition.
Template numbers:
3. Code implementation
3.1 Some common function declarations
import cv2
import numpy as np
# Basically the use of these two libraries, the following code will not be repeated.
#Display cv pictures
def cv_show(name,img):
cv2.imshow(name,img)
cv2.waitKey(0)
cv2.destroyWindow(name)
#Do proportional scaling based on the given width or height
def resize(image, width=None, height=None, inter=cv2.INTER_AREA):
dim=None
(h, w) = image.shape[:2]
if width is None and height is None:
return image
if width is None:
r = height / float(h)
dim = (int(w * r), height)
else:
r = width / float(w)
dim = (width, int(h * r))
resized = cv2.resize(image, dim, interpolation=inter)
return resized
#Function for sorting contours
def sort_contours(contours,method = 'left-to-right'):
reverse=False
i = 0
if method == "right-to-left" or method == "bottom-to-top":
reverse = True
if method == "top-to-bottom" or method == "bottom-to-top":
i = 1
boundingBoxes = [cv2.boundingRect(c) for c in contours] #Use a smallest rectangle to wrap the found shape x, y, h, w
(cnts, boundingBoxes) = zip(*sorted(zip(contours, boundingBoxes),
key=lambda b: b[1][i], reverse=reverse))
return cnts, boundingBoxes
To put it simply:
cv_show: used to display pictures
sort_contours(contours,method = ‘left-to-right’) —->cnts,boundingBoxes:
Used to sort the incoming contours from left to right/top to bottom, cnts: is the sorted contour list, boundingBoxes is the corresponding bounding rectangle list
resize(image, width=None, height=None, inter=cv2.INTER_AREA)—–>image :
Used to scale the incoming image proportionally to the given length or width, and output the scaled image.
3.2 Template feature extraction:
First, let’s take a look at our template. Obviously we can use edge detection to pull the boundaries of the text. Finally, as long as we draw a list of bounding rectangles with more outlines, we can divide it into 0-9 templates.
Code:
temp_img = cv2.imread(r'.\images\ocr_a_reference.png')
gray_temp_img = cv2.cvtColor(temp_img,cv2.COLOR_BGR2GRAY)
#threshold processing
ref = cv2.threshold(gray_temp_img,10,255,cv2.THRESH_BINARY_INV)[1]#Binarization, reverse
#edge detection
contours,_= cv2.findContours(ref.copy(),cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
cv2.drawContours(temp_img,contours,-1,(0,0,255),3)
#contour sorting
contours,_ = sort_contours(contours, method='left-to-right')
digits = {}
for (i,c) in enumerate(contours):
(x,y,w,h) = cv2.boundingRect(c)
roi = ref[y:y + h,x:x + w]
roi = cv2.resize(roi,(57,88)) #Enlarge the box
# cv_show('roi',roi)
digits[i] = roi
Briefly analyze:
1.temp_img is used to store the read template image and then convert it into a grayscale image (this step cannot be omitted. It seems to be the same after conversion. In fact, the grayscale image only has a single color channel, and the subsequent threshold processing is also Must be grayscale input).
2. Then perform threshold processing. (ps: cv2.threshold has two return values. We only need the second return value, which is the processed object.)
3.cv2.findContours completes edge detection and draws the edges (it doesn’t matter whether you draw or not, but you can see the effect)
4. Call sort_contours (custom sorting function) to sort the detected edges
5. Finally, we use the boundingRect function to calculate the bounding rectangle for the sorted contours, and store the rectangle corresponding to each number into the digits dictionary. This way we have a digital template. It is roughly as shown in the figure below:
3.3 Preprocessing credit card images:
Code:
img = cv2.imread(r'.\images\credit_card_04.png')
img = resize(img,width=300)#Scale proportionally, giving width or length is like
img_copy = img.copy()
gray_img = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
#Highlighting features, top hat operation
tophat = cv2.morphologyEx(gray_img,cv2.MORPH_TOPHAT,rectKernel)
#boundary detection
gradX = cv2.Sobel(tophat,ddepth=cv2.CV_32F,dx=1,dy=0,ksize=-1)#ksize=-1 Use 3*3
#Get the absolute value
gradX = np.absolute(gradX)
#Normalized
(minVal,maxVal) = (np.min(gradX),np.max(gradX))
gradX = (255*(gradX-minVal)/(maxVal-minVal))
gradX = gradX.astype("uint8")
There is not much to say about this process. The basic comments have been written. The only point to note is that generally speaking, when we use the Sobel operator to calculate the boundary, we will first calculate x and then y, and then combine the two in proportion. , but the Sobel operator is used here for y, and the effect is very poor, so I only used the x direction.
Image after topper:
After boundary extraction:
It looks blurry, right, but don’t forget, we are not trying to extract every valid contour accurately, but we are trying to determine where they are.
3.4 Determine the position of significant digits
Do you still remember what I said at the beginning? You see, after we obtain the fuzzy contour map above, once we use the closing operation (first expansion and then erosion), then as long as the convolution kernel we choose is of appropriate size, we can A series of silhouettes merged into one piece. (ps: The convolution kernel size is set based on experience and actual results)
Code:
#Defines two different convolution kernels rectKernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(9,3)) sqeKernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(7,7)) gradX = cv2.morphologyEx(gradX,cv2.MORPH_CLOSE,rectKernel) #Binarization thresh = cv2.threshold(gradX,0,255,cv2.THRESH_BINARY|cv2.THRESH_OTSU)[1] #There is a gap. If you don’t want a big rectangle, then do another closing operation. thresh = cv2.morphologyEx(thresh,cv2.MORPH_CLOSE,sqeKernel)
This is the image after the closing operation:
We have roughly obtained some block areas, but they don’t seem to be so full. Then we can do another closing operation to make the content fuller, so that we can more accurately frame the area later.
Second closing operation:
Doesn’t it look more pleasing to the eye like this? Normally, how many closing operations are needed to achieve a perfect acquisition? This requires a small batch experiment on the samples. Based on the data in my hand, I selected about 30. , including all numbers from 0 to 9, basically twice will have a good effect. How should we obtain the area after obtaining the above picture? Oh, by the way, it would be nice to do boundary detection again.
thresh_contours,_ = cv2.findContours(thresh.copy(),cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE) thresh_img=img_copy cv2.drawContours(thresh_img,thresh_contours,-1,(0,0,255),2)
3.5 Candidate box screening
After obtaining the picture above, we only need to filter each border to see which ones meet our needs. I used the actual aspect ratio, width, and length to filter. Of course, we can set different conditions according to the situation. Filter criteria.
Code:
locs=[] #Storage the desired area
for (i,c) in enumerate(thresh_contours):
(x,y,w,h) = cv2.boundingRect(c)
ar = w/float(h) #aspect ratio
if(w>40 and w<55) and (h>10 and h<20):
locs.append((x,y,w,h))
# Obtained 4 outlines, and sorted them by the way. In fact, there are 4 parameters, x, y, width and height.
locs= sorted(locs,key=lambda x:x[0])
3.6 Card number identification
Remember, in the first step, we used the template to extract features and store them in the roi parameters.
Code:
output = []#storage card number
for (i,(gX,gY,gW,gH)) in enumerate(locs):
groupOutput=[]
group = gray_img[gY-5:gY + gH + 5,gX-5:gX + gW + 5] #Relaxed by 5 pixels to prevent missing elements
# cv_show('',group)
#trheshold 0 means the system automatically obtains the threshold
group = cv2.threshold(group,0,255,
cv2.THRESH_BINARY|cv2.THRESH_OTSU)[1]
cv_show('group',group)
group_contours,_ = cv2.findContours(group.copy(),cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE)
digit_contours = sort_contours(group_contours,method='left-to-right')[0]
for c in digit_contours:
(x, y, w, h) = cv2.boundingRect(c)
roi = group[y:y + h, x:x + w]
roi = cv2.resize(roi, (57, 88)) #Resize to the same size as the template
# cv_show('roi', roi)
scores=[]
for (digit, digitROI) in digits.items():
# Template matching
result = cv2.matchTemplate(roi, digitROI,
cv2.TM_CCOEFF)
(_, score, _, _) = cv2.minMaxLoc(result)
scores.append(score)
# Get the most appropriate number
groupOutput.append(str(np.argmax(scores)))
# frame
cv2.rectangle(img, (gX - 5, gY - 5),
(gX + gW + 5, gY + gH + 5), (0, 0, 255), 1)
cv2.putText(img, "".join(groupOutput), (gX, gY - 15),
cv2.FONT_HERSHEY_SIMPLEX, 0.65, (0, 0, 255), 2)
# got the answer
output.extend(groupOutput)
Let’s briefly talk about the process:
1. We use the previously stored x, y, w, h information to extract the corresponding box from the picture.
2. In the for c in digit_contours: loop, we will segment the deducted image, still using the old routine, edge extraction, and then draw a rectangle, and finally cut out the digits based on the rectangle. There is a point to note here. What we did before When doing template feature extraction, one step is actually to expand the image to the size of 57*88. Since we want to do template matching, we also need to synchronize the size.
3. Finally, use the matchTemplate function to do a matching degree calculation, then select the highest one as the recognition result, and then output it. The rendering:
The complete code is attached: This data set is so easy to find. Just find some credit card pictures on the Internet. I won’t post them here.
import cv2
import numpy as np
def sort_contours(contours,method = 'left-to-right'):
reverse=False
i = 0
if method == "right-to-left" or method == "bottom-to-top":
reverse = True
if method == "top-to-bottom" or method == "bottom-to-top":
i = 1
boundingBoxes = [cv2.boundingRect(c) for c in contours] #Use a smallest rectangle to wrap the found shape x, y, h, w
(cnts, boundingBoxes) = zip(*sorted(zip(contours, boundingBoxes),
key=lambda b: b[1][i], reverse=reverse))
return cnts, boundingBoxes
def resize(image, width=None, height=None, inter=cv2.INTER_AREA):
dim=None
(h, w) = image.shape[:2]
if width is None and height is None:
return image
if width is None:
r = height / float(h)
dim = (int(w * r), height)
else:
r = width / float(w)
dim = (width, int(h * r))
resized = cv2.resize(image, dim, interpolation=inter)
return resized
#Specify credit card type
FIRST_NUMBER = {
"3": "American Express",
"4": "Visa",
"5": "MasterCard",
"6": "Discover Card"
}
def cv_show(name,img):
cv2.imshow(name,img)
cv2.waitKey(0)
cv2.destroyWindow(name)
temp_img = cv2.imread(r'.\images\ocr_a_reference.png')
# test = cv2.imread(r'.\images\credit_card_05.png')
# cv_show('img',temp_img)
gray_temp_img = cv2.cvtColor(temp_img,cv2.COLOR_BGR2GRAY)
# cv_show('gray',gray_temp_img)
#Binarization
ref = cv2.threshold(gray_temp_img,10,255,cv2.THRESH_BINARY_INV)[1]#Binarization, reverse
# cv_show('ref',ref)
#Contour detection CHAIN_APPROX_SIMPLE only records endpoints RETR_EXTERNAL only calculates the outer contour
contours,_ = cv2.findContours(ref.copy(),cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
#(0,0,255) Drawing color 3 Line segment pixels
cv2.drawContours(temp_img,contours,-1,(0,0,255),2)
# cv_show('img',temp_img )
contours,_ = sort_contours(contours, method='left-to-right')# Sort the contours. Note that idex=0 represents the contour of 9
digits = {}
# cv2.drawContours(copy,contours,contourIdx=0,color = (0,0,255),thickness=3)
# cv_show('',copy)
#Get the picture of the template
for (i,c) in enumerate(contours):
(x,y,w,h) = cv2.boundingRect(c)
roi = ref[y:y + h,x:x + w]
roi = cv2.resize(roi,(57,88)) #Enlarge the box
# cv_show('roi',roi)
digits[i] = roi
#Generate a structural element in a morphological operation (similar to a convolution kernel)
rectKernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(9,3))
sqeKernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(7,7))
img = cv2.imread(r'.\images\credit_card_04.png')
# cv_show('credit',img)
img = resize(img,width=300)#Scale proportionally, giving width or length is like
img_copy = img.copy()
gray_img = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
# cv_show('gray',gray_img)
# Tophat operation to highlight brighter areas
tophat = cv2.morphologyEx(gray_img,cv2.MORPH_TOPHAT,rectKernel)
# _,threshold_img = cv2.threshold(gray_img,150,255,cv2.THRESH_BINARY)
# tophat = threshold_img
cv_show('tophat',tophat)
# cv_show('',threshold_img)
#boundary detection
gradX = cv2.Sobel(tophat,ddepth=cv2.CV_32F,dx=1,dy=0,ksize=-1)#ksize=-1 Use 3*3
#300*188
#Absolute value
gradX = np.absolute(gradX)
#Normalized
(minVal,maxVal) = (np.min(gradX),np.max(gradX))
gradX = (255*(gradX-minVal)/(maxVal-minVal))
gradX = gradX.astype("uint8")
#At this step, we have framed the wheel buttons of each number, but they are very scattered and not piece by piece as we want, so we use closing (first expansion and then erosion) to complete the merged outline.
# print(np.array(gradX).shape)
cv_show('gradx',gradX)
#Close operation
gradX = cv2.morphologyEx(gradX,cv2.MORPH_CLOSE,rectKernel)
cv_show('',gradX)
thresh = cv2.threshold(gradX,0,255,cv2.THRESH_BINARY|cv2.THRESH_OTSU)[1]
cv_show('thresh',thresh)
#There is a gap. If you don’t want a big rectangle, then do another closing operation.
thresh = cv2.morphologyEx(thresh,cv2.MORPH_CLOSE,sqeKernel)
cv_show('thresh2',thresh)
#Then generate the outline box
thresh_contours,_ = cv2.findContours(thresh.copy(),cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
thresh_img=img_copy
cv2.drawContours(thresh_img,thresh_contours,-1,(0,0,255),2)
cv_show('img',thresh_img)
locs=[] #Storage the desired area
for (i,c) in enumerate(thresh_contours):
(x,y,w,h) = cv2.boundingRect(c)
ar = w/float(h) #aspect ratio
if(w>40 and w<55) and (h>10 and h<20):
locs.append((x,y,w,h))
#Get 4 contours
locs= sorted(locs,key=lambda x:x[0])
output = []
for (i,(gX,gY,gW,gH)) in enumerate(locs):
groupOutput=[]
group = gray_img[gY-5:gY + gH + 5,gX-5:gX + gW + 5] #Relaxed by 5 pixels to prevent missing elements
# cv_show('',group)
#trheshold 0 means the system automatically obtains the threshold
group = cv2.threshold(group,0,255,
cv2.THRESH_BINARY|cv2.THRESH_OTSU)[1]
cv_show('group',group)
group_contours,_ = cv2.findContours(group.copy(),cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE)
digit_contours = sort_contours(group_contours,method='left-to-right')[0]
for c in digit_contours:
(x, y, w, h) = cv2.boundingRect(c)
roi = group[y:y + h, x:x + w]
roi = cv2.resize(roi, (57, 88)) #Resize to the same size as the template
cv_show('roi', roi)
scores=[]
for (digit, digitROI) in digits.items():
# Template matching
result = cv2.matchTemplate(roi, digitROI,
cv2.TM_CCOEFF)
(_, score, _, _) = cv2.minMaxLoc(result)
scores.append(score)
# Get the most appropriate number
groupOutput.append(str(np.argmax(scores)))
# frame
cv2.rectangle(img, (gX - 5, gY - 5),
(gX + gW + 5, gY + gH + 5), (0, 0, 255), 1)
cv2.putText(img, "".join(groupOutput), (gX, gY - 15),
cv2.FONT_HERSHEY_SIMPLEX, 0.65, (0, 0, 255), 2)
# got the answer
output.extend(groupOutput)
print(output)
# print results
# print("Credit Card Type: {}".format(FIRST_NUMBER[output[0]]))
# print("Credit Card #: {}".format("".join(output)))
cv2.imshow("Image", img)
cv2.waitKey(0)
ps: emmm, you’ve all seen this, please give it a thumbs up
Okay, let’s write some extensions that we are thinking of now. For example, we can use Torch to train a model. This model can find the area of the credit card in a picture and identify whether it is the front. The label is roughly represented by 4 coordinate points + 0. The reverse side 1 means the front side. If it is the front side, we make a perspective transformation and then put it on. It is a small project to take pictures and obtain the card number.