Citrus color range size measurement system based on OpenCV

1. Research background and significance

Project ReferenceAAAI Association for the Advancement of Artificial Intelligence

research background and meaning

Citrus is an important fruit with rich nutritional value and wide market demand. Accurately measuring the color range and size of citrus is very important during the growing and selling of citrus. Traditional methods of measuring citrus color range and size usually require manual operations, which are time-consuming, labor-intensive and error-prone. Therefore, developing a citrus color range size measurement system based on OpenCV has important research significance and practical application value.

First of all, the color range of citrus is one of the important indicators of the maturity and quality of citrus. By measuring the color range of citrus, the maturity and quality of citrus can be accurately judged, helping growers and sellers perform reasonable picking and classification, and improving the market competitiveness of citrus.

Secondly, the size of citrus is one of the important factors that determine citrus yield and market value. Traditional dimensional measurement methods usually require manual operations, are prone to errors, and are time-consuming and labor-intensive. The citrus size measurement system based on OpenCV can realize automated size measurement through image processing and computer vision technology, improve measurement accuracy and efficiency, and reduce labor costs.

In addition, the citrus color range size measurement system based on OpenCV can also be applied to the quality inspection and grading of citrus. By measuring the color range and size of citrus, citrus can be automatically classified and graded, improving the quality control and market competitiveness of citrus.

Finally, the citrus color range size measurement system based on OpenCV can also provide scientific basis and technical support for citrus planting and sales. By establishing a database of citrus color ranges and sizes, it can provide reference and guidance for citrus planting and sales, help growers and sellers formulate reasonable planting and sales strategies, and improve citrus yield and market competitiveness.

In summary, the citrus color range size measurement system based on OpenCV has important research significance and practical application value. By developing this system, automated measurement of the color range and size of citrus can be achieved, improving measurement accuracy and efficiency, providing scientific basis and technical support for citrus planting and sales, and promoting the development and improvement of the citrus industry.

2. Picture demonstration

2.png

4.png

5.png

3. Video demonstration

Citrus color range size measurement system based on OpenCV_bilibili_bilibili

4. Color channel selection

First tried different color channels as an example.
image.png
image.png

5. Core code explanation

5.1 color.py

The code after encapsulating it into a class is as follows:

class ImageProcessing:
    def __init__(self):
        self.work_book = xlwt.Workbook()
        self.work_sheet = self.work_book.add_sheet('Test')

    def mosaic(self, img, p1, p2):
        im = img
        im2 = cv2.flip(im, 0, im)
        im2resize = cv2.resize(im2, (abs(p1[0] - p2[0]), abs(p1[1] - p2[1])))
        im[min(p1[1], p2[1]):max(p1[1], p2[1]), min(p1[0], p2[0]):max(p1[0], p2[ 0])] = im2resize
        return im

...

The program file name is color.py, and its main functions are as follows:

  1. Introduced cv2, numpy, random, PIL, os, xlwt and other libraries.
  2. A workbook named work_book is created.
  3. A sheet named Test is added to the workbook.
  4. A series of functions are defined, including mosaic, mixup, cutout, enlarge_2x, enlarge_4x, affine, Perspective, batch, contours_in, findmax_box, draw_wh, caiyang, findmax_box2 and sort_juzi.
  5. The batch function is used to batch process images under the specified path, including operations such as mosaic, cutout, mixup, enlarge_2x, enlarge_4x, affine, and perspective, and save the processed images to the specified path.
  6. The contours_in function is used to obtain the coordinates of all points within the contour.
  7. The findmax_box function is used to find the largest brown area in the image and draw the border of this area on the image
5.2 cutout.py

Here is the core part of encapsulating the code as a class:

class Cutout:
    def __init__(self, image_path):
        self.image_path = image_path
        self.im = cv2.imread(image_path)

    def generate_mask(self):
        p1 = (random.randint(0, self.im.shape[1]), random.randint(0, self.im.shape[0]))
        p2 = (random.randint(0, self.im.shape[1]), random.randint(0, self.im.shape[0]))
        cv2.rectangle(self.im, (min(p1[0], p2[0]), min(p1[1], p2[1])), (max(p1[0], p2[0]), max(p1[1], p2[1])), [0, 0, 0], -1)

    def show_image(self):
        cv2.imshow('cutout', self.im)
        cv2.waitKey(0)

This class is named Cutout, and the image path is passed in during initialization. The generate_mask method generates a random rectangular area and draws a black mask on the image. The show_image method displays the processed image. Finally, create a Cutout object and call the generate_mask and show_image methods to achieve the same function.

This program file is called cutout.py, which uses the OpenCV library to implement an image cutting function. First, it reads an image file called 1.jpg. Then, it randomly generates two points p1 and p2, which are used to determine a rectangular area. Next, the program uses this rectangular area to draw a black mask on the image, setting the pixel values in this area to 0. Finally, the program displays the processed image and waits for the user to press any key to close the window.

5.3 cut_hsv_range.py
class ImageProcessor:
    def __init__(self, image_path):
        self.image = cv2.imread(image_path)
        self.imagecopy = self.image.copy()
        self.list1 = []
        self.list2 = []
        self.num = 0

    def resize_image(self):
        self.image = cv2.resize(self.image, (0, 0), fx=0.3, fy=0.3, interpolation=cv2.INTER_NEAREST)

    def getpos(self, event, x, y, flags, param):
        HSV3 = self.HSV2.copy()
        if event == cv2.EVENT_MOUSEMOVE:
            self.HSV = HSV3
            cv2.line(self.HSV, (0, y), (self.HSV.shape[1] - 1, y), (255, 255, 255), 1, 4)
            cv2.line(self.HSV, (x, 0), (x, self.HSV.shape[0] - 1), (255, 255, 255), 1, 4)
            cv2.imshow("imageHSV", self.HSV)
        elif event == cv2.EVENT_LBUTTONDOWN:
            self.num + = 1
            self.HSV = HSV3
            if self.num == 1:
                self.list1.append([x, y])
                print('Please click on the second point on the HSV image')
            if self.num == 2:
                self.num = 0
                self.list2.append([x, y])
                for i in range(min(self.list1[-1][0], self.list2[-1][0]), max(self.list1[-1][0], self.list2[-1] [0])):
                    for j in range(min(self.list1[-1][1], self.list2[-1][1]), max(self.list1[-1][1], self.list2[-1] [1])):
                        self.hlist.append(self.HSV[j, i][0])
                        self.slist.append(self.HSV[j, i][1])
                        self.vlist.append(self.HSV[j, i][2])
                self.hlist.sort()
                self.slist.sort()
                self.vlist.sort()
                print(self.hlist)
                print(self.slist)
                print(self.vlist)
                print('Please click on the first point on the HSV image')
                print((self.hlist[0], self.slist[0], self.vlist[0]), (self.hlist[-1], self.slist[-1], self.vlist[-1]) )

    def process_image(self):
        self.resize_image()
        self.HSV = self.image.copy()
        self.HSV2 = self.image.copy()
        self.num = 0
        self.hlist = []
        self.slist = []
        self.vlist = []

        cv2.imshow("imageHSV", self.HSV)
        cv2.setMouseCallback("imageHSV", self.getpos)
        cv2.waitKey(0)
...

The file name of this program is cut_hsv_range.py. Its function is to determine a rectangular range by clicking on the HSV image and output the minimum and maximum pixel values within the range. The main steps of the procedure are as follows:

  1. Import the required libraries: cv2, numpy and matplotlib.pyplot.
  2. Read a picture (C2.JPG) and make a backup.
  3. Define some variables and lists to store the coordinates and pixel values of the click.
  4. Enters an infinite loop until any key on the keyboard is pressed to exit.
  5. Resize the image to 0.3 times its original size.
  6. Make a copy of the image, used to draw guide lines when the mouse is moved.
  7. Define a mouse event callback function to handle mouse movement and click events.
  8. In the mouse move event, horizontal and vertical auxiliary lines are drawn and displayed on the HSV picture.
  9. In the mouse click event, the coordinates of the click are recorded based on the number of clicks, and the pixel value within the rectangular range is calculated.
  10. Sort the pixel values and output the minimum and maximum values.
  11. Display the HSV image and set the callback function for mouse events.
  12. Wait for any key on the keyboard to be pressed.
  13. The program ends.
5.4 data_enhance.py
class ImageProcessing:
    def __init__(self, path, savepath):
        self.path = path
        self.savepath = savepath

    def process_images(self):
        if not os.path.exists(self.savepath):
            os.mkdir(self.savepath)
        list = os.listdir(self.path)
        for i in list:
            img = cv2.imread(self.path + '/' + i)
            imgcopy1 = img.copy()
            imgcopy2 = img.copy()
            imgcopy3 = img.copy()
            imgcopy4 = img.copy()
            imgcopy5 = img.copy()
            cv2.imwrite(self.savepath + '/' + 'original-' + i, img, [int(cv2.IMWRITE_JPEG_QUALITY), 100])
            res = cv2.resize(imgcopy1, None, fx=2, fy=2, interpolation=cv2.INTER_CUBIC)
            cv2.imwrite(self.savepath + '/' + '2x-' + i, res, [int(cv2.IMWRITE_JPEG_QUALITY), 100])
            res = cv2.resize(res, None, fx=2, fy=2, interpolation=cv2.INTER_CUBIC)
            cv2.imwrite(self.savepath + '/' + '4x-' + i, res, [int(cv2.IMWRITE_JPEG_QUALITY), 100])
            rows, cols, ch = img.shape
            M = np.float32([[1, 0, 100], [0, 1, 50]])
            dst = cv2.warpAffine(imgcopy2, M, (cols, rows))
            cv2.imwrite(self.savepath + '/' + 'move-' + i, dst, [int(cv2.IMWRITE_JPEG_QUALITY), 100])
            rows, cols, ch = img.shape
            M = cv2.getRotationMatrix2D(((cols - 1) / 2.0, (rows - 1) / 2.0), 90, 1)
            dst = cv2.warpAffine(imgcopy3, M, (cols, rows))
            cv2.imwrite(self.savepath + '/' + '90-' + i, dst, [int(cv2.IMWRITE_JPEG_QUALITY), 100])
            rows, cols, ch = img.shape
            pts1 = np.float32([[50, 50], [200, 50], [50, 200]])
            pts2 = np.float32([[10, 100], [200, 50], [100, 250]])
            M = cv2.getAffineTransform(pts1, pts2)
  ...

The program file name is data_enhance.py, which uses the OpenCV library to enhance images. The main function of the program is to perform various processing on the images under the specified path and save them to the specified saving path.

The specific process of the program is as follows:

  1. Import the required libraries: cv2, numpy and os.
  2. Set the input path and save path.
  3. If the save path does not exist, create the save path.
  4. Get a list of all files under the input path.
  5. Traverse the file list and process each file.
  6. Use the cv2.imread function to read the image.
  7. Duplicate the original image, creating 5 copies.
  8. Use the cv2.imwrite function to save the original image to the save path and name it ‘original-‘ + file name.
  9. Use the cv2.resize function to enlarge the image copy by 2 times, save it to the save path, and name it ‘2x-‘ + file name.
  10. Use the cv2.resize function again to enlarge the image obtained in the previous step by 2 times, save it to the save path, and name it ‘4x-‘ + file name.
  11. Use the cv2.warpAffine function to pan the image copy, save it to the save path, and name it ‘move-‘ + file name.
  12. Use the cv2.getRotationMatrix2D function to rotate the image copy (90 degrees) and save it to the save path and name it ’90-‘ + file name.
  13. Use the cv2.getAffineTransform function to perform an affine transformation operation on the image copy, save it to the save path, and name it ‘fang-‘ + file name.
  14. Use the cv2.getPerspectiveTransform function to perform a perspective transformation operation on the image copy, save it to the save path, and name it ‘tou-‘ + file name.

In short, this program generates multiple enhanced images by performing operations such as magnification, translation, rotation, affine transformation, and perspective transformation on the image, and saves them to the specified path.

5.5 mixup.py

class ImageBlender:
    def __init__(self, img1_path, img2_path):
        self.img1 = Image.open(img1_path).convert('RGBA')
        self.img2 = Image.open(img2_path).convert('RGBA')
        self.img2 = self.img2.resize(self.img1.size)
    
    def blend_images(self, alpha):
        return Image.blend(self.img1, self.img2, alpha)
    
    def save_blend_image(self, output_path, alpha):
        img = self.blend_images(alpha)
        img.save(output_path)
    
.......

The program file is called mixup.py and it uses PIL and the cv2 library. The function of the program is to mix two pictures and save the mixed picture.

The program first imported the PIL library and cv2 library. Then, it uses the Image.open() function to open two images, named img1 and img2, and convert them to RGBA mode.

Next, the program uses the img2.resize() function to resize img2 to the same size as img1.

Then, the program uses the Image.blend() function to mix img1 and img2 with a weight of 0.5 to generate the mixed image img.

Finally, the program uses the img.show() function to display the blended image, and uses the img.save() function to save the blended image as a blend.jpg file.

5.6 mosaic.py
class ImageCutout:
    def __init__(self, image_path1, image_path2):
        self.im = cv2.imread(image_path1)
        self.im2 = cv2.imread(image_path2)
    
    def cutout(self):
        p1 = (random.randint(0, self.im.shape[1]), random.randint(0, self.im.shape[0]))
        p2 = (random.randint(0, self.im.shape[1]), random.randint(0, self.im.shape[0]))
        im2resize = cv2.resize(self.im2, (abs(p1[0] - p2[0]), abs(p1[1] - p2[1])))
        self.im[min(p1[1], p2[1]):max(p1[1], p2[1]), min(p1[0], p2[0]):max(p1[0], p2[0])] = im2resize
        cv2.imshow('cutout', self.im)
        cv2.waitKey(0)
.......

The file name of this program is mosaic.py, and its function is to stitch two pictures together. The program first introduces two libraries, cv2 and random. Then two pictures, 1.jpg and 2.jpg, were read through the cv2.imread function. Next, the program randomly generates two points p1 and p2, whose coordinates are within the range of the image im. Then, the program scales the image im2 and adjusts its size to the distance between the two points p1 and p2. Finally, the program replaces the im2resize part to the corresponding position of im, and displays the result through the cv2.imshow function. The user can close the display window by pressing any key.

6. Overall system structure

Summary of overall functions and architecture:
This project is a citrus color range size measurement system based on OpenCV. It contains multiple program files, each with different functions for processing images, measuring citrus color ranges and sizes, etc.

The following is a summary of the functions of each file:

File name Function
color.py Realize the measurement of the citrus color range, including processing the image and finding the largest brown area
cutout.py Realize the image Cutting function, determine the rectangular area by clicking the mouse and set the pixel value in the area to 0
cut_hsv_range.py Confirm by clicking on the HSV image Rectangular range, and output the minimum and maximum values of pixel values within the range
data_enhance.py Enhance the image, including amplification, Operations such as translation, rotation, affine transformation and perspective transformation
mixup.py Mix two pictures to generate a mixed picture
mosaic.py Splice two pictures

The above is a brief overview of the functionality of each file. Each file has different functions and is used to achieve operations such as measurement, image processing and enhancement of citrus color ranges.

7. Binarization processing

Since the image is mainly distributed in the red and green channel space, the normalized image is divided into red and green, that is, grayscale processing.

The idea is different from the normal grayscale brightness value. Change the comparison method to the value size in the red and green channels.
image.png

8. Morphological operations

Perform morphological operations on the processed images.

Filling small holes: Open operations

Stem treatment: expansion

In order to prevent serious distortion of the image, the kernel should not be too large. Therefore, an opening operation is first used to fill in the white noise, and then the size of the expanded kernel is gradually reduced and expanded four times in a row.
image.png

9. System integration

The complete source code and environment deployment video tutorial below
1.png

Reference blog “Citrus color range size measurement system based on OpenCV”

10. References

[1] Lei Huan, Wu Liangsheng, Jiao Zeyu, et al. Research on automatic detection method of mature bayberry in orchard environment [J]. Automation and Information Engineering. 2021, (3). DOI: 10.3969/j.issn.1674-2605.2021.03.002.

[2] Liu Tianzhen, Teng Guifa, Yuan Yingchun, et al. Winter jujube fruit identification method in natural scenes based on improved YOLO v3 [J]. Journal of Agricultural Machinery. 2021, (5). DOI: 10.6041/j.issn.1000-1298.2021 .05.002.

[3] Zhao Hui, Li Hao, Yue Youjun, et al. Identification and positioning of dwarf apples based on RGB-D camera [J]. Computer Engineering and Design. 2020, (8). DOI: 10.16208/j.issn1000- 7024.2020.08.030.

[4] Yan Bin, Guo Wenchuan. Detection method of ‘Hayward’ kiwi fruit enlargement based on K-means clustering and calyx shape [J]. Journal of Northwest A&F University (Natural Science Edition). 2020, (5). DOI:10.13207/j.cnki.jnwafu.2020.05.018 .

[5] Kou Dalei, Quan Jichuan, Zhang Zhongwei. Research on progress of target detection framework based on deep learning [J]. Computer Engineering and Applications. 2019, (11). DOI: 10.3778/j.issn.1002-8331.1902-0254.

[6] Jin Chaoqi, Liang Xifeng, Zhang Yan, et al. Detection and extraction of tomato fruit bunches based on matlab [J]. Jiangsu Agricultural Sciences. 2018, (8). DOI: 10.15889/j.issn.1002-1302.2018.08.053.

[7] Xiong Juntao, Liu Zhen, Tang Linyue, et al. Research on visual detection technology of green citrus in natural environment [J]. Journal of Agricultural Machinery. 2018, (4). DOI: 10.6041/j.issn.1000-1298.2018.04.005 .

[8] Fu Longsheng, Feng Yali, Elkamil Tola, et al. Multi-cluster kiwi fruit image recognition method in the field based on convolutional neural network [J]. Transactions of the Chinese Society of Agricultural Engineering. 2018, (2). DOI: 10.11975/j.issn.1002- 6819.2018.02.028.

[9] Qiu Ruicheng, Zhang Man, Wei Shuang, et al. Corn stem thickness measurement method based on RGB-D camera [J]. Transactions of the Chinese Society of Agricultural Engineering. 2017, (z1). DOI: 10.11975/j.issn.1002-6819.2017. z1.026.

[10] Li Wenyong, Chen Meixiang, Xu Shupo, et al. Measurement method of immature apple diameter in natural scenes based on watershed and convex hull theory [J]. Transactions of the Chinese Society of Agricultural Engineering. 2014, (23). DOI: 10.3969/j.issn. 1002-6819.2014.23.026.