The image feature vector is a mathematical representation used to describe the content of the image. It can reflect the color, texture, shape and other information of the image. Image feature vectors can be used to do many things, such as image retrieval, classification, recognition, etc.
This article will introduce the extraction of image feature vectors and calculation of similarity, and use C# to implement them.
Before the article begins, let’s take a brief look at OpenCV and OpenCvSharp4. These two libraries are the core of this article.
What is OpenCV
OpenCV (Open Source Computer Vision Library) is a cross-platform computer vision and machine learning software library based on open source distribution. It supports multiple programming languages and contains hundreds of image processing and computer vision algorithms.
What is OpenCvSharp4
OpenCvSharp4 is a cross-platform image processing library developed based on OpenCV, which supports .NET Framework 4.8+ and .NET Core 2.0+. It provides a rich and easy-to-use API that can implement various image processing functions. OpenCvSharp4 only contains the core managed library, so you need to install the native binding package (OpenCvSharp4.runtime.*) corresponding to the operating system.
Image feature vector extraction
There are many methods to extract image feature vectors. This article will use two commonly used algorithms, SIFT and SURF.
SIFT algorithm
The SIFT (Scale Invariant Feature Transform) algorithm is a scale-invariant feature extraction method that can detect stable key points in different scale spaces and generate unique and invariant descriptors. The main advantages of the SIFT algorithm are:
- Scale invariance: The SIFT algorithm uses a Gaussian pyramid to construct images of different scales and performs extreme point detection on each scale, thereby achieving insensitivity to scale changes.
- Rotation invariance: The SIFT algorithm uses the gradient direction histogram to generate the descriptor and performs rotation normalization according to the main direction of the key point, thereby achieving insensitivity to rotation changes.
- Strong discriminability: The SIFT algorithm can generate descriptors with high dimensions and high information content, making each key point unique and distinguishable, improving the reliability of matching.
It is very simple to implement the SIFT algorithm using OpenCvSharp4. You only need to call the SIFT.Create method to create a SIFT object, and then call the DetectAndCompute method to extract feature points and descriptors from the image. Here is the code example:
//Load images
Mat image1 = Cv2.ImRead("image1.jpg", ImreadModes.Grayscale);
Mat image2 = Cv2.ImRead("image2.jpg", ImreadModes.Grayscale);
?
//Create SIFT object
SIFT sift = SIFT.Create();
?
//Extract feature points and descriptors
Mat descriptors1 = new Mat();
Mat descriptors2 = new Mat();
sift.DetectAndCompute(image1, null, out _, descriptors1);
sift.DetectAndCompute(image2, null, out _, descriptors2);
SURF algorithm
SURF (Speeded Up Robust Features) algorithm is a fast and robust feature extraction method, which is improved based on Harris corner detection and Scale Invariant Feature Transform (SIFT) algorithm. The main advantages of the SURF algorithm are:
- Fast: The SURF algorithm uses integral graphs and Haar wavelets to accelerate feature point detection and descriptor generation, which is several times faster than the SIFT algorithm.
- High robustness: The SURF algorithm has good robustness to interference such as rotation, scaling, and brightness changes, and can maintain stable performance in different scenarios.
- High accuracy: The SURF algorithm can extract high-quality feature points and descriptors, improving the matching accuracy.
It is also very simple to implement the SURF algorithm using OpenCvSharp4. You only need to call the SURF.Create method to create a SURF object, and then call the DetectAndCompute method to extract feature points and descriptors from the image. Here is the code example:
//Load images
Mat image1 = Cv2.ImRead("image1.jpg", ImreadModes.Grayscale);
Mat image2 = Cv2.ImRead("image2.jpg", ImreadModes.Grayscale);
?
//Create SURF object
SURF surf = SURF.Create(500); // 500 is the threshold parameter, indicating the minimum response value of the feature point
?
//Extract feature points and descriptors
Mat descriptors1 = new Mat();
Mat descriptors2 = new Mat();
surf.DetectAndCompute(image1, null, out _, descriptors1);
surf.DetectAndCompute(image2, null, out _, descriptors2);
Image similarity calculation
After extracting the feature vector of the image, we can calculate the similarity of the image. There are many methods for calculating image similarity. This article will introduce two commonly used methods: BFMatcher and FlannBasedMatcher. They are both methods based on feature point matching, but there are some differences.
BFMatcher
BFMatcher is a brute force matching method. Its principle is that for each feature point in the first picture, it traverses all the feature points in the second picture and finds the closest one or more feature points as the matching result. . The advantage of BFMatcher is that it is simple and intuitive, but the disadvantage is low efficiency and time complexity of O(n^2), where n is the number of feature points.
It is also very simple to implement BFMatcher using OpenCvSharp4. You only need to call the constructor of the BFMatcher class to create a BFMatcher object, and then call the Match method for matching. Here is the code example:
//Create BFMatcher object BFMatcher bfMatcher = new BFMatcher(NormTypes.L2, false); //NormTypes.L2 means using Euclidean distance as the similarity measure, false means no cross matching ? //Make a match DMatch[] bfMatches = bfMatcher.Match(descriptors1, descriptors2); // bfMatches is an array, each element is a DMatch object, representing a pair of matching results
FlannBasedMatcher
FlannBasedMatcher is an approximate nearest neighbor matching method. Its principle is to use a fast index structure to speed up the search of feature points, thus reducing the time complexity. The advantage of FlannBasedMatcher is high efficiency, but the disadvantage is that the accuracy is slightly lower and some wrong matches may occur.
It is also very simple to implement FlannBasedMatcher using OpenCvSharp4. You only need to call the constructor of the FlannBasedMatcher class to create a FlannBasedMatcher object, and then call the Match method for matching. Here is the code example:
//Create FlannBasedMatcher object FlannBasedMatcher flannMatcher = new FlannBasedMatcher(); ? //Make a match DMatch[] flannMatches = flannMatcher.Match(descriptors1, descriptors2); // flannMatches is an array, each element is a DMatch object, representing a pair of matching results
Similarity Score
There are many ways to calculate the similarity score. Here is a simple method: first calculate the distance of each matching pair. Then average all distances to get a similarity score. The smaller the score, the more similar it is.
We do this calculation for the matching results of both BFMatcher and FlannBasedMatcher.
// Calculate and display the similarity scores of BFMatcher and FlannBasedMatcher. The lower the score, the more similar they are.
Console.WriteLine("The score using BFMatcher is {0}", bfMatches.Average(m => m.Distance));
Console.WriteLine("The score using FlannBasedMatcher is {0}", flannMatches.Average(m => m.Distance));
In this way, image feature vector extraction and similarity calculation are realized. The complete code can be viewed on the official account.
Result comparison
Next we run the program and view the results from four situations.
1. Comparison of two completely different pictures
In this case, we can expect a high similarity score, indicating that the two images have little to no similarity. as the picture shows:
SURF algorithm The score using BFMatcher is 0.77414566 The score using FlannBasedMatcher is 0.77414566 SIFT algorithm The score using BFMatcher is 366.84616 The score using FlannBasedMatcher is 372.25107
2. Comparison of two identical pictures
In this case, we can expect to get a very low similarity score, indicating that the two images are identical. as the picture shows:
SURF algorithm The score using BFMatcher is 0 The score using FlannBasedMatcher is 0 SIFT algorithm The score using BFMatcher is 0 The score using FlannBasedMatcher is 0
3. Compare a certain picture with its partial screenshots
In this case, we can expect to get a moderate similarity score, indicating that the two images partially overlap. as the picture shows:
SURF algorithm The score using BFMatcher is 0.22462595 The score using FlannBasedMatcher is 0.23025486 SIFT algorithm The score using BFMatcher is 105.93032 The score using FlannBasedMatcher is 108.3307
4. Compare two similar pictures
In this case, we can expect a lower similarity score, indicating that the two images have many features in common. For example, we can compare two pictures of the same object taken from different angles. as the picture shows:
SURF algorithm The score using BFMatcher is 0.37855583 The score using FlannBasedMatcher is 0.38878053 SIFT algorithm The score using BFMatcher is 239.1525 The score using FlannBasedMatcher is 248.43388
As can be seen from the above results, both SURF and SIFT algorithms can extract image feature vectors, and there are also differences between BFMatcher and FlannBasedMatcher. Therefore, when selecting an algorithm, you need to make trade-offs based on specific application scenarios and requirements.
If you are interested in this, you can further explore and store image feature vectors in a vector database to achieve more functional requirements. For example, you can use Redis or Elasticsearch, both of which support operations such as adding, deleting, modifying, and querying vector data.
The following are related recommended readings:
1. ChatGPT Embeddings and Redis are powerfully combined to realize text similarity analysis and search.
2. Use Redis to implement vector similarity search: solve the similarity matching problem between text, image and audio
3. C# + Redis Search: How to use Redis to implement high-performance full-text search
Writing is not easy, please indicate the blog post address when reprinting, otherwise reprinting is prohibited! ! !
Thanks for reading, like + share + favorite + follow
The article comes from the WeChat official account of Yuan Huohuo
The knowledge points of the article match the official knowledge files, and you can further learn related knowledge. OpenCV skill tree Home page Overview 23664 people are learning the system