Image processing: Histogram – Backprojection OpenCV v4.8.0

Previous tutorial: Histogram comparison

Next tutorial: Template matching

Original author	Ana Huamán
Compatibility	OpenCV >= 3.0

Goals

In this tutorial you will learn

• What is backprojection and its uses
• How to calculate backprojection using OpenCV function cv::calcBackProject
• How to mix different channels of an image using the OpenCV function cv::mixChannels

Theory

What is backprojection?

• Backprojection is a method of recording how well the pixels of a given image match the distribution of pixels in a histogram model.
• In simple terms for back projection you need to compute the histogram model of a feature and then use it to find that feature in the image.
• Application example: If you have a flesh color histogram (such as a hue-saturation histogram), then you can use it to find flesh color areas in the image:

How does it work?

• Let’s take skin as an example:
• Let’s say you get a skin histogram (hue-saturation) based on the image below. The histogram beyond that will be our model histogram (which we know represents a sample of skin tones). You apply some masks to capture only the histogram of skin areas:
  

T0

T1

• Now, let’s imagine that you get another hand image (test image) like this: (and its respective histogram):
  

T2

T3

• All we have to do is use our model histogram (which we know represents skin tone) to detect areas of skin in the test image. Specific steps are as follows

1. In each pixel of the test image (i.e. p(i,j)), collect data and find the bin position corresponding to that pixel (i.e. (h[i,j],s[i,j])).
2. Find the model histogram in the corresponding bin – (h[i,j],s[i,j]) – and read the bin value.
3. Store this binary value into a new image (BackProjection). Additionally, you might consider normalizing the model histogram first so you can see the output on the test image.
4. Through the above steps, we can get the BackProjection image of the following test image:

5. In terms of statistics, the value stored in BackProjection represents the probability that a pixel in the test image belongs to a skin region, based on the model histogram we used. For example, in our test images, brighter areas are more likely to be skin areas (as is actually the case), while dark areas are less likely (note that these “dark” areas belong to surfaces with some shadows, which in turn Coming over will affect the test results).

Code

• What does this program do?
  • Load image
  • Convert the original image to HSV format and separate out only the hue channels for the histogram (using the OpenCV function cv::mixChannels)
  • Let the user enter the number of partitions to use when calculating the histogram.
  • Calculate the histogram (and update the histogram when diversity changes) and backprojection of the same image.
  • Displays the backprojection and histogram in a window.

C++

• Downloadable code:
  • Click here for the basic version (explained in this tutorial).
  • For more advanced stuff (defining masks for skin areas using H-S histograms and floodFill), check out the improved demo
  • …or you can check out the classic camshiftdemo in the samples.
• Code Overview

#include "opencv2/imgproc.hpp"
#include "opencv2/imgcodecs.hpp"
#include "opencv2/highgui.hpp"
#include <iostream>
using namespace cv;
using namespace std;
Mat hue;
int bins = 25;
void Hist_and_Backproj(int, void* );
int main( int argc, char* argv[] )
{<!-- -->
 CommandLineParser parser( argc, argv, "{@input |Back_Projection_Theory0.jpg| input image}" );
 samples::addSamplesDataSearchSubDirectory("doc/tutorials/imgproc/histograms/back_projection/images");
 Mat src = imread(samples::findFile(parser.get<String>( "@input" )) );
 if( src.empty() )
 {<!-- -->
 cout << "Could not open or find the image!\
" << endl;
 cout << "Usage: " << argv[0] << " <Input image>" << endl;
 return -1;
 }
 Mat hsv;
 cvtColor(src, hsv, COLOR_BGR2HSV);
 hue.create(hsv.size(), hsv.depth());
 int ch[] = {<!-- --> 0, 0 };
 mixChannels( & amp;hsv, 1, & amp;hue, 1, ch, 1 );
 const char* window_image = "Source image";
 namedWindow(window_image);
 createTrackbar("* Hue bins: ", window_image, & amp;bins, 180, Hist_and_Backproj );
 Hist_and_Backproj(0, 0);
 imshow(window_image, src);
 // Wait until the user exits the program
 waitKey();
 return 0;
}
void Hist_and_Backproj(int, void* )
{<!-- -->
 int histSize = MAX(bins, 2);
 float hue_range[] = {<!-- --> 0, 180 };
 const float* ranges[] = {<!-- --> hue_range };
 Mat hist;
 calcHist( & amp;hue, 1, 0, Mat(), hist, 1, & amp;histSize, ranges, true, false );
 normalize( hist, hist, 0, 255, NORM_MINMAX, -1, Mat() );
 Mat backproj;
 calcBackProject( & amp;hue, 1, 0, hist, backproj, ranges, 1, true );
 imshow( "BackProj", backproj );
 int w = 400, h = 400;
 int bin_w = cvRound( (double) w / histSize );
 Mat histImg = Mat::zeros( h, w, CV_8UC3 );
 for (int i = 0; i < bins; i + + )
 {<!-- -->
 rectangle( histImg, Point( i*bin_w, h ), Point( (i + 1)*bin_w, h - cvRound( hist.at<float>(i)*h/255.0 ) ),
 Scalar( 0, 0, 255 ), FILLED );
 }
 imshow( "Histogram", histImg );
}

Java

• Downloadable code:
  • Click here for the basic version (explained in this tutorial).
  • For more advanced stuff (defining masks for skin areas using H-S histograms and floodFill), check out the improved demo
  • …or you can check out the classic camshiftdemo in the samples.
• Code Overview

import java.awt.BorderLayout;
import java.awt.Container;
import java.awt.Image;
import java.util.Arrays;
import java.util.List;
import javax.swing.BoxLayout;
import javax.swing.ImageIcon;
import javax.swing.JFrame;
import javax.swing.JLabel;
import javax.swing.JPanel;
import javax.swing.JSlider;
import javax.swing.event.ChangeEvent;
import javax.swing.event.ChangeListener;
import org.opencv.core.Core;
import org.opencv.core.CvType;
import org.opencv.core.Mat;
import org.opencv.core.MatOfFloat;
import org.opencv.core.MatOfInt;
import org.opencv.core.Point;
import org.opencv.core.Scalar;
import org.opencv.highgui.HighGui;
import org.opencv.imgcodecs.Imgcodecs;
import org.opencv.imgproc.Imgproc;
class CalcBackProject1 {<!-- -->
 private Mat hue;
 private Mat histImg = new Mat();
 private JFrame frame;
 private JLabel imgLabel;
 private JLabel backprojLabel;
 private JLabel histImgLabel;
 private static final int MAX_SLIDER = 180;
 private int bins = 25;
 public CalcBackProject1(String[] args) {<!-- -->
 if (args.length != 1) {<!-- -->
 System.err.println("You must supply one argument that corresponds to the path to the image.");
 System.exit(0);
 }
 Mat src = Imgcodecs.imread(args[0]);
 if (src.empty()) {<!-- -->
 System.err.println("Empty image: " + args[0]);
 System.exit(0);
 }
 Mat hsv = new Mat();
 Imgproc.cvtColor(src, hsv, Imgproc.COLOR_BGR2HSV);
 hue = new Mat(hsv.size(), hsv.depth());
 Core.mixChannels(Arrays.asList(hsv), Arrays.asList(hue), new MatOfInt(0, 0));
 // Create and set up the window.
 frame = new JFrame("Back Projection 1 demo");
 frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
 // Set up the content pane.
 Image img = HighGui.toBufferedImage(src);
 addComponentsToPane(frame.getContentPane(), img);
 // Use the content pane's default border layout. No need
 // setLayout(new BorderLayout());
 //Show the window.
 frame.pack();
 frame.setVisible(true);
 }
 private void addComponentsToPane(Container pane, Image img) {<!-- -->
 if (!(pane.getLayout() instanceof BorderLayout)) {<!-- -->
 pane.add(new JLabel("Container doesn't use BorderLayout!"));
 return;
 }
 JPanel sliderPanel = new JPanel();
 sliderPanel.setLayout(new BoxLayout(sliderPanel, BoxLayout.PAGE_AXIS));
 sliderPanel.add(new JLabel("* Hue bins: "));
 JSlider slider = new JSlider(0, MAX_SLIDER, bins);
 slider.setMajorTickSpacing(25);
 slider.setMinorTickSpacing(5);
 slider.setPaintTicks(true);
 slider.setPaintLabels(true);
 slider.addChangeListener(new ChangeListener() {<!-- -->
 @Override
 public void stateChanged(ChangeEvent e) {<!-- -->
 JSlider source = (JSlider) e.getSource();
 bins = source.getValue();
 update();
 }
 });
 sliderPanel.add(slider);
 pane.add(sliderPanel, BorderLayout.PAGE_START);
 JPanel imgPanel = new JPanel();
 imgLabel = new JLabel(new ImageIcon(img));
 imgPanel.add(imgLabel);
 backprojLabel = new JLabel();
 imgPanel.add(backprojLabel);
 histImgLabel = new JLabel();
 imgPanel.add(histImgLabel);
 pane.add(imgPanel, BorderLayout.CENTER);
 }
 private void update() {<!-- -->
 int histSize = Math.max(bins, 2);
 float[] hueRange = {<!-- -->0, 180};
 Mat hist = new Mat();
 List<Mat> hueList = Arrays.asList(hue);
 Imgproc.calcHist(hueList, new MatOfInt(0), new Mat(), hist, new MatOfInt(histSize), new MatOfFloat(hueRange), false);
 Core.normalize(hist, hist, 0, 255, Core.NORM_MINMAX);
 Mat backproj = new Mat();
 Imgproc.calcBackProject(hueList, new MatOfInt(0), hist, backproj, new MatOfFloat(hueRange), 1);
 Image backprojImg = HighGui.toBufferedImage(backproj);
 backprojLabel.setIcon(new ImageIcon(backprojImg));
 int w = 400, h = 400;
 int binW = (int) Math.round((double) w / histSize);
 histImg = Mat.zeros(h, w, CvType.CV_8UC3);
 float[] histData = new float[(int) (hist.total() * hist.channels())];
 hist.get(0, 0, histData);
 for (int i = 0; i < bins; i + + ) {<!-- -->
 Imgproc.rectangle(histImg, new Point(i * binW, h),
 new Point((i + 1) * binW, h - Math.round(histData[i] * h / 255.0)), new Scalar(0, 0, 255), Imgproc.FILLED);
 }
 Image histImage = HighGui.toBufferedImage(histImg);
 histImgLabel.setIcon(new ImageIcon(histImage));
 frame.repaint();
 frame.pack();
 }
}
public class CalcBackProjectDemo1 {<!-- -->
 public static void main(String[] args) {<!-- -->
 //Load the local OpenCV library
 System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
 // Arrange tasks for the event dispatch thread:
 // Create and display the graphical user interface for this application.
 javax.swing.SwingUtilities.invokeLater(new Runnable() {<!-- -->
 @Override
 public void run() {<!-- -->
 new CalcBackProject1(args);
 }
 });
 }
}

Python

• Downloadable code:
  • Click here for the basic version (explained in this tutorial).
  • For more advanced stuff (defining masks for skin areas using H-S histograms and floodFill), check out the improved demo
  • …or you can check out the classic camshiftdemo in the samples.
• Code Overview

from __future__ import print_function
from __future__ import division
import cv2 as cv
import numpy as np
import argparse
def Hist_and_Backproj(val):
 
 bins = val
 histSize = max(bins, 2)
 ranges = [0, 180] # hue_range
 
 
 hist = cv.calcHist([hue], [0], None, [histSize], ranges, accumulate=False)
 cv.normalize(hist, hist, alpha=0, beta=255, norm_type=cv.NORM_MINMAX)
 
 
 backproj = cv.calcBackProject([hue], [0], hist, ranges, scale=1)
 
 
 cv.imshow('BackProj', backproj)
 
 
 w = 400
 h = 400
 bin_w = int(round(w / histSize))
 histImg = np.zeros((h, w, 3), dtype=np.uint8)
 for i in range(bins):
 cv.rectangle(histImg, (i*bin_w, h), ( (i + 1)*bin_w, h - int(np.round( hist[i]*h/255.0 )) ), (0, 0, 255) , cv.FILLED)
 cv.imshow('Histogram', histImg)
 
parser = argparse.ArgumentParser(description='Code for Back Projection tutorial.')
parser.add_argument('--input', help='Path to input image.', default='home.jpg')
args = parser.parse_args()
src = cv.imread(cv.samples.findFile(args.input))
if src is None:
 print('Could not open or find the image:', args.input)
 exit(0)
hsv = cv.cvtColor(src, cv.COLOR_BGR2HSV)
ch = (0, 0)
hue = np.empty(hsv.shape, hsv.dtype)
cv.mixChannels([hsv], [hue], ch)
window_image = 'Source image'
cv.namedWindow(window_image)
bins = 25
cv.createTrackbar('* Hue bins: ', window_image, bins, 180, Hist_and_Backproj )
Hist_and_Backproj(bins)
cv.imshow(window_image, src)
cv.waitKey()

Description

• Read input image
  C++

 CommandLineParser parser( argc, argv, "{@input |Back_Projection_Theory0.jpg| input image}" );
 samples::addSamplesDataSearchSubDirectory("doc/tutorials/imgproc/histograms/back_projection/images");
 Mat src = imread(samples::findFile(parser.get<String>( "@input" )) );
 if( src.empty() )
 {<!-- -->
 cout << "Could not open or find the image!\
" << endl;
 cout << "Usage: " << argv[0] << " <Input image>" << endl;
 return -1;
 }

Java

 if (args.length != 1) {<!-- -->
 System.err.println("You must supply one argument that corresponds to the path to the image.");
 System.exit(0);
 }
 Mat src = Imgcodecs.imread(args[0]);
 if (src.empty()) {<!-- -->
 System.err.println("Empty image: " + args[0]);
 System.exit(0);
 }

Python

parser = argparse.ArgumentParser(description='Code for Back Projection tutorial.')
parser.add_argument('--input', help='Path to input image.', default='home.jpg')
args = parser.parse_args()
src = cv.imread(cv.samples.findFile(args.input))
if src is None:
 print('Could not open or find the image:', args.input)
 exit(0)

• Convert it to HSV format:
  C++

 Mat hsv;
 cvtColor(src, hsv, COLOR_BGR2HSV);

Java

 Mat hsv = new Mat();
 Imgproc.cvtColor(src, hsv, Imgproc.COLOR_BGR2HSV);

Python

hsv = cv.cvtColor(src, cv.COLOR_BGR2HSV)

• In this tutorial we will plot a 1D histogram using only tonal values (if you want to use a more standard H-S histogram, check out the more complex code in the link above which will produce better results):
  C++

 hue.create(hsv.size(), hsv.depth());
 int ch[] = {<!-- --> 0, 0 };
 mixChannels( & amp;hsv, 1, & amp;hue, 1, ch, 1 );

Java

 hue = new Mat(hsv.size(), hsv.depth());
 Core.mixChannels(Arrays.asList(hsv), Arrays.asList(hue), new MatOfInt(0, 0));

Python

ch = (0, 0)
hue = np.empty(hsv.shape, hsv.dtype)
cv.mixChannels([hsv], [hue], ch)

• As you can see, we use the function cv::mixChannels to get only channel 0 (hue) from the hsv image. It gets the following parameters:
  • & amp;hsv: source array to copy channels from
  • 1: The number of source arrays
  • & amp;hue: Copy the target array of the channel
  • 1: The number of target arrays
  • ch[] = {0,0}: An array of index pairs indicating how to copy the channel. In this example, &hsv ‘s hue (0) channel is copied to &hue ‘s 0-channel (1-channel).
  • 1: Number of index pairs
• Create a Trackbar for users to enter score values. Any change on the Trackbar means a call to the Hist_and_Backproj callback function.
  C++

 const char* window_image = "Source image";
 namedWindow(window_image);
 createTrackbar("* Hue bins: ", window_image, & amp;bins, 180, Hist_and_Backproj );
 Hist_and_Backproj(0, 0);

Java

 JPanel sliderPanel = new JPanel();
 sliderPanel.setLayout(new BoxLayout(sliderPanel, BoxLayout.PAGE_AXIS));
 sliderPanel.add(new JLabel("* Hue bins: "));
 JSlider slider = new JSlider(0, MAX_SLIDER, bins);
 slider.setMajorTickSpacing(25);
 slider.setMinorTickSpacing(5);
 slider.setPaintTicks(true);
 slider.setPaintLabels(true);
 slider.addChangeListener(new ChangeListener() {<!-- -->
 @Override
 public void stateChanged(ChangeEvent e) {<!-- -->
 JSlider source = (JSlider) e.getSource();
 bins = source.getValue();
 update();
 }
 });
 sliderPanel.add(slider);
 pane.add(sliderPanel, BorderLayout.PAGE_START);

Python

window_image = 'Source image'
cv.namedWindow(window_image)
bins = 25
cv.createTrackbar('* Hue bins: ', window_image, bins, 180, Hist_and_Backproj )
Hist_and_Backproj(bins)

• Display the image and wait for the user to exit the program:
  C++

 imshow( window_image, src );
 // Wait until user exits the program
 waitKey();

Java

 // Use the content pane's default BorderLayout. No need for
 // setLayout(new BorderLayout());
 // Display the window.
 frame.pack();
 frame.setVisible(true);

Python

cv.imshow(window_image, src)
cv.waitKey()

• Hist_and_Backproj Function: Initializes the parameters required by cv::calcHist. The number of partitions comes from Trackbar:
  C++

 int histSize = MAX(bins, 2);
 float hue_range[] = {<!-- --> 0, 180 };
 const float* ranges[] = {<!-- --> hue_range };

Java

 int histSize = Math.max(bins, 2);
 float[] hueRange = {<!-- -->0, 180};

Python

 bins = val
 histSize = max(bins, 2)
 ranges = [0, 180] # hue_range

• Calculate the histogram and normalize it to the [0,255] range
  C++

 calcHist( & amp;hue, 1, 0, Mat(), hist, 1, & amp;histSize, ranges, true, false );
 normalize( hist, hist, 0, 255, NORM_MINMAX, -1, Mat() );

Java

 Mat hist = new Mat();
 List<Mat> hueList = Arrays.asList(hue);
 Imgproc.calcHist(hueList, new MatOfInt(0), new Mat(), hist, new MatOfInt(histSize), new MatOfFloat(hueRange), false);
 Core.normalize(hist, hist, 0, 255, Core.NORM_MINMAX);

Python

 hist = cv.calcHist([hue], [0], None, [histSize], ranges, accumulate=False)
 cv.normalize(hist, hist, alpha=0, beta=255, norm_type=cv.NORM_MINMAX)

• Call the function cv::calcBackProject to obtain the back projection result of the same image
  C++

 Mat backproj;
 calcBackProject( & amp;hue, 1, 0, hist, backproj, ranges, 1, true );

Java

 Mat backproj = new Mat();
 Imgproc.calcBackProject(hueList, new MatOfInt(0), hist, backproj, new MatOfFloat(hueRange), 1);

Python

 backproj = cv.calcBackProject([hue], [0], hist, ranges, scale=1)

• All parameters are known (the same ones used when calculating the histogram), except that we add the backproj matrix, which will store the backprojection result of the source image ( & amp;hue)
• Show backproj:
  C++

 imshow( "BackProj", backproj );

Java

 Image backprojImg = HighGui.toBufferedImage(backproj);
 backprojLabel.setIcon(new ImageIcon(backprojImg));

Python

 cv.imshow('BackProj', backproj)

• Plot a one-dimensional hue histogram of an image:
  C++

 int w = 400, h = 400;
 int bin_w = cvRound( (double) w / histSize );
 Mat histImg = Mat::zeros( h, w, CV_8UC3 );
 for (int i = 0; i < bins; i + + )
 {<!-- -->
 rectangle( histImg, Point( i*bin_w, h ), Point( (i + 1)*bin_w, h - cvRound( hist.at<float>(i)*h/255.0 ) ),
 Scalar( 0, 0, 255 ), FILLED );
 }
 imshow( "Histogram", histImg );

Java

 int w = 400, h = 400;
 int binW = (int) Math.round((double) w / histSize);
 histImg = Mat.zeros(h, w, CvType.CV_8UC3);
 float[] histData = new float[(int) (hist.total() * hist.channels())];
 hist.get(0, 0, histData);
 for (int i = 0; i < bins; i + + ) {<!-- -->
 Imgproc.rectangle(histImg, new Point(i * binW, h),
 new Point((i + 1) * binW, h - Math.round(histData[i] * h / 255.0)), new Scalar(0, 0, 255), Imgproc.FILLED);
 }
 Image histImage = HighGui.toBufferedImage(histImg);
 histImgLabel.setIcon(new ImageIcon(histImage));

Python

 w = 400
 h = 400
 bin_w = int(round(w / histSize))
 histImg = np.zeros((h, w, 3), dtype=np.uint8)
 for i in range(bins):
 cv.rectangle(histImg, (i*bin_w, h), ( (i + 1)*bin_w, h - int(np.round( hist[i]*h/255.0 )) ), (0, 0, 255) , cv.FILLED)
 cv.imshow('Histogram', histImg)

Results

Below is the output using a sample image (guess what? the other hand). You can adjust the binary value as you like and see how it affects the result:

R0

R1

R2