In-depth analysis of NV21 image format and code practice-RGB conversion to NV21 and picture frame

1.NV21 format image analysis

The NV21 image format belongs to the YUV420SP format in the YUV color space
Every four Y components share a set of U components and V components. Y is sorted continuously, and U and V are sorted cross-wise.

Key points summary

• uv staggered mode
• 4Y shares a set of uvs (2 pcs)
• Size: UV= half of Y

The arrangement is as follows

Y Y ? Y Y ? Y Y ? Y Y
Y Y ? Y Y ? Y Y ? Y Y

Y Y ? Y Y ? Y Y ? Y Y
Y Y ? Y Y ? Y Y ? Y Y

V U ?V U ?V U? V U

V U ?V U ?V U? V U

2. Convert RGB images to NV21-pixel by pixel

Basic formula

• yuv –> rgb

  R = (298*Y + 411 * V - 57344)>>8
  G = (298*Y - 101* U - 211* V + 34739)>>8
  B = (298*Y + 519* U- 71117)>>8
  

• rgb –> yuv

  Y= ( 66*R + 129*G + 25*B)>>8 + 16
  U= (-38*R - 74*G + 112*B)>>8 + 128
  V= (112*R - 94*G - 18*B)>>8 + 128
  

c++ code

#include <iostream>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <vector>
#include <fstream>
#include <string>
 
void RGB2NV21()
{<!-- -->
const char *filename = "yuv.yuv";
cv::Mat Img = cv::imread("RGB.jpg");
FILE *fp = fopen(filename,"wb");
\t
if (Img.empty())
{<!-- -->
std::cout << "empty!check your image";
return;
}
int cols = Img.cols;
int rows = Img.rows;
 
int Yindex = 0;
int UVindex = rows * cols;
 
unsigned char* yuvbuff = new unsigned char[1.5 * rows * cols];
 
cv::Mat NV21(rows + rows/2, cols, CV_8UC1);
cv::Mat OpencvYUV;
cv::Mat OpencvImg;
cv::cvtColor(Img, OpencvYUV, CV_BGR2YUV_YV12);
\t
int UVRow{<!-- --> 0 };
for (int i=0;i<rows;i + + )
{<!-- -->
for (int j=0;j<cols;j + + )
{<!-- -->
uchar* YPointer = NV21.ptr<uchar>(i);
 
int B = Img.at<cv::Vec3b>(i, j)[0];
int G = Img.at<cv::Vec3b>(i, j)[1];
int R = Img.at<cv::Vec3b>(i, j)[2];
 
//Calculate the value of Y
int Y = (77 * R + 150 * G + 29 * B) >> 8;
YPointer[j] = Y;
yuvbuff[Yindex + + ] = (Y < 0) ? 0 : ((Y > 255) ? 255 : Y);
uchar* UVPointer = NV21.ptr<uchar>(rows + i/2);
//Calculate the values of U and V and perform 2x2 sampling
if (i%2==0 & amp; & amp;(j)%2==0)
{<!-- -->
int U = ((-44 * R - 87 * G + 131 * B) >> 8) + 128;
int V = ((131 * R - 110 * G - 21 * B) >> 8) + 128;
UVPointer[j] = V;
UVPointer[j + 1] = U;
yuvbuff[UVindex + + ] = (V < 0) ? 0 : ((V > 255) ? 255 : V);
yuvbuff[UVindex + + ] = (U < 0) ? 0 : ((U > 255) ? 255 : U);
}
}
}
for (int i=0;i< 1.5 * rows * cols;i + + )
{<!-- -->
fwrite( & amp;yuvbuff[i], 1, 1, fp);
}
fclose(fp);
std::cout << "write to file ok!" << std::endl;
std::cout << "srcImg: " << "rows:" << Img.rows << "cols:" << Img.cols << std::endl;
std::cout << "NV21: " << "rows:" << NV21.rows << "cols:" << NV21.cols << std::endl;
std::cout << "opencv_YUV: " << "rows:" << OpencvYUV.rows << "cols:" << OpencvYUV.cols << std::endl;
 
cv::imshow("src", Img);//original image
cv::imshow("YUV", NV21);//Converted image
cv::imshow("opencv_YUV", OpencvYUV); //Opencv converted picture
cv::imwrite("NV21.jpg", NV21);
cv::waitKey(30000);
}
 
int main()
{<!-- -->
RGB2NV21();
return 0;
}

3.NV21 image pixel by pixel frame

Result display

Overall process

Save original image to 1D array

2 horizontal lines

2 vertical lines

The main difficulty is to find the iterative formula of the two uv starting coordinates

Horizontal line drawing

y: The outer loop updates the next row, and the inner loop changes the value of one row

uv: The outer loop updates the next row, and the inner loop changes the value of one row.

Because of the staggered mode, the uv inner loop update needs to be separated by 2

Because 4 y corresponds to a set of uv, the number of uv outer loops is less than half the number of y outer loops.

Y component starting position update formula

//Calculate the starting position of the Y component in the nv21Data array: Y data + draw the abscissa position
yStartIndex = (y + i_r) * imageWidth + x;

yStartIndex: the starting position of the Y component in the nv21Data array
i_r: row update value
imageWidth: image width
x: The abscissa position of the drawing starting point

UV component starting position update formula

//Calculate the starting position of the UV component in the nv21Data array: Y data + (UV data row/2*data width) + draw the abscissa (column) position
uvStartIndex = imageWidth * imageHeight + ((y + i_r)/ 2 * imageWidth) + x;

uvStartIndex: the starting position of the uv component in the nv21Data array
i_r: row update value
imageWidth: image width
imageHeight: image height
x: The abscissa position of the drawing starting point

Draw vertical lines

y: The outer loop updates the next column, and the inner loop changes the value of one column – interval mageWidth rendering

uv: The outer loop updates the next column, and the inner loop changes the value of one column.

Because of the staggered mode, the uv outer loop updates + 2

Because 4 y corresponds to a set of uv, the number of loops within uv is less than half the number of loops within y.

Y component starting position update formula

//Calculate the starting position of the Y component in the nv21Data array: Y data + draw the abscissa position
yStartIndex = (y) * imageWidth + x + i_c;

yStartIndex: the starting position of the Y component in the nv21Data array
i_c: Column update value
imageWidth: image width
x: The abscissa position of the drawing starting point

UV component starting position update formula

//Calculate the starting position of the UV component in the nv21Data array: Y data + (UV data row/2*data width) + drawing column position
uvStartIndex = imageWidth * imageHeight + (y/2 * imageWidth) + x + (i_c);

uvStartIndex: the starting position of the uv component in the nv21Data array
i_c: Column update value
imageWidth: image width
imageHeight: image height
x: The abscissa position of the drawing starting point

Complete code

#include <stdio.h>
#include <stdlib.h>
#include <string.h>



/**
 * Draw horizontal lines
 * y: The outer loop updates the next row, and the inner loop changes the value of one row
 * uv: The outer loop updates the next row, and the inner loop changes the value of one row
 * Because of the staggered mode, the uv inner loop update must be separated by 2
 * Because 4 y corresponds to a group of uv, the number of uv outer loops is less than half the number of y outer loops
 *
 * Parameters:
 * unsigned char *nv21Data, int imageWidth, int imageHeight: image data, width and height
 * int x: the abscissa coordinate of the starting point of the line
 * int y: vertical coordinate of the starting point of the line
 * int line_len: length of line
*/
void draw_line_Horizontal(unsigned char *nv21Data,int imageWidth, int imageHeight,int x,int y,int line_len){<!-- -->
    //Parameter judgment
    if(nv21Data==NULL || x>imageWidth || y>imageHeight||line_len>imageWidth){<!-- -->
        return;
    }

    int line_width = 3;//Set the width of the drawn line to 3
    int y_width = line_width;//Change the width of Y
    // 4 y share a group of vu, and /2 + 1 is required when rendering
    int uv_width = y_width/2 + 1; //Set and change the width of UV to Y/2 + 1

    int yStartIndex, uvStartIndex;//Define the starting position

    //Set Y component
    for(int i_r=0;i_r<y_width;i_r + + ){<!-- -->//i_r represents the row update value
        // Calculate the starting position of the Y component in the nv21Data array: Y data + draw the abscissa position
        yStartIndex = (y + i_r) * imageWidth + x;
        // start set
        for (int i = 0; i < line_len; i + + ) {<!-- -->
            //Set the Y component to blue
            nv21Data[yStartIndex + i] = 60;
        }

    }

    //Set UV component
    for(int i_r=0;i_r<uv_width;i_r + + ){<!-- -->//Update row position
        // Calculate the starting position of the UV component in the nv21Data array: Y data + (UV data row/2*data width) + draw the abscissa (column) position
        uvStartIndex = imageWidth * imageHeight + ((y + i_r)/ 2 * imageWidth) + x;
        // start set
        for (int i = 0; i < line_len; i + =2) {<!-- -->
            //Set the UV component to blue: because the UVs are staggered, + 2
            nv21Data[uvStartIndex + i] = 100;
            nv21Data[uvStartIndex + i + 1] = 212;
        }
    }

}

/**
 * Draw vertical lines
 * Render at an interval of imageWidth
 * y: The outer loop updates the next column, and the inner loop changes the value of one column
 * uv: The outer loop updates the next column, and the inner loop changes the value of one column.
 * Because of the staggered mode, uv outer loop update + 2
 * Because 4 y corresponds to a group of uv, the number of loops within uv is less than half the number of loops within y
 *
 * Parameters:
 * unsigned char *nv21Data, int imageWidth, int imageHeight: image data, width and height
 * int x: the abscissa coordinate of the starting point of the line
 * int y: vertical coordinate of the starting point of the line
 * int line_len: length of line
*/
void draw_line_Vertical(unsigned char *nv21Data,int imageWidth, int imageHeight,int x,int y,int line_len){<!-- -->
    //Parameter judgment
    if(nv21Data==NULL || x>imageWidth || y>imageHeight||line_len>imageWidth){<!-- -->
        return;
    }
    int line_width = 3;//Set the width of the drawn line to 3
    int y_width = line_width;//Change the width of Y
    // 4 y share a group of vu, and /2 + 1 is required when rendering
    int uv_width = y_width/2 + 1; //Set and change the width of UV to Y

    int yStartIndex, uvStartIndex;//Define the starting position

    //Set Y component
    for(int i_c=0;i_c<y_width;i_c + + ){<!-- -->
        // Calculate the starting position of the Y component in the nv21Data array: Y data + draw the abscissa position
        yStartIndex = (y + i_c) * imageWidth + x;
        // start set
        for (int i = 0; i < line_len; i + + ) {<!-- -->
            //Set the Y component to blue
            int index_y = yStartIndex + imageWidth*i;
            nv21Data[index_y] = 65;
        }

    }

    //Set UV component
    for(int i_c=0;i_c<uv_width;i_c + =2){<!-- -->//When updating in the outer loop, because the staggered mode = each update column position + 2
        // Calculate the starting position of the UV component in the nv21Data array: Y data + (UV data row/2*data width) + drawing column position
        uvStartIndex = imageWidth * imageHeight + (y/2 * imageWidth) + x + (i_c);
        // Start set:
        // Because the vertical direction uv is half of y, so line_len/2
        // And because the vertical inner loop has no interleaving mode effect, there is no need to skip 2
        for (int i = 0; i < line_len/2; i + + ) {<!-- -->
            int index_u = uvStartIndex + imageWidth*i;//Next row index: interval imageWidth*i
            int index_v = uvStartIndex + imageWidth*i + 1;
            nv21Data[index_u] = 100;
            nv21Data[index_v] = 212;
        }
    }
}


void drawRectOnNv21Image(const char *pImagePath, int imageWidth, int imageHeight, int left, int top, int right, int bottom) {<!-- -->
    /*
        Parameter detection
    */

    if(pImagePath==NULL || imageHeight==0 || imageWidth==0){<!-- -->
        printf("Error: Failed parameter\
");
        return;
    }

    /*
        Program running body
    */
    FILE *file = fopen(pImagePath, "rb");
    if (file == NULL) {<!-- -->
        printf("Error: Failed fopen pImagePath\
");
        return;
    }

    // Calculate the total size of NV21 images
    int imageSize = imageWidth * imageHeight * 3 / 2;
    fseek(file, 0, SEEK_END);
    int file_size = ftell(file);
    fseek(file, 0, SEEK_SET);

    // Determine whether the file is in NV21 format: file_size!=imageSize
    if(file_size!=imageSize){<!-- -->
        printf("Error: Failed imageSize!=1.5 * imageWidth * imageHeight \
");
        fclose(file);
        return;
    }
    // Apply for space to store image yuv data
    unsigned char *nv21Data = (unsigned char *)malloc(imageSize);
    if (nv21Data == NULL) {<!-- -->
        printf("Error: Failed malloc\
");
        fclose(file);
        return;
    }

    //Read NV21 image data
    fread(nv21Data, sizeof(unsigned char), imageSize, file);
    fclose(file);

    //Draw a rectangular box
    int rectWidth = right - left;
    int rectHeight = bottom - top;

    // top border
    draw_line_Horizontal(nv21Data,imageWidth,imageHeight,left,top,rectWidth);

    // bottom border
    draw_line_Horizontal(nv21Data,imageWidth,imageHeight,left,bottom,rectWidth);

    // left border
    draw_line_Vertical(nv21Data,imageWidth,imageHeight,left,top,rectHeight);

    // right border
    draw_line_Vertical(nv21Data,imageWidth,imageHeight,right,top,rectHeight);

    //Save the result as a new NV21 image
    char output_image_path[] = "output_nv21_image.nv21";
    FILE *outputFile = fopen(output_image_path, "wb");
    if (outputFile == NULL) {<!-- -->
        printf("Error: Failed fopen output_image_path\
");
        free(nv21Data);
        return;
    }
    fwrite(nv21Data, sizeof(unsigned char), imageSize, outputFile);
    fclose(outputFile);

    free(nv21Data);
    printf("Program ok!\
");
    printf("Output image save path:%s\
",output_image_path);
}

int main() {<!-- -->
    const char *pImagePath = "input_nv21_iamge.nv21"; // NV21 image file path
    int imageWidth = 640; // Image width
    int imageHeight = 480; // Image height
    int left = 200; //x coordinate of the upper left corner of the rectangular box
    int top = 170; // Y coordinate of the upper left corner of the rectangular box
    int right = 430; // x coordinate of the lower right corner of the rectangular box
    int bottom = 380; // y coordinate of the lower right corner of the rectangular box

    drawRectOnNv21Image(pImagePath, imageWidth, imageHeight, left, top, right, bottom);
    return 0;
}