Work notes-YUV dump and read

Work notes-YUV dump and read

• Work notes-YUV dump and read
• Use dump to generate images yuv2img
• yuv2img code

Work notes-YUV dump and read

The work involves work related to model verification. Here is the joint effect of the three models. When the perception model reads the image, it replaces the input with its own given input to ensure consistency, and then looks at the output.
After knowing the width, height and step size of the model input, there are the following dump_yuv and read_yuv functions. Where bpu_addr and bpu_addr_uv are the addresses of the y component and uv component.

bool ReadYUVImage(const std::string & amp;img_path, int width, int height,
                  uint64_t bpu_addr, uint64_t bpu_addr_uv) {<!-- -->
  std::ifstream input_img_file(img_path, std::ios::binary);
  input_img_file.seekg(0, std::ios::end);
  int len = input_img_file.tellg();
  input_img_file.seekg(0, std::ios::beg);
  std::vector<uint8_t> img_data(width * height * 3 / 2);
  if (len < width * height * 3 / 2) {<!-- -->
    HSLOG_E << "file length is not right " << len << " img_file:" << img_path;
    return false;
  } else {<!-- -->
    input_img_file.read(reinterpret_cast<char *>(img_data.data()),
                        width * height * 3 / 2);
    memcpy(reinterpret_cast<void *>(bpu_addr), img_data.data(), width * height);
    memcpy(reinterpret_cast<void *>(bpu_addr_uv),
           img_data.data() + width * height, width * height / 2);
  }
  input_img_file.close();
  return true;
}

void DumpYUVImage(const std::string & amp;out_file, int width, int height,
                  uint64_t bpu_addr, uint64_t bpu_addr_uv) {<!-- -->
  std::ofstream fout(out_file, std::ios::out | std::ios::binary);
  std::vector<uint8_t> src_input_image(width * height * 3 / 2);
  memcpy(src_input_image.data(), reinterpret_cast<void *>(bpu_addr),
         width * height);
  memcpy(src_input_image.data() + width * height,
         reinterpret_cast<void *>(bpu_addr_uv), width * height / 2);
  fout.write(reinterpret_cast<char *>(src_input_image.data()),
             width * height * 3 / 2);
  fout.close();
}

Use dump to generate images yuv2img

Here you need to understand the arrangement knowledge related to yuv.
There are two major categories of YUV formats: planar and packed.
For the YUV format of planar, the Y of all pixels is stored continuously, followed by the U of all pixels, followed by the V of all pixels.
For the packed YUV format, the Y, U, and V of each pixel are stored consecutively and alternately.

YUV is divided into three components. “Y” represents brightness (Luminance or Luma), which is the gray value; while “U” and “V” represent chrominance (Chrominance or Chroma), which is used to describe the image. Color and saturation, used to specify the color of pixels.

The YUV format of planar is divided into YUV420P and YUV420SP. YUV420P includes I420 and YV12. The difference between the I420 format and the YV12 format lies in the different positions of the U plane and V plane. In the I420 format, the U plane follows the Y plane, and then the V plane (ie: YUV); but in YV12 it is the opposite (ie: YVU).
YUV420SP, Y component plane format, UV packed format, namely NV12. NV12 is similar to NV21, U and V are staggered, the difference is in the UV order.
I420: YYYYYYYY UU VV =>YUV420P
YV12: YYYYYYYY VV UU =>YUV420P
NV12: YYYYYYYY UVUV =>YUV420SP
NV21: YYYYYYYY VUVU =>YUV420SP

Note that when converting, you need to know the order of the uv arrangement of the file. The color difference of the picture after the measured u and v are reversed will change.
Correct result

Wrong result

In the core code, there is an opencv pitfall. When merging, the error Invalid number of channels in input image: ‘VScn::contains(scn)’ ‘scn’ will be reported. The reason is that the shape of Y U V is inconsistent. The solution is to interpolate UV into Same as Y, in merge, the shape becomes (h, w, 3). Then call cv2.COLOR_YUV2BGR to do the conversion.

 # Separate UV channels into U and V
    U = UV[:, ::2]
    V = UV[:, 1::2]

    # Expand the U and V channels to the same size as the Y channel
    U = cv2.resize(U, (width, height), interpolation=cv2.INTER_LINEAR)
    V = cv2.resize(V, (width, height), interpolation=cv2.INTER_LINEAR)

    # Merge Y, U, V channel data
    YUV420p_image = cv2.merge((Y, U, V))
    # Convert to RGB format
    image_rgb = cv2.cvtColor(YUV420p_image, cv2.COLOR_YUV2BGR)

yuv2img code

The following is a picture generated based on the yuv file of the dump. The three functions are grayscale (only y component), 420sp is directly converted to RGB, and 420sp extracts the U and V components, and then interpolates the Y component to form YUV444, and then converts it to RGB. .

Y = np.frombuffer(buffer[:Y_size], dtype=np.uint8).reshape(height, width)
UV = np.frombuffer(buffer[Y_size:], dtype=np.uint8).reshape(height // 2, width)

# Separate UV channels into U and V
U = UV[:, ::2] # Actually selects all rows of the UV array, but only selects the even column elements in each row, that is, column 0, column 2, column 4, etc.
V = UV[:, 1::2]

# Expand the U and V channels to the same size as the Y channel
U = cv2.resize(U, (width, height), interpolation=cv2.INTER_LINEAR)
V = cv2.resize(V, (width, height), interpolation=cv2.INTER_LINEAR)

# Merge Y, U, V channel data
YUV420p_image = cv2.merge((Y, U, V))

import cv2
import numpy as np
import re
import os

def convert_YUV420sp_RGB(yuv_file):
        # Extract width and height from filename
    match = re.search(r'_w_(\d + )_h_(\d + )', yuv_file)
    if match:
        width = int(match.group(1))
        height = int(match.group(2))
    else:
        print("Unable to extract resolution information:", yuv_file)
        return
    print(width, height)
    with open(yuv_file, "rb") as f:
        buffer = f.read()
    image = np.frombuffer(buffer, np.uint8).reshape(height*3//2, width)
    print(image.size, image.shape)
    image_rgb = cv2.cvtColor(image, cv2.COLOR_YUV420SP2RGB)
    
    yuv_file = os.path.basename(yuv_file)
    output_file = yuv_file.replace(".yuv", ".jpg")
    output_file = os.path.join(output_folder_path, f"{<!-- -->output_file}.jpg")
    cv2.imwrite(output_file, image_rgb)
    # print(output_file)

def convert_YUV420sp_YUV444_RGB(yuv_file):
    # Extract width and height from filename
    match = re.search(r'_w_(\d + )_h_(\d + )', yuv_file)
    if match:
        width = int(match.group(1))
        height = int(match.group(2))
    else:
        print("Unable to extract resolution information:", yuv_file)
        return

    # Read YUV420sp image data
    with open(yuv_file, "rb") as f:
        buffer = f.read()

    # Parse Y and UV channel data
    Y_size = width * height
    UV_size = width * height // 2

    Y = np.frombuffer(buffer[:Y_size], dtype=np.uint8).reshape(height, width)
    UV = np.frombuffer(buffer[Y_size:], dtype=np.uint8).reshape(height // 2, width)

    # Separate UV channels into U and V
    U = UV[:, ::2]
    V = UV[:, 1::2]

    # Expand the U and V channels to the same size as the Y channel
    U = cv2.resize(U, (width, height), interpolation=cv2.INTER_LINEAR)
    V = cv2.resize(V, (width, height), interpolation=cv2.INTER_LINEAR)

    # Merge Y, U, V channel data
    YUV420p_image = cv2.merge((Y, U, V))
    # Convert to RGB format
    image_rgb = cv2.cvtColor(YUV420p_image, cv2.COLOR_YUV2BGR)

    yuv_file = os.path.basename(yuv_file)
    output_file = yuv_file.replace(".yuv", ".jpg")
    output_file = os.path.join(output_folder_path, f"{<!-- -->output_file}.jpg")
    cv2.imwrite(output_file, image_rgb)
    # print(output_file)

def convert_YUV420sp_GRAY(yuv_file):
    file_name = os.path.splitext(yuv_file)[0]
    match = re.search(r'_w_(\d + )_h_(\d + )', yuv_file)
    if match:
        width = int(match.group(1))
        height = int(match.group(2))
    else:
        print("Unable to extract resolution information:", yuv_file)
        return

    #Open YUV file and read data
    
    with open(yuv_file, 'rb') as file:
        buffer = file.read()

    Y_size = width * height
    Y_data = buffer[:Y_size]

    # Convert Y data to NumPy array
    Y = np.frombuffer(Y_data, dtype=np.uint8).reshape((height, width))
    gray_image = cv2.cvtColor(Y, cv2.COLOR_GRAY2BGR)

    yuv_file = os.path.basename(yuv_file)
    output_file = yuv_file.replace(".yuv", ".jpg")
    output_file = os.path.join(output_folder_path, f"{<!-- -->output_file}.jpg")
    cv2.imwrite(output_file, gray_image)
    return

#Define input YUV folder path
input_folder_path = "./dump_yuv" # Replace with the path of the folder containing the YUV file

#Define output JPEG folder path
output_folder_path = input_folder_path + "_output" # Folder path to save JPEG files
if not os.path.exists(output_folder_path):
    os.makedirs(output_folder_path)

# Get all YUV files in the input folder
yuv_files = [file for file in os.listdir(input_folder_path) if file.endswith(".yuv")]

for yuv_file in yuv_files:
    print(yuv_file)
    # convert_YUV420sp_YUV444_RGB(input_folder_path + "/" + yuv_file)
    # convert_YUV420sp_RGB(input_folder_path + "/" + yuv_file)
    convert_YUV420sp_GRAY(input_folder_path + "/" + yuv_file)

model_id:0 percepts_[0] rect num:1
model_id:0 rect:388.95 1034.04 520.791 1163.19 7.03816 0
model_id:0 percepts_[5] rect num:8
model_id:0 rect:2164.7 1034.81 2261.26 1110.25 8.7623 5
model_id:0 rect:2464.56 996.751 2519.25 1059.25 7.19807 5
model_id:0 rect:2654.99 593.22 2744.83 696.733 8.5591 5
model_id:0 rect:2914.5 570.069 3007.19 666.134 8.76954 5
model_id:0 rect:3044.18 553.215 3152.04 664.448 8.68193 5
model_id:0 rect:3185.64 543.125 3295.19 652.673 7.15572 5
model_id:0 rect:3750.75 609.548 3836 781.291 8.73659 5
model_id:0 rect:3761.16 792.687 3836 1073.72 9.44854 5
model_id:0 percepts_[11] rect num:3
model_id:0 rect:247.464 1250.77 334.153 1311.08 7.29449 11
model_id:0 rect:1664.7 1409.73 2243.79 1526.45 7.30121 11
model_id:0 rect:3057.62 1149.15 3528.76 1186.85 7.29843 11
model_id:0 percepts_[13] rect num:3
model_id:0 rect:1390.61 1212.43 1441.98 1423.03 8.79426 13
model_id:0 rect:1992.3 1189.52 2025.68 1335.91 7.29169 13
model_id:0 rect:2816.59 1182.99 2865.39 1378.19 7.29921 13

Attach an additional code for the picture frame

import cv2
import re

# Read pictures
image_path = './dump_yuv_v3_output/model_50_frame_1694608049106_w_3840_h_2160.jpg.jpg'
image = cv2.imread(image_path)

#Open the file containing the coordinates of the rectangular box
with open('rect.txt', 'r') as file:
    lines = file.readlines()

# Regular expression pattern, used to match lines containing model_id and percepts
model_pattern = re.compile(r'model_id:\d + percepts_\[(\d + )\] rect num:(\d + )')

# Regular expression pattern, used to match lines containing model_id and rect
rect_pattern = re.compile(r'model_id:\d + rect:(\d + (?:\.\d + )?) (\d + (?:\.\d + )?) (\d + (?:\.\d + )?) (\d + (?:\.\d + )?)')

# Initialize a dictionary to store the mapping of percepts_values and colors
percepts_color_mapping = {<!-- -->
    "0": (0, 0, 255), # red
    "5": (0, 255, 0), # green
    "11": (255, 0, 0), # blue
    "13": (0, 255, 255), # yellow
}


# Iterate through each line in the file
for line in lines:
    # Try to match lines containing rectangular coordinates
    model_match = model_pattern.match(line)
    if model_match:
        percept_index = model_match.group(1)
        num_rects = model_match.group(2)
        print(f"Percept Index: {<!-- -->percept_index}")
        print(f"Number of Rectangles: {<!-- -->num_rects}")

    # Match the corresponding box
    rect_match = rect_pattern.search(line)
    if rect_match:
        x1, y1, x2, y2 = map(float, rect_match.groups())
        
        # Get color
        color = percepts_color_mapping.get(percept_index) # Default is red
        
        # Draw a rectangular box
        x1, y1, x2, y2 = int(x1), int(y1), int(x2), int(y2)
        print(x1, y1, x2, y2)
        cv2.rectangle(image, (x1, y1), (x2, y2), color, 2)

        font = cv2.FONT_HERSHEY_SIMPLEX
        cv2.putText(image, str(percept_index), (x1, y1 - 10), font, 0.5, color, 1, cv2.LINE_AA)

# Save the image containing the drawing box
output_image_path = 'output_image.png'
cv2.imwrite(output_image_path, image)

# Save the image containing the drawing box
output_image_path = 'output_image.png'
cv2.imwrite(output_image_path, image)