Understand image formats RAW, RGB, YUV, Packed/Unpacked, Bayer, MIPI, Planar, Semi-Planar, Interleaved in one article

Table of Contents

1. General attributes

1.Packed/Unpacked

2. Compressed/uncompressed

2. RAW

1. Bayer format

2. Classification

3. MIPI RAW

3. RGB

Classification

4. YUV

1. YUV to RGB conversion

2. Classification

3. Memory computing

5. Compression format

Some people miss it and will never find it again in their lifetime.

This article analyzes in detail the classification and memory distribution of various image formats (RAW, RGB, YUV). An article for you to understand.

1. Common attributes

1. Packed/Unpacked

A pixel occupies n bits. If n is not a multiple of 8, then there are concepts of Packed and Unpacked. Imagine that one pixel in a certain format is 10 bits (such as Raw10). If it is Unpacked, then the actual pixel will occupy 2Byte=16bit, of which only the first 10bits have image data, and the last 6bits occupy space (usually 0). If it is Packed, then there is no placeholder.

2. Compressed/uncompressed

In order to facilitate the storage and transmission of large-size images, compression methods are commonly used.

For ISPs, common compression algorithms include FLC (Fixed Length Compression), AFBC (Arm Frame Buffer Compression), JPEG, etc.

The memory distribution mentioned below refers to the uncompressed format, because the compressed memory distribution is determined by the algorithm.

二、RAW

RAW means “unprocessed”.

RAW images are images directly collected by sensors such as CCD/CMOS. They are the original records of the levels when optical signals are converted into electrical signals.

1. Bayer format

RAW is generally arranged in Bayer format (a single pixel is R/G/B light intensity information). Since the human eye is more sensitive to green, the sampling of green light is increased. There are four main sorts:

RGGB
BGGR
GRBG
GBRG

For example, a sensor with a pixel size of 8*8 and a RAW image in BGGR format:

Or you may ask, if a pixel has only one color R/G/B, then the image we see, for example, the first pixel does not necessarily only have B. Because the other components of each pixel will be interpolated later! to reduce power consumption.

2. Category

RAW8: 1 pixel 8 bits
RAW10Unpacked: 1 pixel 2 Bytes (16bits, 6bits are useless)
RAW10Packed: 1 pixel 10 bits
RAW12Unpacked/RAW12Packed
RAW14Unpacked/RAW14Packed
RAW16
…

The RAW format is large and often requires compression to be stored in DDR.

3. MIPI RAW

MIPI RAW can be regarded as a kind of Packed RAW, but it must be packed according to the storage rules of MIPI RAW. Since it is Packed, for example, for MIPI RAW10, 5 Bytes (40bits) can store 4 pixels of data (4*10bits), and there are no empty bits.

三、RGB

Each pixel is composed of three primary colors R/G/B. Note the difference from RAW. Each pixel in RAW has only one three primary colors.

四、YUV

YUV uses a brightness component Y and two chrominance components U and V to describe a color. If there is only Y, it is a grayscale image.

1. YUV and RGB conversion

YUV to RGB

R = Y + 1.13983 * V
G = Y - 0.39465 * U - 0.58060 * V
B = Y + 2.03211 * U

RGB to YUV

Y = 0.299 * R + 0.587 * G + 0.114 * B
U = -0.14713 * R - 0.28886 * G + 0.436 * B
V = 0.615 * R - 0.51499 * G - 0.10001 * B

2. Classification

Since the human eye is much more sensitive to Y than to U and V, sometimes multiple Ys can be used to share a set of U and V, which not only saves space but also ensures quality.

Commonly used format categories:

YUV420, 4 Ys share a set of UVs, the number of data is 1.5*w*h
YUV422, 2 Y share a set of UV, the number of data is 2*w*h
YUV444, not shared, 1 Y set of UV, the number of data is 3*w*h

If a pixel is 8bit, then the storage occupation for YUV420 is 1.5*w*h*8 bits=1.5*w*h Bytes.

The memory distribution of YUV can be divided into

Planar, the three components are stored separately
Semi-Planar, the Y component is stored separately, and the UV components are staggered and stored together.
Packed/Interleaved, the three components are interleaved and stored together

Note: Packed here and Packed of the image are two different concepts! Packed here means that all pixels components are interleaved; Packed of an image means that one pixel n bits are arranged without gaps.

I420 (also called YU12, belongs to YUV420 Planar)

The total number of data w*h + 0.25*w*h + 0.25*w*h=1.5*w*h

YYYY
YYYY
UU
VV

Of course, for computer memory, it only stores from a certain addr, first Y, then U, then V.

YV12 (belongs to YUV420 Planar)

The total number of data w*h + 0.25*w*h + 0.25*w*h=1.5*w*h

YYYY
YYYY
VV
UU

NV12 (belongs to YUV420 Semi-Planar)

The total number of data w*h + 0.5*w*h=1.5*w*h

YYYY
YYYY
UVUV

NV21 (belongs to YUV420 Semi-Planar)

The total number of data w*h + 0.5*w*h=1.5*w*h

YYYY
YYYY
VUVU

I422 (belongs to YUV422 Planar)