How to interact with the HAL layer when Android SurfaceFlinger does Layer synthesis

Zero, the scope of the issues discussed in this article

The above figure shows how Surface is rendered.
To borrow a paragraph from the official website: “No matter what rendering API the developer uses, everything will be rendered to Surface. Surface represents the producer in the buffer queue, and the buffer queue is usually consumed by SurfaceFlinger. Every file created on the Android platform Each window is backed by a Surface. All visible Surfaces that are rendered are composited to the screen by SurfaceFlinger.”

The scope of this article is the part in the red box in the above picture. How does SurfaceFlinger interact with the HAL layer when doing Layer synthesis?

1. Problem: Dilemma in choosing implementation method for SurfaceFlinger layer synthesis

When processing Layer composition, SurfaceFlinger will interact with OpenGL ES and HWC. For HWC, since the implementation of each OEM is different and the capabilities supported are also different, it is difficult to directly use the API to indicate the number of layers that the hardware device supports for synthesis, whether the layer can perform rotation and blending mode operations, and the layer positioning and hardware Synthesis limitations, etc. This makes it difficult for SurfaceFlinger, as the calling end, to make decisions: Based on the current number of Layers to be synthesized and the amount of data, should I choose to use OpenGL ES? Or HWC?

1.1 From the perspective of OpenGL ES and HWC itself

In the Android system, compared to HWC, OpenGL ES is not very good at Layer synthesis.
BecauseOpenGL ES is primarily designed for game development and graphics rendering, it is not optimized for layer compositing. If you use OpenGL ES for layer synthesis, it will occupy a lot of GPU resources, causing the application to be unable to use the GPU for its own rendering. Therefore, HWC is more suitable for use in scenarios that require efficient layer synthesis.

1.2 HWC-led judgment logic

Since the HWC capabilities provided by each OEM are different, some are strong and some are weak, but if you want to give priority to HWC for Layer synthesis tasks, then the selection strategy of SurfaceFlinger is left to HWC to judge:
SurfaceFlinger decides to use OpenGL ES or HWC to handle Layer synthesis tasks based on the HWC hardware capabilities.

The specific logic is as follows:
SurfaceFlinger provides HWC with a complete list of all Layers, allowing HWC to decide how to handle these Layers based on its hardware capabilities.
HWC will mark the synthesis method for each Layer, whether it is synthesized through GPU or HWC.
SurfaceFlinger is responsible for first compositing all Layers marked for GPU synthesis (using OpenGL ES) into an output Buffer, and then handing this output Buffer to HWC together with other Layers marked for HWC synthesis, allowing HWC to complete the synthesis and display of the remaining Layers.

2. Specific implementation framework for interaction between SurfaceFlinger and HAL layer

2.1 SurfaceFlinger calls OpenGL ES process

SurfaceFlinger and OpenGL ES
Gralloc is Graphics Alloc graphics allocation. The Android system provides a Gralloc module in the HAL layer, which encapsulates all access operations to Framebuffer.
The Gralloc module contains two devices: fb and gralloc: fb is responsible for opening the Framebuffer in the kernel, initializing the configuration, and providing post, setSwapInterval and other operations. It is the HAL layer abstraction of the underlying graphics card; gralloc manages the allocation and release of the frame buffer. The upper layer Applications can only access fb through Gralloc.

In the source code of Android12, the path defined by the HAL layer header file is: source code root directory/hardware/libhardware/include/hardware/*.h

2.2 FrameBuffer

The FrameBuffer mechanism imitates the function of the graphics card and is an abstraction of the graphics card hardware. You can think of FrameBuffer as an image of the video memory. After mapping it to the process address space, you can directly perform read and write operations. It is simply understood as a piece of memory. The user application keeps writing data to this memory, and the display controller keeps reading data from it and displaying it.

2.3 SurfaceFlinger calling HWC process

SurfaceFlinger and HWC

3. Understand the above concepts through HAL definition

3.1 hwcomposer.h

We mentioned above that HWC will mark the synthesis method for each Layer

typedef struct hwc_layer_1 {<!-- -->
int32_t compositionType;
}

In the structure hwc_layer_1, a compositionType variable is defined, indicating the composition type of the current Layer. Optional constants are HWC_BACKGROUND, HWC_FRAMEBUFFER_TARGET, HWC_FRAMEBUFFER, HWC_OVERLAY, HWC_SIDEBAND, HWC_CURSOR_OVERLAY.
Among them: HWC_FRAMEBUFFER is commented like this (excerpt):

 * Set by the HWC implementation during (*prepare)(), this indicates
     * that the layer will be drawn into the framebuffer using OpenGL ES.

In the preparation phase of HWC, the Layer type may be set to HWC_FRAMEBUFFER, which means that this Layer will be drawn into the framebuffer using OpenGL ES.
Look again at HWC_BACKGROUND

 * Always set by the caller before calling (*prepare)(), this value
     * indicates this is a special "background" layer. The only valid field
     * is backgroundColor.
     * The HWC can toggle this value to HWC_FRAMEBUFFER to indicate it CANNOT
     * handle the background color.

This value is always set by the caller before calling (*prepare)(), indicating that this is a special “background” layer. The only valid field is backgroundColor. HWC can switch this value to HWC_FRAMEBUFFER to indicate that it cannot handle the background color (HWC cannot handle it, so it can only use OpenGl ES to handle it). The caller here should be SurfaceFlinger.

Hardware Composer must support events, one of which is VSYNC (the other is Hot Plug to support Plug and Play HDMI)

3.2 fb.h

Let’s extract a few attributes and get a general impression.

typedef struct framebuffer_device_t {<!-- -->
    /* dimensions of the framebuffer in pixels */
    const uint32_t width;
    const uint32_t height;

    /* framebuffer pixel format */
    const int format;

    /* resolution of the framebuffer's display panel in pixel per inch*/
    const float xdpi;
    const float ydpi;

    /* framebuffer's display panel refresh rate in frames per second */
    const float fps;
    ...
}

The properties of the FrameBuffer device are: width and height, pixel format, resolution in dip units, and fps.
Let’s look at another method:

 /*
     * Post <buffer> to the display (display it on the screen)
     * The buffer must have been allocated with the
     * GRALLOC_USAGE_HW_FB usage flag.
     * buffer must be the same width and height as the display and must NOT
     * be locked.
     *
     * The buffer is shown during the next VSYNC.
     *
     * Returns 0 on success or -errno on error.
     */
    int (*post)(struct framebuffer_device_t* dev, buffer_handle_t buffer);

The post function will post the buffer to the display and display it on the screen.
The width of the buffer must be consistent with the width and height of the fb device and cannot be locked.
When the next VSYNC signal arrives, the buffer will be displayed.

3.3 gralloc.h

/**
 * Name of the graphics device to open
 */
#define GRALLOC_HARDWARE_GPU0 "gpu0"

It can be seen that the device operated by gralloc is the GPU.

enum {<!-- -->
    /* buffer will be used as an OpenGL ES texture */
    GRALLOC_USAGE_HW_TEXTURE = 0x00000100U,
    /* buffer will be used as an OpenGL ES render target */
    GRALLOC_USAGE_HW_RENDER = 0x00000200U,
    /* buffer will be used by the 2D hardware blitter */
    GRALLOC_USAGE_HW_2D = 0x00000400U,
    /* buffer will be used by the HWComposer HAL module */
    GRALLOC_USAGE_HW_COMPOSER = 0x00000800U,
    /* buffer will be used with the framebuffer device */
    GRALLOC_USAGE_HW_FB = 0x00001000U,
    /* buffer may be used as a cursor */
    GRALLOC_USAGE_CURSOR = 0x00008000U,
    /* buffer will be used with the HW video encoder */
    GRALLOC_USAGE_HW_VIDEO_ENCODER = 0x00010000U,
    /* buffer will be written by the HW camera pipeline */
    GRALLOC_USAGE_HW_CAMERA_WRITE = 0x00020000U,
    /* buffer will be read by the HW camera pipeline */
    GRALLOC_USAGE_HW_CAMERA_READ = 0x00040000U,
    ...
}

Gralloc graphics allocation literally means graphics allocation, but it actually means opening up a section of memory. It operates on the GPU, which is the video memory.
An enumeration is defined here to mark the categories of allocated buffers and classify them according to their usage.
There are those used as OpenGL ES textures, used by the HWComposer HAL module, used by the hardware video encoder, and written/read by the camera pipeline. Another interesting thing I found is that the GRALLOC_USAGE_CURSOR cursor is actually a separate type of buffer.

Core methods:

typedef struct alloc_device_t {<!-- -->
    struct hw_device_t common;
    /*
     * (*alloc)() Allocates a buffer in graphic memory with the requested
     * parameters and returns a buffer_handle_t and the stride in pixels to
     * allow the implementation to satisfy hardware constraints on the width
     * of a pixmap (eg: it may have to be multiple of 8 pixels).
     * The CALLER TAKES OWNERSHIP of the buffer_handle_t.
     *
     * If format is HAL_PIXEL_FORMAT_YCbCr_420_888, the returned stride must be
     * 0, since the actual strides are available from the android_ycbcr
     * structure.
     *
     * Returns 0 on success or -errno on error.
     */
    
    int (*alloc)(struct alloc_device_t* dev,
            int w, int h, int format, int usage,
            buffer_handle_t* handle, int* stride);

This method is to allocate video memory and pass in the parameters width, height, format, and usage. Return buffer_handle_t, step size.

The above brief browse of the HAL layer header file will help us have a deeper impression of these concepts.

Directory