Android MediaCodec framework based on codec2

What is the purpose of this series of articles?

Roughly:

  • What basic services are required for decoding?
  • What is the calling process for standard decoding?
  • What is the role of each process?
  • Decoding frame levels?
  • What are the functions of each level?

Refinement:

  • Configuration of decoding parameters?
  • Decode the flow of input packets?
  • Application and management of decoded output frame memory?

Article directory

      • HIDL upstream
      • HIDL downstream
      • HIDL interface
      • Basic codec2 service

First, sort out a path from MediaCodec to the specific decoding Component, and then understand the details inside. This article starts from MediaCodec to understand the various levels of the Android decoding framework. In general, it can be divided into three parts.

  • Above the HIDL layer, this layer mainly provides interfaces to external applications and provides input and output buffer management, process control, etc.
  • Under the HIDL layer, it provides the creation of specific decoding components, the implementation of decoding components, decoding data packets and returning decoded images.
  • The codec service provides a service for creating decoding components. This service is called from the HIDL layer to the HIDL layer.

HIDL upstream

As shown in the figure, the upstream mainly includes the following parts.

  • MediaCodec

MediaCodec will first create ccodec, and subsequent operations are called through the ccodec codecbase (this is for compatibility with ACodec and Codec2). At the same time, the CCodecBufferChannel created by the codec is also obtained through this ccodec.

  1. Create codecbase. This is CCodec. At the MediaCodec level, CCodec is called.
  2. Register the created codec into looper. This looper is set to mediacodec by the application layer.
  3. Register CodecCallback to ccodec, and register BufferCallback to CCodecBufferChannel.
  • CCodec
    1. Create CCodecBufferChannel and CCodecConfig.
    2. Obtain the componentStore through the codec2 service, and create the decoder component through the componentStore. This is mainly done through the codec2client class.
    3. Set the created component to CCodecBufferChannel for subsequent calls.
    4. Call back some error and other information to MediaCodec.

codec2client:

The client that interacts with downstream HIDL mainly calls the interfaces of IComponetStore and IComponet.

  1. See how codec2client is created?
std::shared_ptr<Codec2Client> client = _CreateFromIndex(index);


std::shared_ptr<Codec2Client> Codec2Client::_CreateFromIndex(size_t index) {
    std::string const & name = GetServiceNames()[index];
    LOG(WARNING) << "Creating a Codec2 client to service "" << name << """;
    sp<Base> baseStore = Base::getService(name);
    CHECK(baseStore) << "Codec2 service "" << name << """
                        " inaccessible for unknown reasons.";
    LOG(WARNING) << "Client to Codec2 service "" << name << "" created";
    return std::make_shared<Codec2Client>(baseStore, index);
}
  • GetServiceNames(). Get hal’s name through Manifest

Manifest defines the name of HAL “android.hardware.media.c2”, the hidl transmission method “hwbinder”, the name of interface “IComponentStore”, and the name of instance “default”. GetServiceNames also uses this information to locate the specific HAL.

  • Base::getService(name): Base is the IComponentStore type, which is the service side. Get the service on the service side by name. Then assign it to baseStore.

  • Then use this baseStore to initialize and create codec2client (that is, mBase is baseStore).

    std::make_shared(baseStore, index)

  • Therefore, the interface called by codecclinet will be called to the ComponentStore on the service side.

CCodecBufferChannel: A place where input and output buffers are managed. When there are input and output buffers, they are reported to MediaCodec through callbacks, and then MediaCodec reports to the application.

HIDL downstream

Downstream includes two aspects, one is componentStore and the other is Component

  • componentStore

    The calling relationship takes createComponent as an example. The calling process is as follows

    codec2client—–>(HIDL)comometStore (get the real store)——>C2PlatformComponentStore (or componentstore implemented by the vendor itself) —–> C2SoftAvcDecFactory.

    When codec2clinet in the HIDL upper layer obtains the componentStore service, it will call the following function to return C2PlatformComponentStore. Then call createComponent to call this class.

    In the creation component of this class, the component will be found according to the specific name to call its createComponent, such as avc’s C2SoftAvcDecFactory’s createComponent

c2store.cpp
std::shared_ptr<C2ComponentStore> GetCodec2PlatformComponentStore() {
    static std::mutex mutex;
    static std::weak_ptr<C2ComponentStore> platformStore;
    std::lock_guard<std::mutex> lock(mutex);
    std::shared_ptr<C2ComponentStore> store = platformStore.lock();
    if (store == nullptr) {
        store = std::make_shared<C2PlatformComponentStore>();
        platformStore = store;
    }
    return store;
}

c2_status_t C2PlatformComponentStore::createComponent(
        C2String name, std::shared_ptr<C2Component> *const component) {
    // This method SHALL return within 100ms.
    component->reset();
    std::shared_ptr<ComponentModule> module;
    c2_status_t res = findComponent(name, & amp;module);
    if (res == C2_OK) {
        // TODO: get a unique node ID
        res = module->createComponent(0, component);
    }
    return res;
}

  • component

    The component call is also called to SimpleC2Component through the HIDL interface, and then SimpleC2Component calls the specific avc, hevc, etc. component. SimpleC2Component is the base class for every component.

    Taking the queue interface as an example, the codec2bufferChannel in the upper layer of HIDL will call the queue interface of the specific decoding component to put the data packet to be decoded into the specific component. First, it calls the queue of Codec2Client, which calls the component, and then calls the queue_nb of SimpleC2Component. The queue_nb sends the message. , call the process function of the subclass in the message processing thread.

    c2_status_t Codec2Client::Component::queue(
        std::list<std::unique_ptr<C2Work>>* const items) {
    Return<Status> transStatus = mBase1_0->queue(workBundle);
}

// Methods from ::android::hardware::media::c2::V1_1::IComponent
Return<Status> Component::queue(const WorkBundle & amp; workBundle) {
    return static_cast<Status>(mComponent->queue_nb( & amp;c2works));
}

c2_status_t SimpleC2Component::queue_nb(std::list<std::unique_ptr<C2Work>> * const items) {
    {
    if (queueWasEmpty) {
        (new AMessage(WorkHandler::kWhatProcess, mHandler))->post();
    }
}

bool SimpleC2Component::processQueue() {
    }
    process(work, mOutputBlockPool);
}

HIDL interface

  • IComponentStore

    C2ComponentStore (This defines various interfaces, codec2client/C2PlatformComponentStore all inherit it and implement the interfaces inside.)
    What interfaces are there? Mainly
    createComponent: Create various codec components
    createInterface: Create configurations that define various components
    listComponents: List all components.

  • IComponent

    It mainly defines various operations on components, which can actually be divided into data flow and control flow. The data flow includes configuring the encoding input surface, decoding the output surface, inputting the decoding package, and clearing the encoding and decoding data. Control flow: start components, exit components, release components, etc.

    connectToInputSurface: Use surface to start the component

    queue: Put the work into the component.
    drain: Empty the component, not block the operation.

    setOutputSurface: Set the output surface.

    start: Start the component.

    stop: stop component.

Basic codec2 service

frameworks\av\media\codec2\hidl\services\vendor.cpp
An [email protected] will be started in the rc inside.
What is implemented in this main function is a componentStore.

 store = new utils::ComponentStore(
                std::make_shared<StoreImpl>());
            constexpr char const* serviceName = "default";
            if (store->registerAsService(serviceName) != OK) {
                LOG(ERROR) << "Cannot register Codec2's IComponentStore service"
                              " with instance name << ""
                           << serviceName << "".";
            } else {
                LOG(DEBUG) << "Codec2's IComponentStore service registered. "
                              "Instance name: "" << serviceName << "".";
            }