Use MediaCodec purpose
MediaCodec is a part of Android’s underlying multimedia framework. It is usually used in conjunction with MediaExtractor, MediaMuxer, and AudioTrack to encode common audio and video formats such as H264, H265, AAC, and 3gp.
MediaCodec works by processing input data to produce output data
MediaCodec workflow
The data stream of MediaCodec is divided into input and output streams, and the two data streams are processed asynchronously. MediaCodec does not process the data until the output buffer is manually released.
- input stream: the client inputs data to be decoded or encoded
- output stream: decoded or encoded data output by the client
MediaCodec API description
- getInputBuffers: Get the required input stream queue and return the ByteBuffer array
- queueInputBuffer: input inflow queue
- dequeueInputBuffer: Get data from the input stream queue for encoding operation
- getOutputBuffers: Get the data output stream queue after encoding and decoding, and return the ByteBuffer array
- dequeueOutputBuffer: Take out the data after the encoded operation from the output queue
- releaseOutputBuffer: processing is complete, release the output buffer
Basic process
- The basic use of MediaCodec follows the above diagram, and its life cycle is as follows:
- Stopped: create a MediaCodec, configure it, or an error occurs
- Uninitialized: When a MediaCodec object is created, MediaCodec is in Uninitialized at this time, and the reset() method is called in any state to return MediaCodec to the Uninitialized state
- Configured: Use the configure(…) method to configure MediaCodec to the Configured state
- Error: An error occurred
- Executing: You can return to the Flushed state by calling the flush() method at any time in the Executing state
- Flushed: MediaCodec immediately enters the Flushed state after calling the start() method
- Running: After calling dequeueInputBuffer, MediaCodec will transfer to the Running state
- End-of-Stream: After the codec ends, MediaCodec will transfer to the End-of-Stream substate
- Released: After using MediaCodec, you must call the release() method to release its resources
Basic usage
//decoder
val mVideoDecoder = MediaCodec.createDecoderByType("video/avc")
//Encoder
val mVideoEncoder = MediaCodec.createEncoderByType("video/avc")
MediaCodec decoding H264/H265
Use MediaCodec to decode H264/H265 stream video, then we must talk about the artifact of MediaCodec. The data flow chart attached to the official website is as follows:
input: ByteBuffer input side;
output: ByteBuffer output side;
- The user requests an empty input buffer (ByteBuffer) from MediaCodec, fills it with data and passes it to MediaCodec for processing.
- MediaCodec processes the data and outputs the processing result to an empty output buffer (ByteBuffer).
- The user obtains the output buffer data from MediaCodec, consumes the data in it, and releases it back to the codec after using the output buffer data.
The sample code of H264 code stream decoding is as follows (basically commented)
package com.zqfdev.h264decodedemo;
?
import android.media.MediaCodec;
import android.media.MediaFormat;
import android.util.Log;
import android.view.Surface;
?
import java.io.ByteArrayOutputStream;
import java.io.DataInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.ByteBuffer;
?
/**
* @author zhangqingfa
* @createDate 2020/12/10 11:39
* @description Decoding H264 playback
*/
public class H264DeCodePlay {
?
private static final String TAG = "zqf-dev";
//video path
private String videoPath;
//Use android MediaCodec to decode
private MediaCodec mediaCodec;
private Surface surface;
?
H264DeCodePlay(String videoPath, Surface surface) {
this. videoPath = videoPath;
this.surface = surface;
initMediaCodec();
}
?
private void initMediaCodec() {
try {
Log.e(TAG, "videoPath " + videoPath);
//Create decoder H264 type is AAC
mediaCodec = MediaCodec.createDecoderByType("video/avc");
//Create configuration
MediaFormat mediaFormat = MediaFormat.createVideoFormat("video/avc", 540, 960);
//Set the expected frame rate for decoding [the key for the frame rate of the video format in frames per second]
mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 15);
//Configure binding mediaFormat and surface
mediaCodec. configure(mediaFormat, surface, null, 0);
} catch (IOException e) {
e.printStackTrace();
//Create decoding failed
Log.e(TAG, "Failed to create decoding");
}
}
?
/**
* Decode playback
*/
void decodePlay() {
mediaCodec. start();
new Thread(new MyRun()).start();
}
?
private class MyRun implements Runnable {
?
@Override
public void run() {
try {
//1. Read h264 files in IO stream mode [Load videos that are too large in batches]
byte[] bytes = null;
bytes = getBytes(videoPath);
Log. e(TAG, "bytes size " + bytes. length);
//2. Get all queue buffer[] of mediaCodec
ByteBuffer[] inputBuffers = mediaCodec. getInputBuffers();
//start position
int startIndex = 0;
//h264 total bytes
int totalSize = bytes. length;
//3, analysis
while (true) {
//Determine if it matches
if (totalSize == 0 || startIndex >= totalSize) {
break;
}
//find the index
int nextFrameStart = findByFrame(bytes, startIndex + 1, totalSize);
if (nextFrameStart == -1) break;
MediaCodec. BufferInfo info = new MediaCodec. BufferInfo();
// After querying for 10000 milliseconds, if all the buffers of the dSP chip are occupied, return -1; if it exists, it will be greater than 0
int inIndex = mediaCodec. dequeueInputBuffer(10000);
if (inIndex >= 0) {
//According to the returned index, get the buffer that can be used
ByteBuffer byteBuffer = inputBuffers[inIndex];
//Empty the cache
byteBuffer. clear();
//Start filling the buffer with data
byteBuffer.put(bytes, startIndex, nextFrameStart - startIndex);
// After filling the data, notify mediacodec to query the buffer of the inIndex index,
mediaCodec.queueInputBuffer(inIndex, 0, nextFrameStart - startIndex, 0, 0);
//Prepare for the next frame, the beginning of the next frame is the end of the previous frame.
startIndex = nextFrameStart;
} else {
//Wait for an empty buffer
continue;
}
//mediaCodec query "mediaCodec's output queue" to get the index
int outIndex = mediaCodec.dequeueOutputBuffer(info, 10000);
Log.e(TAG, "outIndex " + outIndex);
if (outIndex >= 0) {
try {
//Temporarily slow down the playback speed by sleeping thread
Thread. sleep(33);
} catch (InterruptedException e) {
e.printStackTrace();
}
//If the surface is bound, directly input to the surface to render and release
mediaCodec. releaseOutputBuffer(outIndex, true);
} else {
Log.e(TAG, "Did not decode successfully");
}
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
?
?
// read a frame of data
private int findByFrame(byte[] bytes, int start, int totalSize) {
for (int i = start; i < totalSize - 4; i ++ ) {
//Analysis of the output.h264 file can read the real data through the separator 0x00000001
if (bytes[i] == 0x00 & amp; & amp; bytes[i + 1] == 0x00 & amp; & amp; bytes[i + 2] == 0x00 & amp; & amp; bytes[i + 3 ] == 0x01) {
return i;
}
}
return -1;
}
private byte[] getBytes(String videoPath) throws IOException {
InputStream is = new DataInputStream(new FileInputStream(new File(videoPath)));
int len;
int size = 1024;
byte[] buf;
ByteArrayOutputStream bos = new ByteArrayOutputStream();
buf = new byte[size];
while ((len = is. read(buf, 0, size)) != -1)
bos.write(buf, 0, len);
buf = bos.toByteArray();
return buf;
}
}
H265 sample code is as follows
package com.zqfdev.h264decodedemo;
?
import android.media.MediaCodec;
import android.media.MediaFormat;
import android.util.Log;
import android.view.Surface;
?
import java.io.ByteArrayOutputStream;
import java.io.DataInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.ByteBuffer;
?
/**
* @author zhangqingfa
* @createDate 2020/12/10 11:39
* @description Decoding H264 playback
*/
public class H265DeCodePlay {
?
private static final String TAG = "zqf-dev";
//video path
private String videoPath;
//Use android MediaCodec to decode
private MediaCodec mediaCodec;
private Surface surface;
?
H265DeCodePlay(String videoPath, Surface surface) {
this. videoPath = videoPath;
this.surface = surface;
initMediaCodec();
}
?
private void initMediaCodec() {
try {
Log.e(TAG, "videoPath " + videoPath);
//Create decoder H264 type is AAC
mediaCodec = MediaCodec.createDecoderByType("video/hevc");
//Create configuration
MediaFormat mediaFormat = MediaFormat.createVideoFormat("video/hevc", 368, 384);
//Set the expected frame rate for decoding [the key for the frame rate of the video format in frames per second]
mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 15);
//Configure binding mediaFormat and surface
mediaCodec. configure(mediaFormat, surface, null, 0);
} catch (IOException e) {
e.printStackTrace();
//Create decoding failed
Log.e(TAG, "Failed to create decoding");
}
}
?
/**
* Decode playback
*/
void decodePlay() {
mediaCodec. start();
new Thread(new MyRun()).start();
}
?
private class MyRun implements Runnable {
?
@Override
public void run() {
try {
//1. Read h264 files in IO stream mode [Load videos that are too large in batches]
byte[] bytes = null;
bytes = getBytes(videoPath);
Log. e(TAG, "bytes size " + bytes. length);
//2. Get all queue buffer[] of mediaCodec
ByteBuffer[] inputBuffers = mediaCodec. getInputBuffers();
//start position
int startIndex = 0;
//h264 total bytes
int totalSize = bytes. length;
//3, analysis
while (true) {
//Determine if it matches
if (totalSize == 0 || startIndex >= totalSize) {
break;
}
//find the index
int nextFrameStart = findByFrame(bytes, startIndex + 1, totalSize);
if (nextFrameStart == -1) break;
Log.e(TAG, "nextFrameStart " + nextFrameStart);
MediaCodec. BufferInfo info = new MediaCodec. BufferInfo();
// After querying for 10000 milliseconds, if all the buffers of the dSP chip are occupied, return -1; if it exists, it will be greater than 0
int inIndex = mediaCodec. dequeueInputBuffer(10000);
if (inIndex >= 0) {
//According to the returned index, get the buffer that can be used
ByteBuffer byteBuffer = inputBuffers[inIndex];
//Clear the byteBuffer cache
byteBuffer. clear();
//Start filling the buffer with data
byteBuffer.put(bytes, startIndex, nextFrameStart - startIndex);
// After filling the data, notify mediacodec to query the buffer of the inIndex index,
mediaCodec.queueInputBuffer(inIndex, 0, nextFrameStart - startIndex, 0, 0);
//Prepare for the next frame, the beginning of the next frame is the end of the previous frame.
startIndex = nextFrameStart;
} else {
//Wait for an empty buffer
continue;
}
//mediaCodec query "mediaCodec's output queue" to get the index
int outIndex = mediaCodec.dequeueOutputBuffer(info, 10000);
Log.e(TAG, "outIndex " + outIndex);
if (outIndex >= 0) {
try {
//Temporarily slow down the playback speed by sleeping thread
Thread. sleep(33);
} catch (InterruptedException e) {
e.printStackTrace();
}
//If the surface is bound, directly input to the surface to render and release
mediaCodec. releaseOutputBuffer(outIndex, true);
} else {
Log.e(TAG, "Did not decode successfully");
}
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
?
?
// read a frame of data
private int findByFrame(byte[] bytes, int start, int totalSize) {
for (int i = start; i < totalSize - 4; i ++ ) {
//Analysis of the output.h264 file can read the real data through the separator 0x00000001
if (bytes[i] == 0x00 & amp; & amp; bytes[i + 1] == 0x00 & amp; & amp; bytes[i + 2] == 0x00 & amp; & amp; bytes[i + 3 ] == 0x01) {
return i;
}
}
return -1;
}
private byte[] getBytes(String videoPath) throws IOException {
InputStream is = new DataInputStream(new FileInputStream(new File(videoPath)));
int len;
int size = 1024;
byte[] buf;
ByteArrayOutputStream bos = new ByteArrayOutputStream();
buf = new byte[size];
while ((len = is. read(buf, 0, size)) != -1)
bos.write(buf, 0, len);
buf = bos.toByteArray();
return buf;
}
}
MainActivity code is as follows
package com.zqfdev.h264decodedemo;
?
import android.content.pm.PackageManager;
import android.os.Bundle;
import android.os.Environment;
import android.util.Log;
import android.view.SurfaceHolder;
import android.view.SurfaceView;
?
import java.io.File;
?
import androidx.annotation.NonNull;
import androidx.appcompat.app.AppCompatActivity;
import androidx.core.app.ActivityCompat;
?
public class MainActivity extends AppCompatActivity {
private String[] permit = {"android.permission.WRITE_EXTERNAL_STORAGE", "android.permission.READ_EXTERNAL_STORAGE"};
private H264DeCodePlay h264DeCodePlay;
// private H265DeCodePlay h265DeCodePlay;
private String videoPath;
?
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R. layout. activity_main);
checkPermiss();
initView();
}
private void checkPermiss() {
int code = ActivityCompat. checkSelfPermission(this, permit[0]);
if (code != PackageManager. PERMISSION_GRANTED) {
// No write permission, apply for write permission
ActivityCompat. requestPermissions(this, permit, 11);
}
}
private void initView() {
File dir = getExternalFilesDir(Environment. DIRECTORY_DOWNLOADS);
if (!dir. exists()) dir. mkdirs();
final File file = new File(dir, "output.h264");
// final File file = new File(dir, "output.h265");
if (!file. exists()) {
Log.e("Tag", "File does not exist");
return;
}
videoPath = file. getAbsolutePath();
final SurfaceView surface = findViewById(R.id.surface);
final SurfaceHolder holder = surface. getHolder();
holder.addCallback(new SurfaceHolder.Callback() {
@Override
public void surfaceCreated(@NonNull SurfaceHolder surfaceHolder) {
h264DeCodePlay = new H264DeCodePlay(videoPath, holder.getSurface());
h264DeCodePlay.decodePlay();
// h265DeCodePlay = new H265DeCodePlay(videoPath, holder.getSurface());
// h265DeCodePlay. decodePlay();
}
?
@Override
public void surfaceChanged(@NonNull SurfaceHolder surfaceHolder, int i, int i1, int i2) {
}
@Override
public void surfaceDestroyed(@NonNull SurfaceHolder surfaceHolder) {
}
});
}
}
The tested H264/H265 stream video can be extracted by FFmpeg.
Command Line:
ffmpeg -i source video.mp4 -codec copy -bsf: h264_mp4toannexb -f h264 output video.h264
The final effect is as follows:
