4. Use MediaExtractor+MediaCodec+AudioTrack to play audio files

Use MediaExtractor + MediaCodec + AudioTrack to play audio files

First look at a picture:

If you want to play a video, you have to deprotocol and decapsulate the video file. Then you can get the audio data and video data (separately), decode them separately again, and finally play them synchronously (because the devices that play audio and video are different, you need to audio and video synchronization).

This task is to implement the left half of the picture and play audio.
(MediaPlayer in the first few tasks helps us complete the unblocking, decoding and synchronization operations)

MediaExtractor

Used to separate video and audio data

For a common video file, there is usually a video track and an audio track. MediaExtractor is used to separate the audio and video track data.

Of course, there may be multiple video streams and multiple audio streams in one video file.

The steps of decapsulation are reflected in MediaExtractor. MediaExtractor separates and obtains data in different formats (video/audio) from the encapsulation format.

MediaCodec

The decoding step is reflected in MediaCodec, which decodes data in a specific format to obtain original audio data.

MediaCodec has two ByteBuffer arrays, namely InputBuffers and OutputBuffers, which are used to import and export data before and after decoding. The general process is:

  1. Pass the data to be decoded into MediaCodec
  2. Decoding (automatically)
  3. Get decoded data
  4. Render to Surface or load to AudioTrack

States

Stopped, Executing or Released

First use configure(…) to configure, and then call start() to enter the Executing state. The codec is immediately in the refresh substate, where it saves all buffers.

Use stop() to enter the Uninitialized state. If you still need to use it, you must configure it again.

After using the codec, release() is used to end it.

Create

createDecoderByType(String)

createByCodecName(String)

AudioTrack & amp;Surface

After decoding, combine Surface and AudioTrack for audio and video playback.

To play audio in AudioTrack, you need to obtain the number of audio channels and sampling rate to configure it.

MediaPlayer can play sound files in multiple formats, such as MP3, WAV, OGG, AAC, MIDI, etc. However, AudioTrack can only play PCM data streams. Of course, there is still a close connection between the two. When MediaPlayer plays audio, it will still create AudioTrack in the framework layer, pass the decoded PCM data stream to AudioTrack, and finally AudioFlinger will mix it and pass the audio to the hardware for playback. Using AudioTrack to play just skips the decoding part of Mediaplayer. Reference link

AudioTrack five steps to implement PCM audio playback

  • Configure basic parameters

  • Get the minimum buffer size

    bufferSizeInBytes parameter (an input parameter when building AudioTrack):

    If the track’s creation mode is MODE_STREAM , this should be the desired buffer size for the AudioTrack to satisfy the application’s latency requirements. If bufferSizeInBytes is less than the minimum buffer size for the output sink, it is increased to the minimum buffer size. The method getBufferSizeInFrames() returns the actual size in frames of the buffer created, which determines the minimum frequency to write to the streaming AudioTrack to avoid underrun. See getMinBufferSize(int,int,int) to determine the estimated minimum buffer size for an AudioTrack instance in streaming mode. ——–From Google
    You can use getBufferSizeInFrames() and getMinBufferSize(int,int,int) to get the minimum buffer size.

  • Create AudioTrack object

    AudioTrack(AudioAttributes attributes, AudioFormat format, int bufferSizeInBytes, int mode, int sessionId)
    It is also recommended to use Builder to build AudioTrack———AudioTrack.Builder | Android Developers (google.cn)

    Class constructor with AudioAttributes and AudioFormat.

    AudioAttributes Class that encapsulates audio stream information
    A class to encapsulate a collection of attributes describing information about an audio stream.

    • usage: “why” you are playing a sound, what is this sound used for. This is achieved with the “usage” information. Examples of usage are USAGE_MEDIA and USAGE_ALARM.
    • content type: “what” you are playing. The content type expresses the general category of the content. This information is optional. CONTENT_TYPE_MOVIE or CONTENT_TYPE_MUSIC .
    • flags: “how” is playback to be affected, see the flag definitions for the specific playback behaviors they control.

    AudioFormat Audio Frame This class is used to access audio format and channel configuration constants
    The AudioFormat class is used to access a number of audio format and channel configuration constants.

    • Sample rate: The sampling rate of the playback content
    • Encoding: Audio encoding method. For linear PCM (because AudioTrack only supports PCM encoded data), the audio encoding describes the sample size (8, 16, or 32 bits) and the sample representation (integer or floating point).
      When compressed audio is sent out through a direct AudioTrack, it need not be written in exact multiples of the audio access unit; this differs from MediaCodec input buffers.
    • Channel masks: Channel masks, used to describe samples and their arrangement in audio frames.
      There are two types of channel masks:
      • Channel position masks: original. Different channels have different position arrangements, CHANNEL_OUTLEFT
      • Channel index masks: Use index to select a specific channel, similar to a subnet mask.

  • Get the PCM file and convert it into DataInputStream

  • Start/stop playback

    If you don’t call write() first, or if you call write() but with an insufficient amount of data, then the track will be in underrun state at play().

    It will only be played when data is written to the buffer using write(), otherwise it will only enter a ready state after play().

Code

Android MediaCodec decodes audio, AudioTrack plays audio, and PCM data is written to files – CSDN Blog

Use MediaExtractor and MediaCodec to decode audio and video to implement a simple player_mediacode decoding audio playback-CSDN Blog

First, the relevant initialization work needs to be performed, and then the decoding operation is performed. The decoding will load the relevant data into the buffer area and then write it to AudioTrack for playback, and then clear the buffer area, and then loop until the end of the file.

Initialize MediaExtractor, AudioTrack, MediaCodec

@RequiresApi(api = Build.VERSION_CODES.M)
private void initPlayer() throws IOException {<!-- -->
    //Select audio track
    extractor = new MediaExtractor();
    try {<!-- -->
        //extractor.setDataSource("//sdcard/Movies/big_buck_bunny.mp4");
        extractor.setDataSource("//sdcard/Movies/lesson.mp4");
    } catch (IOException e) {<!-- -->
        e.printStackTrace();
    }
    int numTracks = extractor.getTrackCount();
    int trackIndex = 0;
    for (int i = 0; i < numTracks; + + i) {<!-- -->
        MediaFormat format = extractor.getTrackFormat(i);
        String mime = format.getString(MediaFormat.KEY_MIME);
        // The data type corresponding to the format can be obtained (implemented by key-value pairs, the value corresponding to the key KEY_MIME records the data stream type,
        // Video starts with video/, audio starts with audio/)
        if (mime.startsWith("audio/")) {<!-- -->
            trackIndex = i;
         }
    }
    MediaFormat audioFormat = extractor.getTrackFormat(trackIndex);
    Log.i(TAG, String.valueOf(audioFormat));
    extractor.selectTrack(trackIndex);

    // Get the number of channels and sampling rate to calculate the minimum buffer size
    int audioChannels = audioFormat.getInteger(MediaFormat.KEY_CHANNEL_COUNT);
    int audioSampleRate = audioFormat.getInteger(MediaFormat.KEY_SAMPLE_RATE);
    Log.i(TAG, "Channels is " + audioChannels + " SamplerRate is " + audioSampleRate);
    int minBuffSize = AudioTrack.getMinBufferSize(audioSampleRate,
    AudioFormat.CHANNEL_OUT_MONO,
    AudioFormat.ENCODING_PCM_16BIT);
    Log.i(TAG, "minBuffSize is " + minBuffSize);
    //Configure Configure AudioTrack
    audioTrack = new AudioTrack.Builder()
        .setAudioAttributes(new AudioAttributes.Builder()
            .setUsage(AudioAttributes.USAGE_MEDIA)
            .setContentType(AudioAttributes.CONTENT_TYPE_MUSIC)
            .build())
        .setAudioFormat(new AudioFormat.Builder()
             .setEncoding(AudioFormat.ENCODING_PCM_16BIT)
             .setSampleRate(audioSampleRate)
             .setChannelMask(AudioFormat.CHANNEL_OUT_MONO)
             .build())
        .setBufferSizeInBytes(Math.max(minBuffSize, 2048))
            .setTransferMode(AudioTrack.MODE_STREAM)
            .build();
    // The play() here in AudioTrack is a ready state. It needs to read data from the buffer before it can be played, so it can only be played after MediaCodec decodes the data and inputs it into the buffer.
    audioTrack.play();


    // Configuration of MediaCodec decoder
    audioCodec = MediaCodec.createDecoderByType(audioFormat.getString(MediaFormat.KEY_MIME));
    Log.i(TAG, "MediaCodec.createDecoderByTypes audioFormat is " + audioFormat.getString(MediaFormat.KEY_MIME));
    audioCodec.configure(audioFormat, null, null, 0);
    audioCodec.start();
}

Play audio

private void decodeAudio() {<!-- -->
    MediaCodec.BufferInfo audioBufferInfo = new MediaCodec.BufferInfo();
    long startMs = System.currentTimeMillis();
    ByteBuffer inputBuffer;
    boolean isEOS = false;
    while (!isEOS & amp; & amp; !Thread.currentThread().isInterrupted()) {<!-- -->
// !Thread.currentThread().isInterrupted() is used to get whether the current thread is interrupted. If it is interrupted, it will end directly.
        // Get the index of the available input buffer
        int inputBufferId = audioCodec.dequeueInputBuffer(10000);
        Log.i(TAG, "The inputBufferId is " + inputBufferId);
        if (inputBufferId >= 0) {<!-- -->
            // Get the input buffer and write data to the buffer
            inputBuffer = audioCodec.getInputBuffer(inputBufferId);
            int sampleSize = extractor.readSampleData(inputBuffer, 0);
            if (sampleSize > 0) {<!-- -->
                // Submit the inputBuffer filled with data to the encoding queue
                audioCodec.queueInputBuffer(inputBufferId, 0, sampleSize, extractor.getSampleTime(), 0);
                extractor.advance();
            } else {<!-- -->
                isEOS = true;
                break;
            }
        }
        // Get the index of the output buffer that has been successfully encoded and decoded
        int outputBufferId = audioCodec.dequeueOutputBuffer(audioBufferInfo, 10000);
        Log.i(TAG, "the outputBufferId is " + outputBufferId);
        byte[] pcmData;
        while (outputBufferId >= 0) {<!-- -->
            // Get the output buffer
            ByteBuffer outputBuffer = audioCodec.getOutputBuffer(outputBufferId);
            pcmData = new byte[audioBufferInfo.size];
            if (outputBuffer != null) {<!-- -->
                outputBuffer.get(pcmData);
                outputBuffer.clear();//Empty it after use and reuse it
            }
            audioTrack.write(pcmData, 0, audioBufferInfo.size);
            Log.i(TAG, "audioTrack.write");
            //Get the decoded data
            int outputIndex = audioCodec.dequeueOutputBuffer(audioBufferInfo, 10000);

            audioCodec.releaseOutputBuffer(outputBufferId, false);
            outputBufferId = audioCodec.dequeueOutputBuffer(audioBufferInfo, 10000);
        }
    }
    audioCodec.stop();
    audioCodec.release();
    extractor.release();
    audioTrack.stop();
    audioTrack.release();
}

Interrupt thread

When the thread’s run method executes the last statement in the method body and then executes the return statement to return, or an exception that is not caught in the method occurs, the thread will be terminated.

In early versions of Java, there was also a stop method to terminate the thread, but this method has been deprecated.

Now except for the deprecated stop method (because it is unsafe), there is no way to force the thread to terminate.

However, you can use the interrupt method to request the termination of a thread:

When the interrupt method is called on a thread, the interrupt status of the thread is set, which is a boolean value. Each thread will check this flag from time to time to determine whether the thread is interrupted.

while (!Thread.currentThread().isInterrupted())
{
do more work
}