Overview
Maoer FM is China’s largest voice content sharing platform for the post-95s generation and one of the important platforms of Bilibili. It has in-depth cooperation with top domestic voice actor studios and created hundreds of high-quality radio dramas, with a total broadcast volume of more than 10 billion times on the entire site.
MEPlayer is a cross-process playback framework developed by the Maoer Android technical team and is suitable for various scenarios such as audio and video, live broadcast, and special effects playback. Currently supports:
- Audio and video, live broadcast, special effects playback.
- It supports custom playback kernels. Currently, exo and bbp (lightweight playback kernels developed by the multimedia department) are built-in. Both of them have added support for simultaneous playback. You can expand your support for ijk and other kernels by implementing a few fixed interfaces.
- It is completely decoupled from the business and has been used by multiple teams within the company.
- Play audio, video, live broadcast, and special effects across processes, and automatically resume after the playback process is killed.
- Automatically manage audio focus and support ignoring focus preemption (playing simultaneously with other audio and video applications).
- It supports display in the notification bar and broadcast control center, and has been adapted to domestic systems including Hongmeng (Maoer FM has passed Huawei testing and is configured in the Hongmeng system source code whitelist).
- Background playback optimization: bbp background video playback supports pausing video decoding; maintaining network connection in the background during playback; it can simultaneously increase the priority of the main process and the playback process to ensure that the application survives longer during playback.
- Supports resolution switching, audio and video switching during playback, automatic retry when starting playback errors or errors during playback.
- Basic functions: playlist, loop mode, progress jump, fast forward, rewind, double speed playback, set volume, skip the beginning and end of the title, timed pause, play the entire first pause, etc.
For specific usage scenarios, please refer to Maoer FM APP:
Audio and video main playback scene: Audio and video playback page;
Live broadcast, special effects: Live broadcast room;
Short video scene: Home page recommendation -> Little Dreamland;
List playback and transition to the play page scene: Home page recommendation -> Play big card;
Voiceover Show: Discovery -> Events -> Voiceover Events;
Single audio and video, special effects playback scenes: Personal homepage avatar sound, homepage click blind box theater, event-> fortune voice, homepage sound recommended UP main playback under the lover tab, mine-> startup sound, etc. .
Origin
A large number of audio and video playback scenarios in the old version of Maoer FM APP used a variety of player solutions such as ijk, ExoPlayer, MediaPlayer, etc., and the playback logic and business logic were highly coupled. When new demands arise in the playback scenarios, the cost of modification is huge. Moreover, bugs are prone to occur in the process of writing required code; the code related to the original playback scene lacks modularity and the degree of code reuse is low, which in turn affects later maintenance. Therefore, the project urgently needs a unified playback framework to meet the needs of different scenarios. After investigating the mainstream playback frameworks, we found that it is difficult to satisfy our diverse scenarios at the same time. After investigating the mainstream playback frameworks, we found that there is no ready-made solution that can meet the diverse scenarios of the project, so we developed MEPlayer, which has 0 repetitive logic, 0 business coupling, API-friendly, and minimal development understanding and access costs.
Player process
The picture below is a simple playback flow chart.
MEPlayer and MEDirectPlayer are two player portals that are in direct contact with audio, video and live broadcast services. MEPlayer supports cross-process playback, while MEDirectPlayer plays directly in the main process. The basic API and playback logic code of these two Players are shared. The differences are Compared with MEPlayer, MEDirectPlayer lacks the ability to connect to the broadcast control center in terms of the connection between the player entry instance and the kernel encapsulated Player.
Why do you need MEDirectPlayer? Because for scenes such as splash screens and startup sounds that need to be played within one or two seconds of starting the APP, it is too late to play across processes. There may be a situation where the process is not connected when the playback is needed. The cross-process logic is relatively complex, so it is more friendly to separate the player entrance for later maintenance and business understanding.
For video and special effects playback, you need to bind the Surface of the video/special effects container. SurfaceListener is managed inside the player. The business only needs to pass the container View to the playback framework. Currently, TextureView and SurfaceView are supported. If the business has set up SurfaceListener, the framework It will also be compatible. When the corresponding method is called back, the old listener will be called back at the same time. When the video card in the list scene is switched, the listener set by the business will be returned to the previous card. Special effects playback is quite special. The player entrance is AlphaVideoPlayer, and the playback core API used is also different. They are independent methods in cross-process AIDL calls, but the APIs called by the business are consistent with audio and video playback.
The state machine of the playback framework is shown in the figure below:
The interceptor mode used in the start-up processing process. For global https, stream-free processing and other operations, you can customize an interceptor and inject it into the player. When there is no url information for a certain item in the list playback, you can also inject it into the default In the interceptor callback, the request interface returns a new URL for playback.
interface PlayerPreProcessor {
Val name: String
/**
* Processor id, business-customized id starts from 100, the first 100 are reserved for the framework
*/
valid:Int
/**
* Processor call priority. The larger the value, the greater the priority. The maximum is 100. When setting, pay attention to check the priorities of other existing processors and try not to repeat them.
*/
@get:IntRange(from = 0L, to = 100L)
val priority: Int
/**
* @param url original url
* @param playItem The current item in the playlist, empty if there is no list
* @param playParam play parameters
* @param scope coroutine scope
* @return output result
*/
suspend fun process(url: String?, playItem: PlayItem?, playParam: PlayParam?, scope: CoroutineScope): PlayerPreProcessResult
}
Player callbacks are uniformly in the form of kotlin dsl. A simple example is as follows:
private val mPlayer = MEPlayer(this).apply {
onReady {
// Open url resource successful callback
}
onDuration {
//Update duration
}
onPlayingStateChanged { isPlaying, from ->
//Update playback status
}
onPositionUpdate {
//Update playback progress
}
onCompletion {
// End of playback
}
onRetry {
// If a playback error occurs, onRetry will be automatically called to retry. If the business is not implemented, it will jump to onError.
// onRetry is a suspend method that can perform time-consuming operations and needs to return a url, which can be player.originUrl or a new url returned by the request backend.
}
onError {
// error handling
}
}
MEPlayer supports passing in LifecycleOwner, which can be automatically released when LifecycleOwner onDestroy. The construction method is:
/**
* Player construction method, most scenes should use MEPlayer, which will play across processes
*
* @param lifecycleOwner LifecycleOwner object. For scenes that can continue to play after exiting the page, you can pass ProcessLifecycleOwner.get(). For other scenes, you can pass the LifecycleOwner of the page.
* @param from is used to display the business source on the log tag. You can pass the TAG of the page. By default, the className of the page where lifecycleOwner is located is used.
* @param type player type, the default value is PLAYER_TYPE_AUTO
* PLAYER_TYPE_AUTO -> Select the player based on the value corresponding to "player_type" in the disk cache key-value pair. If it is "exo", use ExoPlayer.
* If "bbp", use BBP player, and use ExoPlayer by default.
* PLAYER_TYPE_BB_PLAYER -> use BBP player
* PLAYER_TYPE_EXO_PLAYER -> Use ExoPlayer
* @param scope Coroutine scope, used to create coroutines in the player object and manage the coroutine life cycle. The default value is lifecycleOwner.lifecycleScope
*/
class MEPlayer @JvmOverloads constructor(
lifecycleOwner: LifecycleOwner,
from: String = lifecycleOwner.tagName(),
@PlayerType type: String = PLAYER_TYPE_AUTO,
scope: CoroutineScope = lifecycleOwner.lifecycleScope
)
The playback framework also supports multi-instance scenes. The dubbing show and the Little Dreamland scene are both silent videos and audio played together, so when playing across processes, multiple instances must be supported to play at the same time. Let’s take a look at a log from the player:
// Audio // playback process I/ServicePlayer.Hypnosis.bbp.core1 onReady I/ServicePlayer.Hypnosis.bbp.core1 onPlaying, needRequestFocus: true I/ServicePlayer.Hypnosis.bbp.core1 updatePlaybackState, shouldShowInMediaSession: true, enableNotification: true, enableRating: false, enableLyric: false // main process I/MEPlayer.Hypnosis.bbp.core1 onReady I/MEPlayer.Hypnosis.bbp.core1 updatePlayingState, isPlaying: true, reason: 1 (open), position: 12 (00:00), notifyCallback: true, notifyNotification: true // video // playback process I/ServicePlayer.HypnosisHomeFragment.bbp.core2 onReady I/ServicePlayer.HypnosisHomeFragment.bbp.core2 onPlaying, needRequestFocus: false I/ServicePlayer.HypnosisHomeFragment.bbp.core2 updatePlaybackState, shouldShowInMediaSession: false, enableNotification: false, enableRating: false, enableLyric: false // main process I/MEPlayer.HypnosisHomeFragment.bbp.core2 onReady I/MEPlayer.HypnosisHomeFragment.bbp.core2 updatePlayingState, isPlaying: true, reason: 1 (open), position: 21 (00:00), notifyCallback: true,
It can be seen that the player log adopts a multi-level TAG structure. In each class of the main process of the playback framework, the printed log can directly see the class, business, playback kernel type and kernel instance where the log is currently printed. index. Player instances are stored using SparseArrayCompat, and the main process and the playback process ensure a one-to-one correspondence between instance indexes.
In the scene where the list video playback transitions to the play page, the instance needs to be seamlessly transitioned. The framework will pass the parameters of the play page instance to the list instance, and then release the original instance. The entire process of playback is continuous.
Player optimization
In terms of network connection, ExoPlayer has officially supported Cronet. Through cooperation with the multimedia department and the main website, bbp has also added Cronet support. Cronet is a network library developed by Google and is also the network stack of Chrome. It provides high performance and reliable network access capabilities, supporting HTTP, HTTP/2 and HTTP/3 protocols. Under HTTP/3, 90% of users’ playback speed is increased by more than 100ms.
In addition, ExoPlayer’s caching support is actually not friendly. An essential function of the audio APP is to continuously cache the entire audio during playback, and the progress bar will update the cache progress. However, it is difficult to directly implement this with ExoPlayer. The industry generally uses AndroidVideoCache to implement it, which is not elegant. Here I have modified part of the source code of ExoPlayer and added support. The content is too long to explain in detail.
Audio focus management
Audio focus is automatically applied for and released within the framework. The business only needs to set the audio focus type and whether to ignore focus preemption (that is, play simultaneously with other applications) when initializing the player.
player.run {
audioFocusGain = AUDIO_FOCUS_GAIN_TRANSIENT
ignoreFocusLoss = true
}
There will be focus monitoring and handling in each player instance
Background playback optimization
After the application retreats to the background, if the process (including the main process) is not a foreground process, it is likely to be killed by the system within a few seconds. Then you need to set the playback process as the foreground process by calling startForeground(int id, Notification notification) during playback. The foreground process needs to bind a notification. After retreating to the background, you can find the playback process The survival rate is significantly improved, but after playing for a while you will find that the main process is gone. That is to say, both the main process and the playback process need to be set as foreground processes, but in terms of product requirements we only have one player notification, so the main process needs to use the same notification content as the playback process to start the foreground process to ensure that the user does not switch audio. See a non-playing notification flashing up. Here, our main process also opens a notification service to update notifications. The playback process only needs to bind notifications when the foreground process is started, and subsequent notification updates are completed by the main process. When playing back and printing the priority in the background, you can see that both processes have higher priority.
> adb shell $ cat /proc/`pidof cn.missevan`/oom_adj 3 $ cat /proc/`pidof cn.missevan:player`/oom_adj 3
Another situation is that the main process is alive, but the playback process is killed, or the playback process has a problem and crashes. At this time, the main process needs to resume the playback process. It not only needs to start the process, but also needs to maintain the original progress and resume playback. , you also need to create a new notification to start the foreground process. These steps all need to get the original data, and it is not reliable to store these data in the playback process. Therefore, the steps executed by the main process need to save the data for use after the playback process reconnects.
Playback failure retries include scenarios such as network disconnection but media data is not cached, link failure, seek failure, resolution switching failure, audio and video switching failure, etc. The retry logic of these scenarios is different, and the code logic must be ensured. It is difficult to clearly meet the requirements without duplicating the code. Fortunately, after sorting out the similarities and differences, the logic is gathered together, which is also more friendly for later expansion. Here, playType is used to distinguish scenarios. The core logic is as follows:
val playParamApplier: PlayParam.() -> Unit = {
//Reuse the last parameters when retrying
from(currentPlayParam)
// Retries will keep the original setting of playWhenReady. Even if the original request does not require keepPlayingState, the retry can be set to true, because the original request has taken effect, and it can be maintained by retrying.
keepPlayingState = true
isSwitchUrl = true
stopPrevious = false
isRetry = true
// For some errors, convert the playback type
when (errorCode) {
PLAYER_ERROR_CODE_OPEN_FAILED -> {
// If the opening fails, just reopen it according to the original parameters. isSwitchUrl must pass false, otherwise there will be no onReady and onDuration callbacks.
isSwitchUrl = false
position = [email protected]
}
PLAYER_ERROR_CODE_SEEK_FAILED -> {
playType = PLAYER_PLAY_TYPE_SEEK_RETRY
}
PLAYER_ERROR_CODE_SWITCH_QUALITY_FAILED -> {
// After the first error in bbp switching resolution, it will go here to perform a retry. To retry, you need to change the playback type.
playType = PLAYER_PLAY_TYPE_SWITCH_QUALITY_RETRY
}
}
}
To pause video decoding after entering the background and leaving the video page, you need to set the LifecycleOwner corresponding to the page where the video container is located, and call videoPageLifecycleOwner = this@XXXFragment. If not set, the LifecycleOwner in the construction method will be used. When playing in the background, use WifiLockManager and WakeLockManager to enable Wi-Fi lock and wake lock so that the application can continue to connect to the Internet in the background to ensure smooth playback.
In domestic ROMs, if you want to continue playing in the background, it is safest to ensure that the relevant permissions for running the application are sufficient. Therefore, we have also added a background playback optimization settings page. This page is not provided in the framework and needs to be set by the business itself. accomplish.
Notification bar and broadcast control center
For the notification bar, there are business requirements for using system media notification styles and custom layouts. These different styles of notifications basically only differ in UI display and button click processing. Other notification logic is basically the same. The Maoer playback framework allows the business to only set the difference part, and other API calls remain consistent. The notification basic data settings are as follows:
// Audio and video notification bar
player.updateNotificationData {
smallIcon = R.drawable.ic_player_notification
actionList = arrayListOf(
PLAYER_NOTIFICATION_ACTION_PLAY,
PLAYER_NOTIFICATION_ACTION_PAUSE,
PLAYER_NOTIFICATION_ACTION_PREVIOUS,
PLAYER_NOTIFICATION_ACTION_NEXT,
PLAYER_NOTIFICATION_ACTION_FAST_FORWARD,
PLAYER_NOTIFICATION_ACTION_REWIND
)
showActionsInCompactView = arrayListOf(1, 2, 3)
contentAction = AppConstants.PLAY_ACTION
contentClassName = MainActivity::class.java.name
bizType = PLAYER_FROM_MAIN
groupId = NotificationChannels.Play.groupId
channelId = NotificationChannels.Play.channelId
channelName = NotificationChannels.Play.channelName
channelDesc = NotificationChannels.Play.channelDescription
visibility = NotificationCompat.VISIBILITY_PUBLIC
}
// Live notification bar
updateNotificationData {
smallIcon = R.drawable.ic_notification_small
forceOngoing = true
customLayout = R.layout.layout_notification_live_meplayer
coverRadius = 4
defaultCover = R.drawable.notification_live_default_avatar
contentAction = AppConstants.PLAY_ACTION
contentClassName = MainActivity::class.java.name
bizType = PLAYER_FROM_LIVE
groupId = NotificationChannels.Live.groupId
channelId = NotificationChannels.Live.channelId
channelName = NotificationChannels.Live.channelName
channelDesc = NotificationChannels.Live.channelDescription
visibility = NotificationCompat.VISIBILITY_PUBLIC
}
For the adaptation of broadcast control, we mainly need to consider the differences between domestic ROMs such as MIUI and ColorOS and Hongmeng. Except for Hongmeng, you can basically update the MediaSession according to the official documents. For Hongmeng, more adaptations are required. For example, Hongmeng supports the two figures below. Scenario:
Logics such as lyrics, collections, fast forward and rewind need to be processed according to different business settings. Currently, the business only needs to call the corresponding fields of the player to set up, and it is relatively simple to use.
Summary
This article introduces Maoer FM’s practical experience in developing a media playback framework on the Android platform, including architecture design, core technology, optimization and improvement, etc. I hope that this article can provide some useful reference and inspiration to the majority of Android developers. We also welcome your valuable comments and suggestions.
Android learning notes
Android performance optimization: https://qr18.cn/FVlo89
The underlying principles of Android Framework: https://qr18.cn/AQpN4J
Android car version: https://qr18.cn/F05ZCM
Android reverse security study notes: https://qr18.cn/CQ5TcL
Android audio and video: https://qr18.cn/Ei3VPD
Jetpack family bucket article (including Compose): https://qr18.cn/A0gajp
OkHttp source code analysis notes: https://qr18.cn/Cw0pBD
Kotlin article: https://qr18.cn/CdjtAF
Gradle article: https://qr18.cn/DzrmMB
Flutter article: https://qr18.cn/DIvKma
Eight major bodies of Android knowledge: https://qr18.cn/CyxarU
Android core notes: https://qr21.cn/CaZQLo
Android interview questions from previous years: https://qr18.cn/CKV8OZ
The latest Android interview question set in 2023: https://qr18.cn/CgxrRy
Interview questions for Android vehicle development positions: https://qr18.cn/FTlyCJ
Audio and video interview questions: https://qr18.cn/AcV6Ap