Android application development (36) frame rate API test based on Surfaceview

Android application development study notes – directory index

Refer to the android official website:

1. Frame rate | Android media | Android Developers
2. Multiple Refresh Rates | Android Open Source Project | Android Open Source Project
3. WindowManager.LayoutParams | Android Developers

Android 11 has added support for devices with multiple refresh rates. At present, flagship phones on the market are basically popularized with LTPO screens. This chapter analyzes the code related to frame rate, and then writes a test program for verification (based on Android application development (35 ) Basic usage of SufaceView).

1. API interface called by APP

1. APP obtains all frame rates supported by the screen

The application obtains the display refresh rate actually supported by the device, which can be obtained by calling Display.getSupportedModes(), and Mode.mRefreshRate is the frame rate information, so that it is safe to call setFrameRate()

// Display.java
Display.Mode[] mSupportedModes = getWindowManager().getDefaultDisplay().getSupportedModes();
for (Display. Mode mode : mSupportedModes) {
   Log.d(TAG, "getSupportedModes: " + mode.toString());
   Log.d(TAG, "getRefreshRate: " + mode.getRefreshRate());
}
 

public static final class Mode implements Parcelable {
        public static final Mode[] EMPTY_ARRAY = new Mode[0];
 
        private final int mModeId;
        private final int mWidth;
        private final int mHeight;
        private final float mRefreshRate;
        @NonNull
        private final float[] mAlternativeRefreshRates;
...
}

2. API for APP setting frame rate

setFrameRate( )

Android exposes several methods of accessing and controlling the interface, so there are multiple versions of the setFrameRate() API. Each version of the API takes the same parameters and works the same as the other versions:

• Surface. setFrameRate()
• SurfaceControl.Transaction.setFrameRate()
• ANativeWindow_setFrameRate()
• ASurfaceTransaction_setFrameRate()

To see if calling setFrameRate() causes the display refresh rate to change, register for display change notifications by calling DisplayManager.registerDisplayListener() or AChoreographer_registerRefreshRateCallback().

When calling setFrameRate(), it is better to pass the exact frame rate rather than rounding to an integer. For example, when rendering video recorded at 29.97Hz, pass in 29.97 instead of rounding to 30.

For video applications, the compatibility parameter passed to setFrameRate() should be set to Surface.FRAME_RATE_COMPATIBILITY_FIXED_SOURCE to provide an additional hint to the Android platform that the application will use pulldown to accommodate mismatched display refresh rates (which would cause judder).

surface.setFrameRate(contentFrameRate,
FRAME_RATE_COMPATIBILITY_FIXED_SOURCE,
CHANGE_FRAME_RATE_ONLY_IF_SEAMLESS);
// When calling setFrameRate(), it is better to pass in the exact frame rate instead of rounding to an integer. For example, when rendering video recorded at 29.97Hz, pass in 29.97 instead of rounding to 30.
Parameters: Surface.FRAME_RATE_COMPATIBILITY_FIXED_SOURCE applies only to video applications. For non-video use, use FRAME_RATE_COMPATIBILITY_DEFAULT.
Choose a strategy for changing the frame rate:
Google strongly recommends that apps call setFrameRate(fps, FRAME_RATE_COMPATIBILITY_FIXED_SOURCE, CHANGE_FRAME_RATE_ALWAYS) when displaying long-running videos such as movies, where fps is the frame rate of the video.
It is strongly recommended that you do not call setFrameRate() with CHANGE_FRAME_RATE_ALWAYS when you expect video playback to last a few minutes or less.

SurfaceControl.Transaction.setFrameRate()
The parameters are the same as surface.setFrameRate

setFrameRate( ) and PreferredDisplayModeId

WindowManager.LayoutParams.preferredDisplayModeId is another way for applications to indicate their frame rate to the platform. Some applications only want to change the display refresh rate, but not other display mode settings, such as display resolution. In general, use setFrameRate() instead of preferredDisplayModeId. The setFrameRate() function is easier to use because the application does not need to search the list of display modes to find a mode with a specific frame rate.

setFrameRate() gives the platform more opportunities to choose a compatible framerate in the presence of multiple surfaces running at different framerates. For example, consider a scenario where two apps are running in split-screen mode on a Pixel 4, where one app is playing a 24Hz video and the other app presents a scrollable list to the user. The device supports two display refresh rates: 60Hz and 90Hz. Using the preferredDisplayModeId API, the video surface is forced to choose 60Hz or 90Hz. By using the setFrameRate()24Hz call, the video surface provides the platform with more information about the frame rate of the source video, enabling the platform to choose a display refresh rate of 90Hz, which is better than 60Hz in this scenario .

However, in some scenarios preferredDisplayModeId should be used instead of setFrameRate(), for example:

• Use the preferredDisplayModeId if the application wants to change the resolution or other display mode settings.
• setFrameRate()The platform will only switch display modes in response to calls if the mode switch is lightweight and unlikely to be noticed by the user. If an application prefers toggling display refresh rates even if it requires a lot of mode switching (for example, on Android TV devices), use preferredDisplayModeId.
• Applications that cannot handle displays running at multiples of the application’s frame rate (which would require setting a presentation timestamp on each frame) should use the preferredDisplayModeId.

Display.Mode[] mSupportedModes;
mSupportedModes = getWindowManager().getDefaultDisplay().getSupportedModes();

WindowManager.LayoutParams params = getWindow().getAttributes();
params.preferredDisplayModeId = mSupportedModes[x].getModeId();
getWindow().setAttributes(params);

Log.d(TAG, “RefreshRate:” + mSupportedModes[x].getRefreshRate());

setFrameRate( ) vs. PreferredRefreshRate

WindowManager.LayoutParams#preferredRefreshRate Sets the preferred frame rate on the application window and applies to all surfaces within the window. Regardless of the refresh rate supported by the device, the application should specify its preferred frame rate, similar to setFrameRate(), to give the scheduler a better hint of the application’s expected frame rate.

preferredRefreshRate will be ignored for surfaces using setFrameRate(). Generally use setFrameRate() if possible.

PreferredRefreshRate and PreferredDisplayModeId

If the application only wants to change the preferred refresh rate, it is better to use preferredRefreshRate instead of preferredDisplayModeId.

WindowManager.LayoutParams params = getWindow().getAttributes();
params.PreferredRefreshRate = preferredRefreshRate;

getWindow().setAttributes(params);

Log.d(TAG, “preferredRefreshRate:” + preferredRefreshRate);

Avoid calling setFrameRate( ) too frequently

Although the performance cost of this setFrameRate() call is not high, applications should avoid calling setFrameRate() multiple times per frame or per second. Calling setFrameRate() may cause the display refresh rate to change, which may cause frame drops during transitions. You should calculate the correct frame rate ahead of time and call setFrameRate() once.

for games or other non-video applications

Although video is the primary use case for this API setFrameRate(), it can be used in other applications as well. For example, a game that intends to run at no higher than 60Hz (to reduce power consumption and enable longer playtime) can call Surface.setFrameRate(60, Surface.FRAME_RATE_COMPATIBILITY_DEFAULT). The device running it will run at 60Hz while the game is active, this will avoid the judder that occurs when the game is running at 60Hz and the monitor is running at 90Hz.

2. Code Analysis

1. surface. setFrameRate()

// Surface.java (frameworks\base\core\java\android\view)
    /**
     * Sets the intended frame rate for this surface.
     *
     * <p>On devices that are capable of running the display at different refresh rates,
     * the system may choose a display refresh rate to better match this surface's frame
     * rate. Usage of this API won't introduce frame rate throttling, or affect other
     * aspects of the application's frame production pipeline. However, because the system
     * may change the display refresh rate, calls to this function may result in changes
     * to Choreographer callback timings, and changes to the time interval at which the
     * system releases buffers back to the application.</p>
     *
     * <p>Note that this only has an effect for surfaces presented on the display. If this
     * surface is consumed by something other than the system compositor, e.g. a media
     * codec, this call has no effect.</p>
     *
     * @param frameRate The intended frame rate of this surface, in frames per second. 0
     * is a special value that indicates the app will accept the system's choice for the
     * display frame rate, which is the default behavior if this function isn't
     * called. The <code>frameRate</code> parameter does <em>not</em> need to be a valid refresh
     * rate for this device's display - e.g., it's fine to pass 30fps to a device that can only run
     * the display at 60fps.
     *
     * @param compatibility The frame rate compatibility of this surface. The
     * compatibility value may influence the system's choice of display frame rate.
     * This parameter is ignored when <code>frameRate</code> is 0.
     *
     * @param changeFrameRateStrategy Whether display refresh rate transitions caused by this
     * surface should be seamless. A seamless transition is one that doesn't have any visual
     * interruptions, such as a black screen for a second or two. This parameter is ignored when
     * <code>frameRate</code> is 0.
     *
     * @throws IllegalArgumentException If <code>frameRate</code>, <code>compatibility</code> or
     * <code>changeFrameRateStrategy</code> are invalid.
     */
    public void setFrameRate(@FloatRange(from = 0.0) float frameRate,
            @FrameRateCompatibility int compatibility,
            @ChangeFrameRateStrategy int changeFrameRateStrategy) {
        synchronized(mLock) {
            checkNotReleasedLocked();
            int error = nativeSetFrameRate(mNativeObject, frameRate, compatibility,
                    changeFrameRateStrategy);
            if (error == -EINVAL) {
                throw new IllegalArgumentException("Invalid argument to Surface. setFrameRate()");
            } else if (error != 0) {
                throw new RuntimeException("Failed to set frame rate on Surface");
            }
        }
    }

    /**
     * Sets the intended frame rate for this surface. Any switching of refresh rates is
     * most probably going to be seamless.
     *
     * @see #setFrameRate(float, int, int)
     */
    public void setFrameRate(
            @FloatRange(from = 0.0) float frameRate, @FrameRateCompatibility int compatibility) {
        setFrameRate(frameRate, compatibility, CHANGE_FRAME_RATE_ONLY_IF_SEAMLESS);
    }

// android_view_Surface.cpp (frameworks\base\core\jni)
static jint nativeSetFrameRate(JNIEnv* env, jclass clazz, jlong nativeObject, jfloat frameRate,
                               jint compatibility, jint changeFrameRateStrategy) {
    Surface* surface = reinterpret_cast<Surface*>(nativeObject);
    ANativeWindow* anw = static_cast<ANativeWindow*>(surface);
    // Our compatibility is a Surface.FRAME_RATE_COMPATIBILITY_* value, and
    // NATIVE_WINDOW_SET_FRAME_RATE takes an
    // ANATIVEWINDOW_FRAME_RATE_COMPATIBILITY_* value. The values are identical
    // though, so no need to explicitly convert.
    return anw->perform(surface, NATIVE_WINDOW_SET_FRAME_RATE, double(frameRate), compatibility,
                        int(changeFrameRateStrategy));
}

// Surface.cpp (frameworks\
ative\libs\gui)
int Surface::perform(int operation, va_list args)
{
    case NATIVE_WINDOW_SET_FRAME_RATE:
        res = dispatchSetFrameRate(args);
        break;
 
int Surface::dispatchSetFrameRate(va_list args) {
    float frameRate = static_cast<float>(va_arg(args, double));
    int8_t compatibility = static_cast<int8_t>(va_arg(args, int));
    int8_t changeFrameRateStrategy = static_cast<int8_t>(va_arg(args, int));
    return setFrameRate(frameRate, compatibility, changeFrameRateStrategy);
}
 
status_t Surface::setFrameRate(float frameRate, int8_t compatibility,
                               int8_t changeFrameRateStrategy) {
    ATRACE_CALL();
    ALOGV("Surface::setFrameRate");
 
    if (!ValidateFrameRate(frameRate, compatibility, changeFrameRateStrategy,
                           "Surface::setFrameRate")) {
        return BAD_VALUE;
    }
 
    return composerService()->setFrameRate(mGraphicBufferProducer, frameRate, compatibility,
                                           changeFrameRateStrategy);
}

// ISurfaceComposer.cpp (frameworks\
ative\libs\gui)
    status_t setFrameRate(const sp<IGraphicBufferProducer> & amp; surface, float frameRate,
                          int8_t compatibility, int8_t changeFrameRateStrategy) override {
 ..
        status_t err = remote()->transact(BnSurfaceComposer::SET_FRAME_RATE, data, & amp;reply);
    }
status_t BnSurfaceComposer::onTransact(
...setFrameRate(surface, frameRate, compatibility, changeFrameRateStrategy);
 
// SurfaceFlinger.cpp (miui\frameworks\
ative\services\surfaceflinger)
status_t SurfaceFlinger::setFrameRate(const sp<IGraphicBufferProducer> & amp; surface, float frameRate,
                                      int8_t compatibility, int8_t changeFrameRateStrategy)
    layer->setFrameRate(Layer::FrameRate(Fps{frameRate}, Layer::FrameRate::convertCompatibility(compatibility), strategy))

1. The structure passed by the surfaceflinger layerinfo to save the app is FrameRate
    struct FrameRate {
        using Seamless = scheduler::Seamless;
        Fps rate;
        FrameRateCompatibility type;
        Seamless seamlessness;
}
 
enum class FrameRateCompatibility {
    Default, // layer does not specify any specific processing strategy
    Exact, // Layer needs exact frame rate. such as video
    ExactOrMultiple, // Layer requires exact frame rate (or multiple of it) to render content. Any other value will cause a dropdown.
    NoVote, // The layer does not have any requirements on the refresh rate, regardless of the display refresh rate
};
 
// The seamlessness requirement of a Layer.
enum class Seamless {
    // Indicates a requirement for a seamless mode switch.
    OnlySeamless,
    // Indicates that both seamless and seamed mode switches are allowed.
    Seamed And Seamless,
    // Indicates no preference for seamlessness. For such layers the system will
    // prefer seamless switches, but also non-seamless switches to the group of the
    // default config are allowed.
    Default
};

2, SurfaceControl.Transaction.setFrameRate()

// SurfaceControl.java (frameworks\base\core\java\android\view)
        /**
         * Sets the intended frame rate for this surface. Any switching of refresh rates is
         * most probably going to be seamless.
         *
         * @see #setFrameRate(SurfaceControl, float, int, int)
         */
        @NonNull
        public Transaction setFrameRate(@NonNull SurfaceControl sc,
                @FloatRange(from = 0.0) float frameRate,
                @Surface. FrameRateCompatibility int compatibility) {
            return setFrameRate(sc, frameRate, compatibility,
                    Surface. CHANGE_FRAME_RATE_ONLY_IF_SEAMLESS);
        }

// android_view_SurfaceControl.cpp (frameworks\base\core\jni)
static void nativeSetFrameRate(JNIEnv* env, jclass clazz, jlong transactionObj, jlong nativeObject,
                               jfloat frameRate, jint compatibility, jint changeFrameRateStrategy) {
    auto transaction = reinterpret_cast<SurfaceComposerClient::Transaction*>(transactionObj);
 
    const auto ctrl = reinterpret_cast<SurfaceControl*>(nativeObject);
    // Our compatibility is a Surface.FRAME_RATE_COMPATIBILITY_* value, and
    // Transaction::setFrameRate() takes an ANATIVEWINDOW_FRAME_RATE_COMPATIBILITY_* value. The
    // values are identical though, so no need to convert anything.
    transaction->setFrameRate(ctrl, frameRate, static_cast<int8_t>(compatibility),
                              static_cast<int8_t>(changeFrameRateStrategy));
}

// SurfaceComposerClient.cpp (frameworks\
ative\libs\gui)
SurfaceComposerClient::Transaction & SurfaceComposerClient::Transaction::setFrameRate(
        const sp<SurfaceControl> & sc, float frameRate, int8_t compatibility,
        int8_t changeFrameRateStrategy) {
    layer_state_t* s = getLayerState(sc);
    if (!s) {
        mStatus = BAD_INDEX;
        return *this;
    }
    // Allow privileged values as well here, those will be ignored by SF if
    // the caller is not privileged
    if (!ValidateFrameRate(frameRate, compatibility, changeFrameRateStrategy,
                           "Transaction::setFrameRate",
                           /*privileged=*/true)) {
        mStatus = BAD_VALUE;
        return *this;
    }
    s->what |= layer_state_t::eFrameRateChanged;
    s->frameRate = frameRate;
    s->frameRateCompatibility = compatibility;
    s->changeFrameRateStrategy = changeFrameRateStrategy;
    return *this;
}
// SurfaceFlinger.cpp (frameworks\
ative\services\surfaceflinger)
applyTransactionState
    setClientStateLocked
 
uint32_t SurfaceFlinger::setClientStateLocked(
        const FrameTimelineInfo & amp; frameTimelineInfo, const ComposerState & amp; composerState,
        int64_t desiredPresentTime, bool isAutoTimestamp, int64_t postTime, uint32_t permissions,
        std::unordered_set<ListenerCallbacks, ListenerCallbacksHash> & outListenerCallbacks) {
...
    if (what & layer_state_t::eFrameRateChanged) {
        if (ValidateFrameRate(s. frameRate, s. frameRateCompatibility, s. changeFrameRateStrategy,
                              "SurfaceFlinger::setClientStateLocked", privileged)) {
            const auto compatibility =
                    Layer::FrameRate::convertCompatibility(s. frameRateCompatibility);
            const auto strategy =
                    Layer::FrameRate::convertChangeFrameRateStrategy(s.changeFrameRateStrategy);
 
            if (layer->setFrameRate(Layer::FrameRate(Fps(s. frameRate), compatibility, strategy))) {
                flags |= eTraversalNeeded;
            }
        }
    }

3, preferredDisplayModeId

// RefreshRatePolicy.java (mframeworks\base\services\core\java\com\android\server\wm)
    int getPreferredModeId(WindowState w) {
        // If app is animating, it's not able to control refresh rate because we want the animation
        // to run in default refresh rate.
        if (w. isAnimating(TRANSITION | PARENTS)) {
            return 0;
        }
 
        return w.mAttrs.preferredDisplayModeId;
    }
// DisplayContent.java (frameworks\base\services\core\java\com\android\server\wm)
will save to
private static final class PreferredModeId of ApplySurfaceChangesTransactionState
 
    void applySurfaceChangesTransaction() {
        if (!mWmService.mDisplayFrozen) {
            mWmService.mDisplayManagerInternal.setDisplayProperties(mDisplayId,
                    mLastHasContent,
                    mTmpApplySurfaceChangesTransactionState.preferredRefreshRate,
                    mTmpApplySurfaceChangesTransactionState.preferredModeId,
                    mTmpApplySurfaceChangesTransactionState.preferredMinRefreshRate,
                    mTmpApplySurfaceChangesTransactionState.preferredMaxRefreshRate,
                    mTmpApplySurfaceChangesTransactionState.preferMinimalPostProcessing,
                    true /* inTraversal, must call performTraversalInTrans... below */);
        }
// DisplayManagerService.java (frameworks\base\services\core\java\com\android\server\display)
        public void setDisplayProperties(int displayId, boolean hasContent,
                float requestedRefreshRate, int requestedMode, float requestedMinRefreshRate,
                float requestedMaxRefreshRate, boolean requestedMinimalPostProcessing,
                boolean in Traversal) {
            setDisplayPropertiesInternal(displayId, hasContent, requestedRefreshRate,
                    requestedMode, requestedMinRefreshRate, requestedMaxRefreshRate,
                    requestedMinimalPostProcessing, inTraversal);
        }
setDisplayPropertiesInternal
|--> mDisplayModeDirector.getAppRequestObserver().setAppRequest(
                    displayId, requestedModeId, requestedMinRefreshRate, requestedMaxRefreshRate);
 
// DisplayModeDirector.java (frameworks\base\services\core\java\com\android\server\display)
        public void setAppRequest(int displayId, int modeId, float requestedMinRefreshRateRange,
                float requestedMaxRefreshRateRange) {
            synchronized(mLock) {
                setAppRequestedModeLocked(displayId, modeId);
                setAppPreferredRefreshRateRangeLocked(displayId, requestedMinRefreshRateRange,
                        requestedMaxRefreshRateRange);
            }
        }
 
setAppRequest finally passes:
updateVoteLocked(displayId, Vote.PRIORITY_APP_REQUEST_REFRESH_RATE_RANGE, vote);
It is the last call to surfaceflinger:
setDesiredDisplayModeSpecs // standard API interface of surface

3. Write an APP verification API

1. Preparatory work: add log verification to surfaceflinger

1, surface.setFrameRate() and SurfaceControl.Transaction.setFrameRate()
These two APIs are finally called bool Layer::setFrameRate(FrameRate frameRate) of layer.cpp, so we add log to this function
bool Layer::setFrameRate(FrameRate frameRate) {
    if (!mFlinger->useFrameRateApi) {
        return false;
    }
    if (mDrawingState. frameRate == frameRate) {
        return false;
    }
 
    mDrawingState.sequence++;
    mDrawingState. frameRate = frameRate;
    mDrawingState.modified = true;
    // Added verification log
    ALOGE("[lzl-test]frameRate = {%d, %s, %s}", frameRate.type, to_string(frameRate.rate).c_str(), toString(frameRate.seamlessness).c_str());
 
    updateTreeHasFrameRateVote();
 
    setTransactionFlags(eTransactionNeeded);
    return true;
}
 

2. WindowManager.LayoutParams preferredDisplayModeId
This api is finally called to setDesiredDisplayModeSpecs, so we add log to this function
status_t SurfaceFlinger::setDesiredDisplayModeSpecs(
        const sp<IBinder> & amp; displayToken, ui::DisplayModeId defaultMode, bool allowGroupSwitching,
        float primaryRefreshRateMin, float primaryRefreshRateMax, float appRequestRefreshRateMin,
        float appRequestRefreshRateMax) {
    ATRACE_CALL();
 
...
// The following is the added verification log
    ALOGD("[lzl-test]setDesiredDisplayConfigSpecs:defaultMode %d primaryRefreshRateMin: %f "
      "primaryRefreshRateMax: %f"
      "appRequestRefreshRateMin:%f appRequestRefreshRateMax:%f",
      defaultMode, primaryRefreshRateMin, primaryRefreshRateMax, appRequestRefreshRateMin,
      appRequestRefreshRateMax);

Reference cts:?cts/tests/tests/graphics/src/android/graphics/cts/FrameRateCtsActivity.java

Introduction to the running interface

adb install -t app-debug.apk

Area 1: Displays the frame rate obtained from the system

Zone 2: Use surface.setFrameRate() to test, select the frame rate, and then click OK.

//check surfaceflinger log 04-14 16:18:02.854 13261 13261 D [lzl-test][MainActivity]: Button: setFrameRate fps:60 04-14 16:18:02.866 888 888 E Layer : [lzl-test] frameRate = {2, 60.00fps, OnlySeamless}

Zone 3: Use preferredDisplayModeId to test, select the frame rate, and click OK

//check surfaceflinger log 04-14 16:17:24.792 13261 13261 D [lzl-test][MainActivity]: Spinner: onItemSelected... 04-14 16:17:24.792 13261 13261 D [lzl-test][MainActivity]: Spinner: preferredDisplayModeId fps:90 04-14 16:17:25.364 13261 13261 D [lzl-test][MainActivity]: Button: preferredDisplayModeId fps:90.0 04-14 16:17:25.379 888 888 E Layer : [lzl-test]frameRate = {