Foreword
When the author was developing a heavy-duty SLG project, because the large map in the project used real-time lighting, Lightmap, fog effects, etc., there were many lighting-related Shader variants. It took several hours to compile the Shader variants during packaging. , and found that the mobile platform occupies a large amount of memory during operation. The Shader resource alone occupies more than 1G of memory, so Shader variant tailoring is imperative;
Essential knowledge
Relevant knowledge about Shader variants can be found in the official manual. The author also read the official manual to research solutions. Generally, I can find this document and I also have an understanding of Shader variants. The link is attached: https://docs.unity3d.com /Manual/shader-variants-and-keywords.html
It explains the relevant principles of Keywords and how Unity handles keywords when packaging;
Idea analysis
The core idea of Shader variant tailoring is to compile only the variants we need when packaging. How to determine which variants we need? We know that the unity editor will automatically record the variants currently used in the game, as shown in the figure:
Therefore, we first run the relevant scenes of the game, let Unity collect relevant variants, and then Save to Asset to obtain the currently collected variants, which is the all file in the above picture;
Then, when packaging and compiling Shader, you can compile the required Shader variants based on the above all files. Unity provides an interface that allows us to customize Shader variant tailoring:
In this interface, we compile the variants we need based on the variants recorded in the all file;
Implementation details:
1. First of all, when the editor and the real machine run the game, the required variants are not necessarily the same, but fortunately, the author only found one, which is:
Be careful not to enable this option in the project. If it is enabled, it will only be effective on the editor and will not be effective on the mobile platform (the author’s project at that time was like this), causing the variants collected by the editor to be different from those required by the mobile platform. body does not match;
We do not rule out other situations. If the mobile platform loses the Shader, use FrameDebug to view the corresponding Keywords under the editor and the mobile platform respectively, and find out the difference analysis to generally find the problem:
2. If you enable variant clipping for all Shaders in the project, it will be very troublesome. You have to collect all the Shader variants. This is an impossible task. For example, some special effects, who knows when they will be triggered, so , the author only turns on variant clipping for Shaders that contain a large number of Keywords, and does not turn on clipping for other Shaders, which are fully compiled by default, so that we only need to collect Shader variations for specific situations;
3. Since projects generally have different qualities, lighting and shadow calculations are generally different under different qualities, so there are different variants; therefore, when we run the game, we have to set each quality in each case to trigger Unity’s collection of variants;
4. Collecting Shader variants for each situation may require writing editor code to support it. For scenes, we only need to run there, but there may be various models on the scene, and the models are only loaded when needed. In order to facilitate the collection of models Shader variants, you can write editor code and manually instantiate all models into the scene, so that we can collect Shader variants of all models;
5. Other situations:
It is necessary to determine whether the code enables certain keywords under certain special circumstances to prevent variants from not being collected;
Pay attention to whether the declaration of Shader keywords is correct. When the author’s project previously upgraded the Unity engine, Unity made modifications to the keywords, resulting in the old Shader not being able to obtain the correct shader variant after compilation. In short, just use FrameDebug to debug;
Note that when building the AB package, if we do not have ForceRebuild, the packaging will use the cache. Once we need to collect the missing variants, building the AB package will not trigger the compilation of the Shader, because the Shader resources have not changed, it is just our processing. The process has changed. Generally, for projects that are officially launched, ForceRebuild must be turned on when starting AB, otherwise the resources may not be rebuilt due to caching;
Results:
The author’s project is to run the game on the Windows platform to collect relevant variants, which can be directly applied to mobile platforms, such as Android and iOS. It is equivalent to just collecting it once under the Windows platform, which is still very stable.
Usage:
The author will publish the source code at the end. You can directly download the source code and import it into the project. There will be the following tools:
Save ShaderVariants, saves the variants currently collected by the game, equivalent to:
Merge ShaderVariants, merge Shader variants:
When we have newly collected Shader variants, we need to add the newly collected sum to all:
Executing Merge ShaderVariants will help us merge;
Extract the variants that need to be preheated and save them in the Resources folder:
Sometimes we need to preheat certain Shader variants, we can use this tool;
From all the Shader variants we collected, extract the variant of the Shader we specified;
Additional code
#define SAVE_SHADER_INFO
using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using System.Reflection;
using System.Text;
using UnityEditor;
using UnityEditor.Build;
using UnityEditor.Rendering;
using UnityEngine;
using UnityEngine.Rendering;
using static UnityEngine.ShaderVariantCollection;
public class ShaderVariantsStrip : IPreprocessShaders
{
public const string ShaderVariantsCollectPath = "Assets/ShaderVariantsForStrip/all.shadervariants";
//These shaders with more variants only retain the variants collected when running the game under the editor.
public static HashSet<string> stripShaderNames = new HashSet<string> {
"FMGame/Lit",
"FMGame/Simple Lit",
"FMGame/Cloud",
"FMGame/Environment/Lit_Pond",
"FMGame/Versatile Blend",
"FMGame/Simple Lit HAlphaBlend",
"FMGame/FMLit_Scanning",
"FMGame/VFX/Unlit_Hologram",
"FMGame/GpuInstancesLit",
"FMGame/GpuInstancesSimpleLit",
"FMGame/Simple Lit GPUInstancing",
"FMGame/Lit GPUInstancing",
"FMGame/Lit_NoiseFlow",
"FMGame/VFX/Lit_BuildingScan",
"FMGame/Lit GPUInstancing Snow",
"FMGame/Simple Lit GPUInstancing Snow",
"FMGame/Lit_Snow",
"FMGame/Lit_Wind",
"FMGame/Lit_Outline",
"FMGame/SimpleLit_Snow",
"FMGame/Simple Lit Wind",
"FMGame/Environment/Lit_Pond_Snow",
"GPUSkinning/GPUSkinning_Outline",
"GPUSkinning/GPUSkinning_Lit",
"GPUSkinning/GPUSkinning_Lit_Skin4",
"GPUSkinning/GPUSkinning_SimpleLit",
"GPUSkinning/GPUSkinning_SimpleLit_Skin4",
"GPUSkinning/GPUSkinning_Sample_Unlit",
"GPUSkinning/GPUSkinning_Unlit",
"GPUSkinning/GPUSkinning_Unlit_Skin4",
"GPUSkinning/GPUSkinningUnlitSkin4_Outline",
"GPUSkinning/GPUSkinningUnlitSkin4_Shadow",
"MTE/URP/3 Textures2",
"MTE/URP/4 Textures",
"MTE/URP/5 Textures",
"MTE/URP/TextureArray2Tex_2Normal",
"MTE/URP/TextureArray3Tex_3Normal",
"MTE/URP/TextureArray4Tex_4Normal_Simplified",
"MTE/URP/TextureArray4Tex_4Normal",
"MTE/URP/TextureArray5Tex_5Normal",
"MTE/URP/TextureArray5Tex_6Normal",
"MTE/URP/TextureArray2Tex_NoNormal",
"MTE/URP/TextureArray3Tex_NoNormal",
"MTE/URP/TextureArray4Tex_NoNormal",
"MTE/URP/TextureArray4Tex_NoNormal_Simplified",
"MTE/URP/TextureArray5Tex_NoNormal",
"MTE/URP/TextureArray6Tex_NoNormal",
"MTE/URP/TextureArray8Tex_NoNormal",
"MTE/URP/TextureArray8Tex_8Normal",
"MTE/URP/TextureArray8Tex_8Normal_Simplified",
"MTE/URP/TextureArray8Tex_NoNormal_Simplified",
"Universal Render Pipeline/Lit",
"Universal Render Pipeline/Complex Lit",
"Universal Render Pipeline/Simple Lit",
"Shader Graphs/PhysicalMaterial3DsMax",
"efronli/UV",
"efronli/UV_b",
"efronli/NQ_whater",
"efronli/UV_Alpha Blended",
"Effect_Mid/Additive",
"Effect_Mid/Alpha Blend",
"Legacy Shaders/Diffuse",
"VFX/ComDissovle",
"JYi/round_VertexColorContrel",
"Kero/Alpha-Blended_HDR_URP",
"Hidden/TerrainEngine/Details/UniversalPipeline/BillboardWavingDoublePass",
"Hidden/TerrainEngine/Details/UniversalPipeline/WavingDoublePass",
"Hidden/TerrainEngine/Details/UniversalPipeline/Vertexlit",
"Hidden/Internal-PrePassLighting",
"Hidden/Internal-DeferredShading",
"Hidden/Universal Render Pipeline/UberPost",
"Hidden/Universal Render Pipeline/StencilDeferred",
};
#region for Build
private Dictionary<Shader, Dictionary<PassType, List<HashSet<string>> > > m_shader2keywords = new Dictionary<Shader, Dictionary<PassType, List<HashSet<string>> > >();
private static ShaderVariantsStrip m_ins = null;
public ShaderVariantsStrip()
{
m_ins = this;
ShaderVariantCollection shaderVariantCollection = AssetDatabase.LoadAssetAtPath<ShaderVariantCollection>(ShaderVariantsCollectPath);
if (shaderVariantCollection == null)
return;
SerializedObject serializedObject = new SerializedObject(shaderVariantCollection);
SerializedProperty m_Shaders = serializedObject.FindProperty("m_Shaders");
for (var i = 0; i < m_Shaders.arraySize; + + i)
{
var entryProp = m_Shaders.GetArrayElementAtIndex(i);
Shader shader = (Shader)entryProp.FindPropertyRelative("first").objectReferenceValue;
if (shader != null)
{
if(!m_shader2keywords.TryGetValue(shader, out Dictionary<PassType, List<HashSet<string>> >passType2keywords))
{
passType2keywords = new Dictionary<PassType, List<HashSet<string>> >();
m_shader2keywords.Add(shader, passType2keywords);
}
var variantsProp = entryProp.FindPropertyRelative("second.variants");
for (var j = 0; j < variantsProp.arraySize; + + j)
{
var prop = variantsProp.GetArrayElementAtIndex(j);
var keywordsStr = prop.FindPropertyRelative("keywords").stringValue;
keywordsStr.Replace("\r\
", string.Empty);
string[] keywordArray = keywordsStr.Split(new char[] { ' ' }, StringSplitOptions.RemoveEmptyEntries);
var passType = (PassType)prop.FindPropertyRelative("passType").intValue;
if(!passType2keywords.TryGetValue(passType, out List<HashSet<string>> keywordSetList))
{
keywordSetList = new List<HashSet<string>>();
passType2keywords.Add(passType, keywordSetList);
}
HashSet<string> keywordSet = new HashSet<string>();
foreach (var item in keywordArray)
{
keywordSet.Add(item);
}
keywordSetList.Add(keywordSet);
}
}
}
}
private StringBuilder stringBuilder = new StringBuilder();
private Dictionary<string, List<string>> shader2Varaints = new Dictionary<string, List<string>>();
public static void SaveBuildInfo(bool IsBuildAssetBundle)
{
if (m_ins != null)
{
foreach (var pair in m_ins.shader2Varaints)
{
foreach (var item in pair.Value)
{
m_ins.stringBuilder.AppendLine(pair.Key + ": " + item);
}
}
if (IsBuildAssetBundle)
File.WriteAllText("Assets/BuildShaderInfo_AssetBundle.txt", m_ins.stringBuilder.ToString());
else
File.WriteAllText("Assets/BuildShaderInfo_Resources.txt", m_ins.stringBuilder.ToString());
}
}
public int callbackOrder => -1000;
Dictionary<ShaderCompilerPlatform, List<HashSet<string>>> m_Platform2ValidKeywordSetList = new Dictionary<ShaderCompilerPlatform, List<HashSet<string>>>();
List<string> m_currentkeywords = new List<string>();
HashSet<ShaderCompilerPlatform> m_platforms = new HashSet<ShaderCompilerPlatform>();
List<HashSet<string>> m_CachHashSets = new List<HashSet<string>>();
private HashSet<string> GetHashSet()
{
HashSet<string> ret;
int count = m_CachHashSets.Count;
if (count > 0)
{
ret = m_CachHashSets[count - 1];
m_CachHashSets.RemoveAt(count - 1);
}
else
{
ret = new HashSet<string>();
}
return ret;
}
private void ReleaseHashSet(List<HashSet<string>> hashSetList)
{
foreach (var hashSet in hashSetList)
{
hashSet.Clear();
m_CachHashSets.Add(hashSet);
}
hashSetList.Clear();
}
/* test code
[MenuItem("Build/Build Shader")]
static void BuildShader()
{
string path = "AssetBundles";
if(Directory.Exists(path))
Directory.Delete(path, true);
Directory.CreateDirectory(path);
BuildPipeline.BuildAssetBundles(path, new AssetBundleBuild[] {
new AssetBundleBuild()
{
assetNames = new string[]{ "Assets/MyShader/TestShader.shader"},
assetBundleName = "test"
}
}, BuildAssetBundleOptions.ForceRebuildAssetBundle, EditorUserBuildSettings.activeBuildTarget);
}
*/
public void OnProcessShader(Shader shader, ShaderSnippetData snippet, IList<ShaderCompilerData> data)
{
m_platforms.Clear();
foreach (var item in data)
{
if(!m_platforms.Contains(item.shaderCompilerPlatform))
{
m_platforms.Add(item.shaderCompilerPlatform);
}
}
m_Platform2ValidKeywordSetList.Clear();
foreach (var platform in m_platforms)
{
if(m_Platform2ValidKeywordSetList.TryGetValue(platform, out List<HashSet<string>> m_ValidKeywordSetList))
{
ReleaseHashSet(m_ValidKeywordSetList);
}
else
{
m_ValidKeywordSetList = new List<HashSet<string>>();
m_Platform2ValidKeywordSetList.Add(platform, m_ValidKeywordSetList);
}
if (m_shader2keywords.TryGetValue(shader, out Dictionary<PassType, List<HashSet<string>>> passType2keywords))
{
if (passType2keywords.TryGetValue(snippet.passType, out List<HashSet<string>> keywordSetList))
{
foreach (var keywordSet in keywordSetList)
{
HashSet<string> newSet = GetHashSet();
foreach (var keyword in keywordSet)
{
LocalKeyword localKeyword = new LocalKeyword(shader, keyword);
if (ShaderUtil.PassHasKeyword(shader, snippet.pass, localKeyword, snippet.shaderType, platform))
newSet.Add(keyword);
}
m_ValidKeywordSetList.Add(newSet);
}
}
}
}
bool needStrip = checkNeedStripShader(shader.name);
for (int i = data.Count - 1; i >= 0; --i)
{
ShaderKeywordSet shaderKeywordSet = data[i].shaderKeywordSet;
ShaderKeyword[] shaderKeywords = shaderKeywordSet.GetShaderKeywords();
m_currentkeywords.Clear();
if (shaderKeywords.Length != 0)
{
for (int j = 0; j < shaderKeywords.Length; j + + )
{
string key = shaderKeywords[j].name;
m_currentkeywords.Add(key);
}
}
bool need = false;
if(needStrip)
{
bool needflag = false;
List<HashSet<string>> m_ValidKeywordSetList = m_Platform2ValidKeywordSetList[data[i].shaderCompilerPlatform];
foreach (var set in m_ValidKeywordSetList)
{
if(m_currentkeywords.Count == set.Count)
{
bool flag = true;
foreach (var keyword in m_currentkeywords)
{
if(!set.Contains(keyword))
{
flag = false;
break;
}
}
if(flag)
{
needflag = true;
break;
}
}
}
need = needflag;
}
else
{
need = true;
}
if (!need)
data.RemoveAt(i);
#if SAVE_SHADER_INFO
if(need)
{
string str1 = snippet.shaderType + ", " + snippet.passType + ", " + snippet.passName + ": ";
string str2 = string.Empty;
foreach (var item in m_currentkeywords)
{
str2 + = item + ", ";
}
List<string> list;
if(!shader2Varaints.TryGetValue(shader.name, out list))
{
list = new List<string>();
shader2Varaints[shader.name] = list;
}
list.Add(str1 + str2);
}
#endif
}
}
private bool checkNeedStripShader(string shaderName)
{
if (stripShaderNames.Contains(shaderName))
return true;
return false;
}
#endregion
#regionforTools
[MenuItem("ShaderVariants/Save ShaderVariants")]
static void SaveShaderVariants()
{
string dir = Path.GetDirectoryName(ShaderVariantsCollectPath);
if (!Directory.Exists(dir))
Directory.CreateDirectory(dir);
string message = "Save shader variant collection";
string assetPath = EditorUtility.SaveFilePanelInProject("Save Shader Variant Collection", "NewShaderVariants", "shadervariants", message, dir);
if (!string.IsNullOrEmpty(assetPath))
{
Type type = typeof(ShaderUtil);
MethodInfo methodInfo = type.GetMethod("SaveCurrentShaderVariantCollection", BindingFlags.Static | BindingFlags.NonPublic);
methodInfo.Invoke(null, new object[] { assetPath });
}
}
/// <summary>
/// Merge the collected shader variants
/// </summary>
[MenuItem("ShaderVariants/Merge ShaderVariants")]
static void MergeShaderVariants()
{
string rootPath = "Assets/ShaderVariantsForStrip";
string allName = "all.shadervariants";
string allPath = rootPath + "/" + allName;
ShaderVariant shaderVariant = new ShaderVariant();
ShaderVariantCollection allCollection = new ShaderVariantCollection();
List<string> keywordList = new List<string>();
string[] fileNames = Directory.GetFiles(rootPath, "*.shadervariants");
foreach (var fileName in fileNames)
{
ShaderVariantCollection curCollection = AssetDatabase.LoadAssetAtPath<ShaderVariantCollection>(fileName);
SerializedObject serializedObject = new SerializedObject(curCollection);
SerializedProperty m_Shaders = serializedObject.FindProperty("m_Shaders");
for (var i = 0; i < m_Shaders.arraySize; + + i)
{
var entryProp = m_Shaders.GetArrayElementAtIndex(i);
Shader shader = (Shader)entryProp.FindPropertyRelative("first").objectReferenceValue;
if (shader != null)
{
shaderVariant.shader = shader;
var variantsProp = entryProp.FindPropertyRelative("second.variants");
for (var j = 0; j < variantsProp.arraySize; + + j)
{
var prop = variantsProp.GetArrayElementAtIndex(j);
var keywordsStr = prop.FindPropertyRelative("keywords").stringValue;
keywordsStr.Replace("\
", string.Empty);
keywordsStr.Replace("\r\
", string.Empty);
string[] keywordArray = keywordsStr.Split(new char[] { ' ' }, StringSplitOptions.RemoveEmptyEntries);
shaderVariant.keywords = keywordArray;
var passType = (UnityEngine.Rendering.PassType)prop.FindPropertyRelative("passType").intValue;
shaderVariant.passType = passType;
if (!allCollection.Contains(shaderVariant))
allCollection.Add(shaderVariant);
//Automatically add non-INSTANCING_ON variants
keywordList.Clear();
keywordList.AddRange(keywordArray);
if (keywordList.Remove("INSTANCING_ON"))
{
string[] keywords = keywordList.ToArray();
shaderVariant.keywords = keywords;
if (!allCollection.Contains(shaderVariant))
allCollection.Add(shaderVariant);
}
}
}
}
}
AssetDatabase.CreateAsset(allCollection, allPath);
AssetDatabase.Refresh(ImportAssetOptions.ForceSynchronousImport);
}
static HashSet<string> NeedWarmShaders = new HashSet<string>
{
"FMGame/Lit",
"FMGame/Simple Lit",
"FMGame/Cloud",
"FMGame/Environment/Lit_Pond",
"FMGame/Versatile Blend",
"FMGame/Simple Lit HAlphaBlend",
"FMGame/FMLit_Scanning",
"FMGame/VFX/Unlit_Hologram",
"FMGame/GpuInstancesLit",
"FMGame/GpuInstancesSimpleLit",
"FMGame/Simple Lit GPUInstancing",
"FMGame/Lit GPUInstancing",
"FMGame/Lit_NoiseFlow",
"FMGame/VFX/Lit_BuildingScan",
"FMGame/Lit GPUInstancing Snow",
"FMGame/Simple Lit GPUInstancing Snow",
"FMGame/Lit_Snow",
"FMGame/Lit_Wind",
"FMGame/Lit_Outline",
"FMGame/SimpleLit_Snow",
"FMGame/Simple Lit Wind",
"FMGame/Environment/Lit_Pond_Snow",
"MTE/URP/3 Textures2",
"MTE/URP/4 Textures",
"MTE/URP/5 Textures",
"MTE/URP/TextureArray2Tex_2Normal",
"MTE/URP/TextureArray3Tex_3Normal",
"MTE/URP/TextureArray4Tex_4Normal_Simplified",
"MTE/URP/TextureArray4Tex_4Normal",
"MTE/URP/TextureArray5Tex_5Normal",
"MTE/URP/TextureArray5Tex_6Normal",
"MTE/URP/TextureArray2Tex_NoNormal",
"MTE/URP/TextureArray3Tex_NoNormal",
"MTE/URP/TextureArray4Tex_NoNormal",
"MTE/URP/TextureArray4Tex_NoNormal_Simplified",
"MTE/URP/TextureArray5Tex_NoNormal",
"MTE/URP/TextureArray6Tex_NoNormal",
"MTE/URP/TextureArray8Tex_NoNormal",
"MTE/URP/TextureArray8Tex_8Normal",
"MTE/URP/TextureArray8Tex_8Normal_Simplified",
"MTE/URP/TextureArray8Tex_NoNormal_Simplified",
"Hidden/Universal Render Pipeline/UberPost",
};
/// <summary>
/// Only keep the variants that need to be cropped
/// </summary>
[MenuItem("ShaderVariants/Extract the variants that need to be preheated and save them in the Resources folder")]
static void FilterStripShaderVariants()
{
string allPath = ShaderVariantsStrip.ShaderVariantsCollectPath;
string resourcePath = "Assets/Resources/ShaderVariants/all.shadervariants";
if (!AssetDatabase.CopyAsset(allPath, resourcePath))
{
Debug.LogError($"Copying {allPath} to {resoursePath} failed");
return;
}
AssetDatabase.Refresh();
ShaderVariantCollection allCollection = AssetDatabase.LoadAssetAtPath<ShaderVariantCollection>(resoursePath);
if (allCollection == null)
{
return;
}
SerializedObject serializedObject = new SerializedObject(allCollection);
SerializedProperty m_Shaders = serializedObject.FindProperty("m_Shaders");
for (var i = m_Shaders.arraySize - 1; i >= 0; --i)
{
var entryProp = m_Shaders.GetArrayElementAtIndex(i);
var firstProperty = entryProp.FindPropertyRelative("first");
Shader shader = (Shader)firstProperty.objectReferenceValue;
if (shader != null)
{
if (!NeedWarmShaders.Contains(shader.name))
m_Shaders.DeleteArrayElementAtIndex(i);
}
else
{
m_Shaders.DeleteArrayElementAtIndex(i);
}
}
serializedObject.ApplyModifiedProperties();
EditorUtility.SetDirty(allCollection);
AssetDatabase.SaveAssets();
}
#endregion
}