Background:
To realize audio recording, audio upload and playback on the mobile terminal, use the recording API provided by the native app (native h5 getUserMedia, which is limited by the app’s kernel browser and does not support the getUserMedia method, and the popular plug-ins js-audio-record and record-core Debugging is supported on PC but not on mobile.
The API interface provided by the app is similar to the interface provided by ByteDance, but it is unknown whether the specific data format is consistent.
Bytedance recording onfframeRecord related api call link 
https://developer.open-douyin.com/docs/resource/zh-CN /mini-game/develop/api/media/record/recorder-manager/recorder-manager-on-frame-recorded
Encountered a problem: Unlike conventional plug-ins that directly obtain MP3 and WAV audio files, onframeRecord obtains binary streaming pcm audio, which is different from the audio obtained by getUserMedia
Question 1: Audio cannot be played directly like MP3 or wav files.
Problem 2: The data stream obtained in real time cannot be uploaded directly using blob objects like non-streaming files.
Try solutions:
1. Existing methods on the Internet to convert pcm to wav format. Use the method to convert file types for easy uploading and playback
Final result: attempt failed;
Reason: The audio stream obtained by the onframeRecord api encapsulated by the app is different from the audio stream obtained by the native getUserMedia. The wav file header is added directly, and it is white noise after playing.
２.Introduce pcmPlayer player to play real-time audio stream
Final result: The real-time audio stream is successfully played, but it cannot be solved after uploading the data and getting the audio stream to play again.
3. Compress the real-time audio stream using jsZip and upload it as a zip file to the backend, and then obtain the file for decompression and playback
Final result: Failure, decompression failed
4. Directly push the real-time audio stream into a variable without performing background upload and then pass it to the interface as a parameter, and obtain it from the interface when retrieving
End result: failure
Reason: Unfeasible: 1. The audio file is too large. 2. The file stream is stored in an array and passed to the background. The background cannot recognize it, and the return value is empty if the save fails.
5. Use unit8Array to visualize the audio stream of arrayBuffer type, and then convert it into a string and pass it to the background as a parameter according to method 4.
End result: failure
Reason: After getting the string from the background, it needs to be converted to unit8array and then converted from unit8Array to buffer. The whole process is cumbersome, the conversion has garbled characters, and the playback fails. There is also the problem of too large parameter memory
6. After various attempts, I thought of receiving all audio streams through variable data and then converting them into txt files and uploading them to the background through blobs. Then, through the download interface, obtain the txt file through the obtained url and read it. The file is returned to the data stream and then played again using pcmPlayer.
Final result: Success, which not only solves the problem of uploading large files, but also solves the problem of playing after uploading.
Part of the development code:
<button
@touchstart.prevent="handelLongPress"
@touchend.prevent="touchend"
></button>
//Long press to record, release to stop
//Long press to start recording
const handelLongPress = () => {
clearTimeout(loop.value);
longTouch.value = false;
loop.value = setTimeout(() => {
longTouch.value = true;
handRecordAudio();
}, 300);
};
//recording
const handRecordAudio = () => {
//Recording monitoring
eventInfo.value = toongine.recorder.onFrameRecorded({
callback: (res) => {
if (res.data.isLastFrame) {
console.log(
"toongine::recorder::onFrameRecorded::ended",
"Callback data acquisition ends",
);
if (recordedChunks.value.length) {
//Upload as txt file
const txtBlob = new Blob(recordedChunks.value, {
type: "text/plain",
});
let fileOfBlob = new File(
[txtBlob],
`Recording${new Date().getTime()}` + ".txt"
);
const formData = new FormData();
formData.append("multipartFiles", fileOfBlob);
//Upload interface
uploadFile(formData).then((res) => {
if (res.code == 200) {
recordedChunks.value = [];
audioFileList.value.push(res.data[0]);
} else {
showToast("Voice upload failed");
}
});
return;
}
}
// To operate data, you need to use typed array view or DataView
// var pcm = new Int16Array(res.data.frameBuffer);
let pcmData = res.data.frameBuffer;
let list = [...recordedChunks.value];
list.push(pcmData);
recordedChunks.value = list;
},
});
//Recording starts
toongine.recorder.start({
params: {
sampleRate: 32000,
numberOfChannels: 2,
frameSize: 1,
bitsPerChannel: 16,
format: "PCM",
},
callback: (res) => {
console.log("Recording 1 starts");
},
});
};
//Long press to record and release
const touchend = () => {
clearTimeout(loop.value); // Clear the timer to prevent repeated registration of timers
if (!longTouch.value) {
//If it is not a long press, execute the click event
console.log("click");
} else {
toongine.recorder.stop({
callback: (res) => {
//Remove recording event monitoring
toongine.removeEventListener(eventInfo.value);
console.log("recordedChunks.value", recordedChunks.value);
if (!recordedChunks.value.length) {
showToast("Recording failed, the audio file was obtained");
}
},
});
}
};
//recording playback
const playRecord = (item) => {
console.log("play", item);
if (item.fileUrl) {
axios
.get("/api/disposal/oss/file/download", {
params: {
fileUrl: item.fileUrl,
},
responseType: "blob", //Define the return data format as Blob
})
.then((res) => {
console.log("return value blob", res, res.data);
let reader = new FileReader();
reader.onload = function (e) { //Read the obtained txt file
// let zipFile = e.target.result[0];
console.log("e", e.target.result);
pcmPlay.value.feed(e.target.result);
};
reader.readAsArrayBuffer(res.data);
});
} else {
showToast("Playback failed");
}
};
Note:
1.Introducing pcm-player player
2. When obtaining the txt stream file, set the corresponding type to blob, and then read the file stream for playback
3. The audio sample number, channel number and other information corresponding to the pcmPlayer player should be set consistent with the parameters of calling the recording start method to prevent abnormal sound playback