Introduction
LM video intermediate audio intercom and broadcast support functions such as one-to-one, one-to-many, many-to-many, controlling the front-end to play custom wav voice files and creating voice chat rooms. It can realize real-time multi-party voice calls and voice calls based on basic security equipment. Voice interaction in scenarios such as shouting and alarm-linked voice playback.
The devices supported by LM video middleware are as follows:
- Full range of Hikvision equipment
- Full range of Dahua equipment
- GB28181 device (internal network UDP/TCP, external network TCP mode)
- EHOME protocol equipment, including ISUP
- LM’s main open interfaces include turning on intercom (broadcast), turning off intercom (broadcast), sending voice and sending wav file playback, etc. For detailed API description, please see the API documentation.
Tips
Requires a microphone and speaker installed on the device
API interface description
Enable intercom (broadcast)
This interface is to start or join the intercom (group), and the request parameters are
- When deviceId can be empty, initiate a new (join) intercom group for the user. This request does not add new devices to the intercom group.
- When the ssid is 0, it means creating a new intercom (broadcast) group. When the ssid is not 0, it means joining an existing intercom group. The ssid intercom group must exist, otherwise an error that the intercom group does not exist will be returned.
Note:
- After the user creates an intercom group, if the interface for sending voice or playing wav files is not called within a period of time (15 seconds), LM will automatically release the current intercom group and close the intercom session with the front-end device.
- If the intercom device is occupied by an intercom group, LM will return a User Already Opening error in the interface. The caller can obtain the intercom group SSID of the occupied device in the array details returned by the interface, and then initiate a request to join the intercom group. Make voice communications.
Request
POST /api/v1/talk/start?token=f384932b-92b9-4947-a567-10c33a82b44f HTTP/1.1
Content-Type: application/json
Accept: */*
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 63
{
"ssid":0,
"deviceId":[
"A20221214135512",
"A123445"
]
}
Answer
The returned information needs to be obtained from the detail array to obtain the activation result of each device
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 177
Connection: keep-alive
Access-Control-Allow-Origin: *
{
"result": 200,
"message": "OK",
"ssid": 1,
"detail": [
{
"deviceId": "A20221214135512",
"result": "OK"
},
{
"deviceId": "A123445",
"result": "Invalid Device ID"
}
]
}
Close intercom (broadcast)
This interface is to close the intercom group interface. You do not need to call this interface when using it. LM will also automatically detect whether there are still users in the intercom group. The interface parameters
- When deviceId is empty, close the intercom group specified by parameter ssid
- ssId is non-0 and this intercom group must exist in LM
Request
POST /api/v1/talk/stop?token=8305918c-69e8-4280-ae22-85aa5d696e0e HTTP/1.1
Content-Type: application/json
Accept: */*
Host: 192.168.3.23:9030
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 20
{
"ssid":636
}
!!! sample “response”
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 42
Connection: keep-alive
Access-Control-Allow-Origin: *
{
"result": 200,
"message": "OK"
}
Send voice
This interface is returned after a successful call to open the intercom interface. The client uses websocket to send local and collected PCM voice data. The voice sampling rate is: 44000, the sampling bit is 16, single channel.
Play wav files
This interface only supports PCM voice files in WAV format
Web page H5 audio collection
Web page audio uses the getUserMedia API of HTML5. The getUserMedia API provides users with an interface to access hardware device media (camera, video, audio, geographical location, etc.). Based on this interface, developers can do this without relying on any browser plug-ins. to access hardware media devices. The browser compatibility line for the getUserMedia API is as shown below:
Note
If the web page H5 needs to support audio collection, there are two ways (take the Chrome browser as an example):
- Enable https on the server side
- Browse local settings exception, enter browser local settings
- Enter chrome://flags/ in the address bar
- Find the option Insecure origins treated as secure on the settings page, add the exception server address, and restart the browser to collect audio in the development environment.
Collect audio
initTalk(){//Initialize multimedia objects
if (!navigator.mediaDevices.getUserMedia) {
alert('The browser does not support audio input');
} else if(this.record == null){
navigator.mediaDevices.getUserMedia = navigator.mediaDevices.getUserMedia || navigator.mediaDevices.webkitGetUserMedia;
navigator.mediaDevices.getUserMedia({audio: true})
.then((mediaStream)=> {
this.record = new this.getRecorder(mediaStream)
})
}
}
getRecorder(stream){
let sampleBits = 16;//Output sample bits 8, 16
let sampleRate = 44100;//Output sampling rate
let bufSize = 8192;
let context = new AudioContext();
let audioInput = context.createMediaStreamSource(stream);
let recorder = context.createScriptProcessor(0, 1, 1);
let audio_resample = new Resampler(context.sampleRate, sampleRate, 1, bufSize);
console.log(audio_resample);
let audioData = {
size: 0, //recording file length
buffer: [], //recording cache
inputSampleRate: sampleRate, //Input sampling rate
inputSampleBits: 16, //Input sample bits 8, 16
outputSampleRate: sampleRate,
outputSampleBits: sampleBits,
clear: function () {
audioData.buffer = [];
audioData.size = 0;
},
input: function (data) {
audioData.buffer.push(new Float32Array(data));
audioData.size + = data.length;
},
compress: function () { //Combined compression
//merge
let data = new Float32Array(this.size);
let offset = 0;
for (let i = 0; i <this.buffer.length; i + + ) {
data.set(this.buffer[i], offset);
offset + = this.buffer[i].length;
}
//compression
let compression = parseInt(this.inputSampleRate / this.outputSampleRate);
let length = data.length / compression;
let result = new Float32Array(length);
let index = 0, j = 0;
while (index < length) {
result[index] = data[j];
j + = compression;
index + + ;
}
return result;
},
encodePCM: function () {//The collected data will not be processed in other formats here, and will be handed over to the server for processing if necessary.
let sampleRate = Math.min(this.inputSampleRate, this.outputSampleRate);
let sampleBits = Math.min(this.inputSampleBits, this.oututSampleBits);
let bytes = this.compress();
let dataLength = bytes.length * (sampleBits / 8);
let buffer = new ArrayBuffer(dataLength);
let data = new DataView(buffer);
let offset = 0;
for (let i = 0; i < bytes.length; i + + , offset + = 2) {
let s = Math.max(-1, Math.min(1, bytes[i]));
data.setInt16(offset, s < 0 ? s * 0x8000 : s * 0x7FFF, true);
}
return new Blob([data]);
}
};
this.start = function () {
audioInput.connect(recorder);
recorder.connect(context.destination);
}
this.stop = function () {
recorder.disconnect();
}
this.getBlob = function () {
return audioData.encodePCM();
}
this.clear = function () {
audioData.clear();
}
recorder.onaudioprocess = function (e) {
//audioData.input(e.inputBuffer.getChannelData(0));
audioData.input(audio_resample.resample(e.inputBuffer.getChannelData(0)));
}
}
getTalkStart(){
this.initTalk()
if(this.talkOpen){ // this.talkOpen is true to enable intercom
let data = {
deviceId: [this.orgId],
ssid: 0
};
talkStartServe(data,this.getToken()).then(res => {
if(res.result==200 & amp; & amp; res.ssid != 0) {
this.ssid = res.ssid
this.imgSrc = require('@/assets/img/yj-open.png')
this.$message.success('Device intercom is on')
this.talkOpen = false
this.sendTalk()
}else {
if(res.detail[0] != undefined){
if(res.detail[0].result == "User Opened Talking"){
this.ssid = res.detail[0].ssid
this.imgSrc = require('@/assets/img/yj-open.png')
this.$message.success('Device intercom is on')
this.talkOpen = false
this.sendTalk()
}
else{
this.$message.error(res.detail[0].result)
this.imgSrc = require('@/assets/img/yunjing.png')
this.talkOpen = true
}
}
else{
this.$message.error("Intercom opening failed");
this.imgSrc = require('@/assets/img/yunjing.png')
this.talkOpen = true
}
}
})
}else { // Close intercom
if(this.record != null){
this.record.stop();
}
if(this.socket != null){
this.socket.close();
}
if(this.pcmPlayer != null){
this.pcmPlayer.destroy();
this.pcmPlayer = null;
}
//No need to close, the background will automatically close the intercom group based on the connection
this.$message.success('Device intercom has been closed')
this.imgSrc = require('@/assets/img/yunjing.png')
this.talkOpen = true
/*talkStopServe({ssid: this.ssid},this.getToken()).then(res => {
if(res.result==200) {
this.$message.success('Device intercom has been closed')
this.imgSrc = require('@/assets/img/yunjing.png')
this.talkOpen = true
}else {
this.$message.error('Operation failed')
this.imgSrc = require('@/assets/img/yj-open.png')
this.talkOpen = false
}
})*/
}
}
Send audio
sendTalk() {// WebSocket sends audio
let that = this // there will be problems with this pointing to that instead
let locUrl = window.location.host
let stemp = '';
if(window.location.protocol === 'https:')
{
stemp ='s';
}
const socketUrl = 'ws' + stemp + '://' + locUrl + '/ws_talk/' + this.ssid + '?token=' + this.getToken();// + ' &sampleRate=' + this.context.sampleRate;
that.socket = new WebSocket(socketUrl)
that.socket.binaryType = 'arraybuffer'
that.socket.onopen = function () {
console.log('Browser WebSocket is opened');
if(that.record != null){
that.record.start();
}
else
{
console.log('Sound collection failed to open');
}
window.timeInte = setInterval(() => {
if(that.socket != null & amp; & amp; that.socket.readyState == 1) {
if (that.record != null & amp; & amp; that.record.getBlob().size != 0) {
that.socket.send(that.record.getBlob()); //Send audio data
that.record.clear();
}
}
},20)
};
if(that.pcmPlayer == null)
{
that.pcmPlayer = new PCMPlayer({
inputCodec: 'Int16',
channels: 1,
sampleRate: 16000,
flushTime: 200
})
}
that.socket.onmessage = function(msg) {//Receive WebSocket messages
that.pcmPlayer.feed(msg.data)
}
}
Common error codes
Communication contact:
Hangzhou Houhang Technology Co., Ltd. http://houhangkeji.com/
QQ technical exchange group: 698793654