As the GB28181 Android client, real-time video and audio on-demand is a must-support function. For the historical video and audio playback function, if it is not supported, you can copy the video file from the device and play it again. However, some scenes cannot be copied, and it is still necessary for Android to support playback.
The playback of historical video and audio is very similar to the signaling of real-time video and audio on demand. Audio and video data are transmitted through RTP. Signaling playback needs to process SIP INFO messages, parse the MANSRTSP protocol, and implement fast forward, slow play, pause, stop, and position. Drag and other remote control commands.
The GB28181 document defines in detail the signaling process for historical video and audio playback. For implementation on Android devices, signaling related to the media stream sender needs to be implemented:
1: The SIP server sends an Invite request to the Android device. The request carries SDP information. The s field in SDP is “Playback” which represents historical playback. The u field represents the playback channel ID and playback type. The t field represents the playback time period. Add the y field. Describes the SSRC value, and the f field describes the media parameters.
2: After receiving the Invite request from the SIP server, the Android device replies with a 200 OK response (before replying with a final response such as 200 OK, it can also reply with a temporary response, such as 180 Ringing, etc.), carrying the SDP message body. The Android device is described in SDP. Send the IP, port, media format, SSRC field and other contents of the media stream.
3: After receiving the 200OK response returned by the Android device, the SIP server sends an ACK request to the Android device. The request does not carry the message body, completing the Invite session establishment process with the Android device.
4: The Android device sends the audio and video RTP package (PS RTP package recommended) to the media server according to the IP address, port and other information given in Invite SDP.
5: During the playback process, the player performs playback control by sending an intra-session Info + MANSRTSP message to the SIP server (which is then forwarded by the SIP server to the Android device), including video pause, play, fast play, slow play, and random drag and drop. Playback and other operations.
6: After the file playback ends, the Android device sends an intra-session Message to notify the SIP server that the playback has ended (for intra-session messages, please refer to RFC3261-12.2 Requests within a Dialog).
7: The SIP server performs corresponding processing after receiving the media notification message, and then the SIP server sends a BYE message to the Android terminal.
8: After receiving the BYE message, the Android device replies with a 200OK response, disconnects the session, and releases related resources.
To facilitate quick code implementation, specific examples of signaling are given below:
INVITE SDP received by Android device:
v=0
o=64010000041310000137 0 0 IN IP4 192.168.0.193
s=Playback
u=64010000041310000137:0
c=IN IP4 192.168.0.193
t=1698218951 1698219270
m=video 20072 RTP/AVP 96a=recvonly
a=rtpmap:96 PS/90000
y=1900000005
s=Playback represents historical playback, SSRC is: 1900000005 (SSRC bit 1 is the identification bit of historical or real-time media stream, 0 is real-time, 1 is historical).
The Android device responds with 200 OK carrying SDP:
v=0
o=64010000041310000137 0 0 IN IP4 192.168.0.151
s=MyAndroidPlaybackTest
c=IN IP4 192.168.0.151
t=0 0
m=video 36010 RTP/AVP 96
a=rtpmap:96 PS/90000
a=sendonly
y=1900000005
During playback, the normal playback speed is changed to 4 times fast forward playback through SIP INFO message + MANSRTSP protocol:
PLAY RTSP/1.0
CSeq: 110379
Scale: 4.000000
During playback, use SIP INFO message + MANSRTSP protocol to change the playback position to the position of 60.27 seconds (Considering that Android generally uses microseconds or nanoseconds, the conversion to microseconds is: 60270000):
PLAY RTSP/1.0
CSeq: 110392
Range:npt=60.27-
After the file playback ends, the Android device sends an in-session Message:
MediaStatus
1738772385
64010000041310000137
121
The notification event type is “121”, which indicates the end of sending historical media files.
Relevant implementation code:
/* * Copyright (C) [email protected]. All rights reserved. */ /** * Some signaling interfaces */ package com.gb.ntsignalling; public interface GBSIPAgent { void addPlaybackListener(GBSIPAgentPlaybackListener playbackListener); void removePlaybackListener(GBSIPAgentPlaybackListener playbackListener); /* *Response to Invite Playback 200 OK */ boolean respondPlaybackInviteOK(long id, String deviceId, String startTime, String stopTime, MediaSessionDescription localMediaDescription); /* *Response to Invite Playback other status codes */ boolean respondPlaybackInvite(int statusCode, long id, String deviceId); /* * The media stream sender sends a Message message after the playback ends to notify the SIP server that the playback file has been sent. * notifyType must be "121" */ boolean notifyPlaybackMediaStatus(long id, String deviceId, String notifyType); /* * Terminate Playback session */ void terminatePlayback(long id, String deviceId, boolean isSendBYE); /* * Terminate all Playback sessions */ void terminateAllPlaybacks(boolean isSendBYE); } /** * Signaling Playback Listener */ package com.gb.ntsignalling; public interface GBSIPAgentPlaybackListener { /* *Received historical playback Invite of s=Playback */ void ntsOnInvitePlayback(long id, String deviceId, SessionDescription sessionDescription); /* *Send Playback invite response exception */ void ntsOnPlaybackInviteResponseException(long id, String deviceId, int statusCode, String errorInfo); /* * Received CANCEL Playback INVITE request */ void ntsOnCancelPlayback(long id, String deviceId); /* *Receive Ack */ void ntsOnAckPlayback(long id, String deviceId); /* * Play command */ void ntsOnPlaybackMANSRTSPPlayCommand(long id, String deviceId); /* * Pause command */ void ntsOnPlaybackMANSRTSPPauseCommand(long id, String deviceId); /* * Fast forward/slow forward commands */ void ntsOnPlaybackMANSRTSPScaleCommand(long id, String deviceId, double scale); /* * Random drag command */ void ntsOnPlaybackMANSRTSPSeekCommand(long id, String deviceId, double position_sec); /* * Stop command */ void ntsOnPlaybackMANSRTSPTeardownCommand(long id, String deviceId); /* *Received Bye */ void ntsOnByePlayback(long id, String deviceId); /* * Not terminate Playback when receiving BYE Message */ void ntsOnTerminatePlayback(long id, String deviceId); /* * The conversation corresponding to the Playback session is terminated. Generally, this callback will not be triggered. Currently, it may only start if it responds to 200K but has not received ACK after 64*T1 time. When you receive this, please clean it up accordingly. */ void ntsOnPlaybackDialogTerminated(long id, String deviceId); } /** * Part of the JNI interface, rtp ps package sending and other codes are implemented in C++ */ public class SmartPublisherJniV2 { /** * Open publisher (start push instance) * * @param ctx: get by this.getApplicationContext() * * @param audio_opt: * if 0: Do not push audio * if 1: Push pre-encoding audio (PCM) * if 2: Push encoded audio (aac/pcma/pcmu/speex). * * @param video_opt: * if 0: Do not push video * if 1: Push the pre-encoding video (NV12/I420/RGBA8888 and other formats) * if 2: Push encoded video (AVC/HEVC) * if 3: layer overlay mode * * <pre>This function must be called firstly.
*
* @return the handle of publisher instance
*/
public native long SmartPublisherOpen(Object ctx, int audio_opt, int video_opt, int width, int height);
/**
* Set stream type
* @param type: 0: indicates live stream, 1: indicates on-demand stream, SDK default is 0 (live stream)
* Note: The stream type setting is currently only valid for GB28181 media stream
* @return {0} if successful
*/
public native int SetStreamType(long handle, int type);
/**
* Deliver video on demand package, currently only used for GB28181 push, note that the ByteBuffer object must be DirectBuffer
*
* @param codec_id: codec id, currently supports H264 and H265, 1:H264, 2:H265
*
* @param packet: video data, please refer to H264/H265 Annex B Byte stream format for packet format, for example:
* 0x00000001 nal_unit 0x00000001 …
* H264 IDR: 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit …. or 0x00000001 IDR_nal_unit ….
* H265 IDR: 0x00000001 vps 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit…. or 0x00000001 IDR_nal_unit….
*
* @param offset: offset
* @param size: packet size
* @param pts_us: timestamp, in microseconds
* @param is_pts_discontinuity: Whether the timestamp is discontinuous, 0: not interrupted, 1: discontinuous
* @param is_key: whether it is a key frame, 0: non-key frame, 1: key frame
* @param codec_specific_data: Optional parameter, null can be passed. For H264 key frame package, if the packet does not contain sps and pps, 0x00000001 sps 0x00000001 pps can be passed
*, for H265 key frame packet, if the packet does not contain vps, sps and pps, 0x00000001 vps 0x00000001 sps 0x00000001 pps can be transmitted
* @param codec_specific_data_size: codec_specific_data size
* @param width: image width, 0 can be passed
* @param height: image height, 0 can be passed
*
* @return {0} if successful
*/
public native int PostVideoOnDemandPacketByteBuffer(long handle, int codec_id,
ByteBuffer packet, int offset, int size, long pts_us, int is_pts_discontinuity, int is_key,
byte[] codec_specific_data, int codec_specific_data_size,
int width, int height);
\t
/**
* Deliver audio on demand package, currently only used for GB28181 push. Note that the ByteBuffer object must be DirectBuffer
*
* @param codec_id: codec id, currently supports PCMA and AAC, 65536:PCMA, 65538:AAC
* @param packet: audio data
* @param offset: packet offset
* @param size: packet size
* @param pts_us: timestamp, in microseconds
* @param is_pts_discontinuity: Whether the timestamp is discontinuous, 0: not interrupted, 1: discontinuous
* @param codec_specific_data: If it is AAC, you need to pass Audio Specific Configuration
* @param codec_specific_data_size: codec_specific_data size
* @param sample_rate: sampling rate
* @param channels: number of channels
*
* @return {0} if successful
*/
public native int PostAudioOnDemandPacketByteBuffer(long handle, int codec_id,
ByteBuffer packet, int offset, int size, long pts_us, int is_pts_discontinuity,
byte[] codec_specific_data, int codec_specific_data_size,
int sample_rate, int channels);
\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t
/**
* After on demand source completes seek, please call
* @return {0} if successful
*/
public native int OnSeekProcessed(long handle);
/**
* Start GB28181 media stream
*
* @return {0} if successful
*/
public native int StartGB28181MediaStream(long handle);
/**
* Stop GB28181 media stream
*
* @return {0} if successful
*/
public native int StopGB28181MediaStream(long handle);
/**
* Close the push instance, and when finished, the close interface must be called to release the resources.
*
* @return {0} if successful
*/
public native int SmartPublisherClose(long handle);
}
/**
*Listener part implementation code
*/
public class PlaybackListenerImpl implements com.gb.ntsignalling.GBSIPAgentPlaybackListener {
/*
*Received Invite for file download with s=Playback
*/
@Override
public void ntsOnInvitePlayback(long id, String deviceId, SessionDescription sdp) {
if (!post_task(new PlaybackListenerImpl.OnInviteTask(this.context_, this.is_exit_, this.senders_map_, deviceId, sdp, id))) {
Log.e(TAG, “ntsOnInvitePlayback post_task failed, ” + RecordSender.make_print_tuple(id, deviceId, sdp.getTime().getStartTime(), sdp.getTime().getStopTime()));
// Don’t send 488 here, you can also wait for the transaction to time out
GBSIPAgent agent = this.context_.get_agent();
if (agent != null)
agent.respondPlaybackInvite(488, id, deviceId);
}
}
/*
*Send Playback invite response exception
*/
@Override
public void ntsOnPlaybackInviteResponseException(long id, String deviceId, int statusCode, String errorInfo) {
Log.i(TAG, “ntsOnPlaybackInviteResponseException, status_code:” + statusCode + “, ”
+ RecordSender.make_print_tuple(id, deviceId) + “, error_info:” + errorInfo);
RecordSender sender = senders_map_.remove(id);
if (null == sender)
return;
PlaybackListenerImpl.StopDisposeTask task = new PlaybackListenerImpl.StopDisposeTask(sender);
if (!post_task(task))
task.run();
}
/*
* Received CANCEL Playback INVITE request
*/
@Override
public void ntsOnCancelPlayback(long id, String deviceId) {
Log.i(TAG, “ntsOnCancelPlayback, ” + RecordSender.make_print_tuple(id, deviceId));
RecordSender sender = senders_map_.remove(id);
if (null == sender)
return;
PlaybackListenerImpl.StopDisposeTask task = new PlaybackListenerImpl.StopDisposeTask(sender);
if (!post_task(task))
task.run();
}
/*
*Receive Ack
*/
@Override
public void ntsOnAckPlayback(long id, String deviceId) {
Log.i(TAG, “ntsOnAckPlayback, ” + RecordSender.make_print_tuple(id, deviceId));
RecordSender sender = senders_map_.get(id);
if (null == sender) {
Log.e(TAG, “ntsOnAckPlayback get sender is null, ” + RecordSender.make_print_tuple(id, deviceId));
GBSIPAgent agent = this.context_.get_agent();
if (agent != null)
agent.terminatePlayback(id, deviceId, false);
return;
}
PlaybackListenerImpl.StartTask task = new PlaybackListenerImpl.StartTask(sender, this.senders_map_);
if (!post_task(task))
task.run();
}
/*
*Received Bye
*/
@Override
public void ntsOnByePlayback(long id, String deviceId) {
Log.i(TAG, “ntsOnByePlayback, ” + RecordSender.make_print_tuple(id, deviceId));
RecordSender sender = this.senders_map_.remove(id);
if (null == sender)
return;
PlaybackListenerImpl.StopDisposeTask task = new PlaybackListenerImpl.StopDisposeTask(sender);
if (!post_task(task))
task.run();
}
/*
* Play command
*/
@Override
public void ntsOnPlaybackMANSRTSPPlayCommand(long id, String deviceId) {
RecordSender sender = this.senders_map_.get(id);
if (null == sender) {
Log.e(TAG, “ntsOnPlaybackMANSRTSPPlayCommand can not get sender ” + RecordSender.make_print_tuple(id, deviceId));
return;
}
sender.post_play_command();
Log.i(TAG, “ntsOnPlaybackMANSRTSPPlayCommand ” + RecordSender.make_print_tuple(id, deviceId));
}
/*
* Pause command
*/
@Override
public void ntsOnPlaybackMANSRTSPPauseCommand(long id, String deviceId) {
RecordSender sender = this.senders_map_.get(id);
if (null == sender) {
Log.e(TAG, “ntsOnPlaybackMANSRTSPPauseCommand can not get sender ” + RecordSender.make_print_tuple(id, deviceId));
return;
}
sender.post_pause_command();
Log.i(TAG, “ntsOnPlaybackMANSRTSPPauseCommand ” + RecordSender.make_print_tuple(id, deviceId));
}
/*
* Fast forward/slow forward commands
*/
@Override
public void ntsOnPlaybackMANSRTSPScaleCommand(long id, String deviceId, double scale) {
if (scale < 0.01) {
Log.e(TAG, "ntsOnPlaybackMANSRTSPScaleCommand invalid scale:" + scale + " " + RecordSender.make_print_tuple(id, deviceId));
return;
}
RecordSender sender = this.senders_map_.get(id);
if (null == sender) {
Log.e(TAG, "ntsOnPlaybackMANSRTSPScaleCommand can not get sender, scale:" + scale + " " + RecordSender.make_print_tuple(id, deviceId));
return;
}
sender.post_scale_command(scale);
Log.i(TAG, "ntsOnPlaybackMANSRTSPScaleCommand, scale:" + scale + " " + RecordSender.make_print_tuple(id, deviceId));
}
/*
* Random drag command
*/
@Override
public void ntsOnPlaybackMANSRTSPSeekCommand(long id, String device_id, double position_sec) {
if (position_sec < 0.0) {
Log.e(TAG, "ntsOnPlaybackMANSRTSPSeekCommand invalid seek pos:" + position_sec + ", " + RecordSender.make_print_tuple(id, device_id));
return;
}
RecordSender sender = this.senders_map_.get(id);
if (null == sender) {
Log.e(TAG, "ntsOnPlaybackMANSRTSPSeekCommand can not get sender " + RecordSender.make_print_tuple(id, device_id));
return;
}
long offset_ms = sender.get_file_start_time_offset_ms();
position_sec + = (offset_ms/1000.0);
sender.post_seek_command(position_sec);
Log.i(TAG, "ntsOnPlaybackMANSRTSPSeekCommand seek pos:" + RecordSender.out_point_3(position_sec) + "s, " + RecordSender.make_print_tuple(id, device_id));
}
/*
* Stop command
*/
@Override
public void ntsOnPlaybackMANSRTSPTeardownCommand(long id, String device_id) {
CallTerminatePlaybackTask call_terminate_task = new CallTerminatePlaybackTask(this.context_, id, device_id, true);
post_task(call_terminate_task);
RecordSender sender = this.senders_map_.remove(id);
if (null == sender) {
Log.w(TAG, "ntsOnPlaybackMANSRTSPTeardownCommand can not remove sender " + RecordSender.make_print_tuple(id, device_id));
return;
}
Log.i(TAG, "ntsOnPlaybackMANSRTSPTeardownCommand " + RecordSender.make_print_tuple(id, device_id));
PlaybackListenerImpl.StopDisposeTask task = new PlaybackListenerImpl.StopDisposeTask(sender);
if (!post_task(task))
task.run();
}
/*
* Not terminate Playback when receiving BYE Message
*/
@Override
public void ntsOnTerminatePlayback(long id, String deviceId) {
Log.i(TAG, "ntsOnTerminatePlayback, " + RecordSender.make_print_tuple(id, deviceId));
RecordSender sender = this.senders_map_.remove(id);
if (null == sender)
return;
PlaybackListenerImpl.StopDisposeTask task = new PlaybackListenerImpl.StopDisposeTask(sender);
if (!post_task(task))
task.run();
}
/*
* The conversation corresponding to the Playback session is terminated. Generally, this callback will not be triggered. Currently, it may only start if it responds to 200K but has not received ACK after 64*T1 time.
When you receive this, please clean it up accordingly.
*/
@Override
public void ntsOnPlaybackDialogTerminated(long id, String deviceId) {
Log.i(TAG, "ntsOnPlaybackDialogTerminated, " + RecordSender.make_print_tuple(id, deviceId));
RecordSender sender = this.senders_map_.remove(id);
if (null == sender)
return;
PlaybackListenerImpl.StopDisposeTask task = new PlaybackListenerImpl.StopDisposeTask(sender);
if (!post_task(task))
task.run();
}
}
Android GB28181 historical video and audio playback implementation code has a lot of code, which is roughly divided into three parts, signaling part, rtp packaging and sending part, file retrieval and sending and other logic. Some are implemented in Java and some are implemented in C++. The signaling is explained here for convenience. For the process and key details, only part of the interface definition and implementation code are given. If you have any questions, please contact qq: 1130758427.
The implementation code for remote playback of video files on Android devices has many details and is cumbersome to process. It is recommended that you copy the video files from the device for playback. In addition, it is strongly recommended to use RTP over TCP for playback audio and video transmission (please refer to RFC 4571 for details).