如何使用 GStreamer 正确连接 mp4 视频
How to properly concatenate mp4 videos with GStreamer
我们对 GStreamer 比较陌生。我们在基于 Linux 的自定义板上使用 运行 GStreamer 1.18.4，我们有一个管道使用 rtspsrc 从 IP 摄像机接收视频并创建 MP4 视频剪辑10 秒。
我们遇到了 缺少 PTS 问题 并且我们能够通过实施从我们的 C 代码（gst_base_parse_set_pts_interpolation 打开）打开插值的解决方法来解决它。现在10秒的视频片段已经正确录制了。
我们用来创建视频的管道如下（元素 h264parse 的插值设置为打开）：
gst-launch-1.0 rtspsrc user-id="admin" user-pw="admin"
location="rtsp://192.168.0.131:554/cam/realmonitor?channel=1&subtype=0&unicast=true&proto=Onvif" !
rtph264depay ! h264parse !
splitmuxsink location=video_%02d.mp4 max-files=400 max-size-time=100000000
现在我们面临一个问题，我们需要连接其中的一些视频（假设连接 6 个 10 秒的视频以获得 1 分钟的长视频）并且我们的管道经常被破坏。
我们认为，当其中一个正在连接的视频内插 PTS（我们实施的解决方法）时，管道会中断。
我们用来连接的管道是：
gst-launch-1.0 concat name=c ! queue ! m.video_0
qtmux name=m ! filesink location=/data/users/videos/output_video.mp4
filesrc location=/data/users/videos/cam1_input_video_1.mp4 ! qtdemux ! h264parse ! c.
filesrc location=/data/users/videos/cam2_input_video_2.mp4 ! qtdemux ! h264parse ! c.
我们已经测试了连接同一视频多次的管道，在这种情况下，一切都按预期工作。连接不同文件时（有时）失败。
我们已尝试对管道进行几种不同的修改，但所有修改都在 qtdemux 元素上失败，并显示以下消息：
gst-launch-1.0 --gst-debug-level=3 concat name=c ! queue ! m.video_0 qtmux name=m ! filesink location=/data/users/videos/output_video.mp4 filesrc location=/data/users/videos/cam1_input_video_1.mp4 ! qtdemux ! h264parse ! c. filesrc location=/data/users/videos/cam1_input_video_2.mp4 ! qtdemux ! h264parse ! c.
Setting pipeline to PAUSED ...
0:00:00.461273255 10102 0x60c030 WARN aggregator gstaggregator.c:2046:gst_aggregator_query_latency_unlocked:<m> Latency query failed
0:00:00.465287035 10102 0x6210a0 WARN basesrc gstbasesrc.c:3688:gst_base_src_start_complete:<filesrc2> pad not activated yet
0:00:00.466665891 10102 0x6210a0 WARN basesrc gstbasesrc.c:3688:gst_base_src_start_complete:<filesrc1> pad not activated yet
0:00:00.467579600 10102 0x6210a0 WARN basesrc gstbasesrc.c:3688:gst_base_src_start_complete:<filesrc0> pad not activated yet
Pipeline is PREROLLING ...
0:00:00.471999988 10102 0x60be60 WARN qtdemux qtdemux_types.c:245:qtdemux_type_get: unknown QuickTime node type pasp
0:00:00.472496716 10102 0x60be60 WARN qtdemux qtdemux.c:3067:qtdemux_parse_trex:<qtdemux2> failed to find fragment defaults for stream 1
0:00:00.475729572 10102 0x60bdb0 WARN qtdemux qtdemux_types.c:245:qtdemux_type_get: unknown QuickTime node type pasp
0:00:00.476105597 10102 0x60bdb0 WARN qtdemux qtdemux.c:3067:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 1
0:00:00.480323577 10102 0x60bef0 WARN qtdemux qtdemux_types.c:245:qtdemux_type_get: unknown QuickTime node type pasp
0:00:00.480799931 10102 0x60bef0 WARN qtdemux qtdemux.c:3067:qtdemux_parse_trex:<qtdemux1> failed to find fragment defaults for stream 1
0:00:00.525340747 10102 0x60c030 FIXME basesink gstbasesink.c:3386:gst_base_sink_default_event:<filesink0> stream-start event without group-id. Consider implementing group-id handling in the upstream elements
0:00:00.526037550 10102 0x60c030 WARN qtmux gstqtmux.c:3077:gst_qt_mux_start_file:<m> Robust muxing requires reserved-moov-update-period to be set
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
0:00:00.529879629 10102 0x60c030 FIXME aggregator gstaggregator.c:1367:gst_aggregator_aggregate_func:<m> Subclass should call gst_aggregator_selected_samples() from its aggregate implementation.
New clock: GstSystemClock
0:00:00.770191724 10102 0x60c090 WARN qtmux gstqtmux.c:5346:gst_qt_mux_can_renegotiate:<m> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)3.1, profile=(string)main, codec_data=(buffer)014d401fffe1002e674d401f8d8d402802dff80b7010101400000fa000027103a1801e9c0003567ebbcb8d0c00f4e0001ab3f5de5c2801000468ee3880, width=(int)1280, height=(int)720, framerate=(fraction)20000/1001, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
0:00:00.782437264 10102 0x60c090 WARN qtmux gstqtmux.c:5346:gst_qt_mux_can_renegotiate:<m> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)3.1, profile=(string)main, codec_data=(buffer)014d401fffe1002e674d401f8d8d402802dff80b7010101400000fa000027103a1801e9c0003567ebbcb8d0c00f4e0001ab3f5de5c2801000468ee3880, width=(int)1280, height=(int)720, framerate=(fraction)20000/1001, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
0:00:00.783267561 10102 0x60c090 WARN qtmux gstqtmux.c:5346:gst_qt_mux_can_renegotiate:<m> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)3.1, profile=(string)main, codec_data=(buffer)014d401fffe1002e674d401f8d8d402802dff80b7010101400000fa000027103a1801e9c0003567ebbcb8d0c00f4e0001ab3f5de5c2801000468ee3880, width=(int)1280, height=(int)720, framerate=(fraction)20000/1001, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
0:00:00.784103274 10102 0x60c090 WARN qtmux gstqtmux.c:5346:gst_qt_mux_can_renegotiate:<m> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)3.1, profile=(string)main, codec_data=(buffer)014d401fffe1002e674d401f8d8d402802dff80b7010101400000fa000027103a1801e9c0003567ebbcb8d0c00f4e0001ab3f5de5c2801000468ee3880, width=(int)1280, height=(int)720, framerate=(fraction)20000/1001, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
0:00:00.785129478 10102 0x60c090 WARN qtmux gstqtmux.c:5346:gst_qt_mux_can_renegotiate:<m> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)3.1, profile=(string)main, codec_data=(buffer)014d401fffe1002e674d401f8d8d402802dff80b7010101400000fa000027103a1801e9c0003567ebbcb8d0c00f4e0001ab3f5de5c2801000468ee3880, width=(int)1280, height=(int)720, framerate=(fraction)20000/1001, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
0:00:00.793758637 10102 0x60c090 WARN qtmux gstqtmux.c:5346:gst_qt_mux_can_renegotiate:<m> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)3.1, profile=(string)main, codec_data=(buffer)014d401fffe1002e674d401f8d8d402802dff80b7010101400000fa000027103a1801e9c0003567ebbcb8d0c00f4e0001ab3f5de5c2801000468ee3880, width=(int)1280, height=(int)720, framerate=(fraction)20000/1001, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
0:00:00.794520312 10102 0x60bef0 WARN qtdemux qtdemux.c:6545:gst_qtdemux_loop:<qtdemux1> error: Internal data stream error.
0:00:00.794615432 10102 0x60bef0 WARN qtdemux qtdemux.c:6545:gst_qtdemux_loop:<qtdemux1> error: streaming stopped, reason not-negotiated (-4)
ERROR: from element /GstPipeline:pipeline0/GstQTDemux:qtdemux1: Internal data stream error.
Additional debug info:
../gst-plugins-good-1.18.4/gst/isomp4/qtdemux.c(6545): gst_qtdemux_loop (): /GstPipeline:pipeline0/GstQTDemux:qtdemux1:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.605594272
Setting pipeline to NULL ...
Freeing pipeline ...
知道我们如何改进管道以使串联更好、更稳健以应对 PTS 的可能 errors/inconsistencies 吗？
我们可以通过将 concat 元素替换为 splitmuxsrc.
来解决问题
我们最终使用的管道是：
splitmuxsrc name=splitsrc ! h264parse ! qtmux ! filesink name=sink
要连接的文件列表通过 format-location 信号从 C 传递到 splitsrc 元素：
GStrv *format_location_callback (GstElement * splitmux,
gpointer udata)
我们对 GStreamer 比较陌生。我们在基于 Linux 的自定义板上使用 运行 GStreamer 1.18.4，我们有一个管道使用 rtspsrc 从 IP 摄像机接收视频并创建 MP4 视频剪辑10 秒。 我们遇到了 缺少 PTS 问题 并且我们能够通过实施从我们的 C 代码（gst_base_parse_set_pts_interpolation 打开）打开插值的解决方法来解决它。现在10秒的视频片段已经正确录制了。
我们用来创建视频的管道如下（元素 h264parse 的插值设置为打开）：
gst-launch-1.0 rtspsrc user-id="admin" user-pw="admin"
location="rtsp://192.168.0.131:554/cam/realmonitor?channel=1&subtype=0&unicast=true&proto=Onvif" !
rtph264depay ! h264parse !
splitmuxsink location=video_%02d.mp4 max-files=400 max-size-time=100000000
现在我们面临一个问题，我们需要连接其中的一些视频（假设连接 6 个 10 秒的视频以获得 1 分钟的长视频）并且我们的管道经常被破坏。
我们认为，当其中一个正在连接的视频内插 PTS（我们实施的解决方法）时，管道会中断。
我们用来连接的管道是：
gst-launch-1.0 concat name=c ! queue ! m.video_0
qtmux name=m ! filesink location=/data/users/videos/output_video.mp4
filesrc location=/data/users/videos/cam1_input_video_1.mp4 ! qtdemux ! h264parse ! c.
filesrc location=/data/users/videos/cam2_input_video_2.mp4 ! qtdemux ! h264parse ! c.
我们已经测试了连接同一视频多次的管道，在这种情况下，一切都按预期工作。连接不同文件时（有时）失败。
我们已尝试对管道进行几种不同的修改，但所有修改都在 qtdemux 元素上失败，并显示以下消息：
gst-launch-1.0 --gst-debug-level=3 concat name=c ! queue ! m.video_0 qtmux name=m ! filesink location=/data/users/videos/output_video.mp4 filesrc location=/data/users/videos/cam1_input_video_1.mp4 ! qtdemux ! h264parse ! c. filesrc location=/data/users/videos/cam1_input_video_2.mp4 ! qtdemux ! h264parse ! c.
Setting pipeline to PAUSED ...
0:00:00.461273255 10102 0x60c030 WARN aggregator gstaggregator.c:2046:gst_aggregator_query_latency_unlocked:<m> Latency query failed
0:00:00.465287035 10102 0x6210a0 WARN basesrc gstbasesrc.c:3688:gst_base_src_start_complete:<filesrc2> pad not activated yet
0:00:00.466665891 10102 0x6210a0 WARN basesrc gstbasesrc.c:3688:gst_base_src_start_complete:<filesrc1> pad not activated yet
0:00:00.467579600 10102 0x6210a0 WARN basesrc gstbasesrc.c:3688:gst_base_src_start_complete:<filesrc0> pad not activated yet
Pipeline is PREROLLING ...
0:00:00.471999988 10102 0x60be60 WARN qtdemux qtdemux_types.c:245:qtdemux_type_get: unknown QuickTime node type pasp
0:00:00.472496716 10102 0x60be60 WARN qtdemux qtdemux.c:3067:qtdemux_parse_trex:<qtdemux2> failed to find fragment defaults for stream 1
0:00:00.475729572 10102 0x60bdb0 WARN qtdemux qtdemux_types.c:245:qtdemux_type_get: unknown QuickTime node type pasp
0:00:00.476105597 10102 0x60bdb0 WARN qtdemux qtdemux.c:3067:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 1
0:00:00.480323577 10102 0x60bef0 WARN qtdemux qtdemux_types.c:245:qtdemux_type_get: unknown QuickTime node type pasp
0:00:00.480799931 10102 0x60bef0 WARN qtdemux qtdemux.c:3067:qtdemux_parse_trex:<qtdemux1> failed to find fragment defaults for stream 1
0:00:00.525340747 10102 0x60c030 FIXME basesink gstbasesink.c:3386:gst_base_sink_default_event:<filesink0> stream-start event without group-id. Consider implementing group-id handling in the upstream elements
0:00:00.526037550 10102 0x60c030 WARN qtmux gstqtmux.c:3077:gst_qt_mux_start_file:<m> Robust muxing requires reserved-moov-update-period to be set
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
0:00:00.529879629 10102 0x60c030 FIXME aggregator gstaggregator.c:1367:gst_aggregator_aggregate_func:<m> Subclass should call gst_aggregator_selected_samples() from its aggregate implementation.
New clock: GstSystemClock
0:00:00.770191724 10102 0x60c090 WARN qtmux gstqtmux.c:5346:gst_qt_mux_can_renegotiate:<m> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)3.1, profile=(string)main, codec_data=(buffer)014d401fffe1002e674d401f8d8d402802dff80b7010101400000fa000027103a1801e9c0003567ebbcb8d0c00f4e0001ab3f5de5c2801000468ee3880, width=(int)1280, height=(int)720, framerate=(fraction)20000/1001, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
0:00:00.782437264 10102 0x60c090 WARN qtmux gstqtmux.c:5346:gst_qt_mux_can_renegotiate:<m> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)3.1, profile=(string)main, codec_data=(buffer)014d401fffe1002e674d401f8d8d402802dff80b7010101400000fa000027103a1801e9c0003567ebbcb8d0c00f4e0001ab3f5de5c2801000468ee3880, width=(int)1280, height=(int)720, framerate=(fraction)20000/1001, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
0:00:00.783267561 10102 0x60c090 WARN qtmux gstqtmux.c:5346:gst_qt_mux_can_renegotiate:<m> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)3.1, profile=(string)main, codec_data=(buffer)014d401fffe1002e674d401f8d8d402802dff80b7010101400000fa000027103a1801e9c0003567ebbcb8d0c00f4e0001ab3f5de5c2801000468ee3880, width=(int)1280, height=(int)720, framerate=(fraction)20000/1001, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
0:00:00.784103274 10102 0x60c090 WARN qtmux gstqtmux.c:5346:gst_qt_mux_can_renegotiate:<m> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)3.1, profile=(string)main, codec_data=(buffer)014d401fffe1002e674d401f8d8d402802dff80b7010101400000fa000027103a1801e9c0003567ebbcb8d0c00f4e0001ab3f5de5c2801000468ee3880, width=(int)1280, height=(int)720, framerate=(fraction)20000/1001, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
0:00:00.785129478 10102 0x60c090 WARN qtmux gstqtmux.c:5346:gst_qt_mux_can_renegotiate:<m> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)3.1, profile=(string)main, codec_data=(buffer)014d401fffe1002e674d401f8d8d402802dff80b7010101400000fa000027103a1801e9c0003567ebbcb8d0c00f4e0001ab3f5de5c2801000468ee3880, width=(int)1280, height=(int)720, framerate=(fraction)20000/1001, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
0:00:00.793758637 10102 0x60c090 WARN qtmux gstqtmux.c:5346:gst_qt_mux_can_renegotiate:<m> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)3.1, profile=(string)main, codec_data=(buffer)014d401fffe1002e674d401f8d8d402802dff80b7010101400000fa000027103a1801e9c0003567ebbcb8d0c00f4e0001ab3f5de5c2801000468ee3880, width=(int)1280, height=(int)720, framerate=(fraction)20000/1001, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
0:00:00.794520312 10102 0x60bef0 WARN qtdemux qtdemux.c:6545:gst_qtdemux_loop:<qtdemux1> error: Internal data stream error.
0:00:00.794615432 10102 0x60bef0 WARN qtdemux qtdemux.c:6545:gst_qtdemux_loop:<qtdemux1> error: streaming stopped, reason not-negotiated (-4)
ERROR: from element /GstPipeline:pipeline0/GstQTDemux:qtdemux1: Internal data stream error.
Additional debug info:
../gst-plugins-good-1.18.4/gst/isomp4/qtdemux.c(6545): gst_qtdemux_loop (): /GstPipeline:pipeline0/GstQTDemux:qtdemux1:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.605594272
Setting pipeline to NULL ...
Freeing pipeline ...
知道我们如何改进管道以使串联更好、更稳健以应对 PTS 的可能 errors/inconsistencies 吗？
我们可以通过将 concat 元素替换为 splitmuxsrc.
来解决问题我们最终使用的管道是：
splitmuxsrc name=splitsrc ! h264parse ! qtmux ! filesink name=sink
要连接的文件列表通过 format-location 信号从 C 传递到 splitsrc 元素：
GStrv *format_location_callback (GstElement * splitmux,
gpointer udata)