在 RTMP 流上添加时间覆盖的 Gstreamer 问题
Gstreamer issue with adding timeoverlay on RTMP stream
我需要向 rtmp 流添加时间叠加，并保存到磁盘。以下没有覆盖的管道工作正常：
gst-launch-1.0 -v \
rtmpsrc location=rtmp://192.168.x.x/live/0 do-timestamp=true ! queue2 ! flvdemux name=demux \
flvmux name=mux \
demux.video ! queue ! decodebin \
! nvvidconv \
! 'video/x-raw(memory:NVMM),width=1920,height=1080, format=(string)I420, framerate=50/1' \
! nvv4l2h264enc ! h264parse \
! mux.video \
demux.audio ! queue name="dmx_aud_q" ! mux.audio \
mux.src ! queue name="mux_q" ! filesink location="rtmp.flv"
一旦我添加了时间覆盖（甚至时钟覆盖），管道就不会 运行:
gst-launch-1.0 -v \
rtmpsrc location=rtmp://192.168.0.168/x.x do-timestamp=true ! queue2 ! flvdemux name=demux \
flvmux name=mux \
demux.video ! queue ! decodebin \
! timeoverlay \
! nvvidconv \
! 'video/x-raw(memory:NVMM),width=1920,height=1080, format=(string)I420, framerate=50/1' \
! nvv4l2h264enc ! h264parse \
! mux.video \
demux.audio ! queue name="dmx_aud_q" ! mux.audio \
mux.src ! queue name="mux_q" ! filesink location="rtmp.flv"
GST_DEBUG=3条日志在最后添加。管道图如下。
据我了解，在 decodebin 移交给 timeoverlay 时，上限协商出现了一些问题。我无法弄清楚如何使 timeoverlay 以管道可以继续多路复用的方式接受或输出数据。任何有助于理解这里发生的事情以及如何找到解决方案的帮助都会很棒。
这是在 Nvidia Jetson Nano 板上，运行宁 Ubuntu 18.0。 （以 "nv" 开头的插件通常可以替换为其他系统上的常规插件，我相信 - nvvidconv 与 videoconvert，nvv4l2h264enc 与 omxh264enc 等
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 0:00:00.119491546 834 0x55b9d05600 WARN
v4l2
gstv4l2object.c:2370:gst_v4l2_object_add_interlace_mode:0x55b9cf2360
Failed to determine interlace mode 0:00:00.119570298 834
0x55b9d05600 WARN v4l2
gstv4l2object.c:2370:gst_v4l2_object_add_interlace_mode:0x55b9cf2360
Failed to determine interlace mode 0:00:00.119623164 834
0x55b9d05600 WARN v4l2
gstv4l2object.c:2370:gst_v4l2_object_add_interlace_mode:0x55b9cf2360
Failed to determine interlace mode 0:00:00.119721552 834
0x55b9d05600 WARN v4l2
gstv4l2object.c:4408:gst_v4l2_object_probe_caps:
Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error
-1 Pipeline is PREROLLING ... 0:00:00.406155973 834 0x55b9a72370 WARN flvdemux
gstflvdemux.c:659:gst_flv_demux_parse_tag_script: failed
reading a tag, skipping
/GstPipeline:pipeline0/GstQueue:dmx_aud_q.GstPad:sink: caps =
audio/mpeg, mpegversion=(int)4, framed=(boolean)true,
stream-format=(string)raw, rate=(int)44100, channels=(int)2,
codec_data=(buffer)1210
/GstPipeline:pipeline0/GstQueue:dmx_aud_q.GstPad:src: caps =
audio/mpeg, mpegversion=(int)4, framed=(boolean)true,
stream-format=(string)raw, rate=(int)44100, channels=(int)2,
codec_data=(buffer)1210
/GstPipeline:pipeline0/GstFlvMux:mux.GstFlvMuxPad:sink_1: caps =
audio/mpeg, mpegversion=(int)4, framed=(boolean)true,
stream-format=(string)raw, rate=(int)44100, channels=(int)2,
codec_data=(buffer)1210
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps =
video/x-h264, stream-format=(string)avc, width=(int)1920,
height=(int)1080,
codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps =
video/x-h264, stream-format=(string)avc, width=(int)1920,
height=(int)1080,
codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80
/GstPipeline:pipeline0/GstDecodeBin:decodebin0.GstGhostPad:sink.GstProxyPad:proxypad0:
caps = video/x-h264, stream-format=(string)avc, width=(int)1920,
height=(int)1080,
codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind.GstPad:src:
caps = video/x-h264, stream-format=(string)avc, width=(int)1920,
height=(int)1080,
codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstH264Parse:h264parse1.GstPad:sink:
caps = video/x-h264, stream-format=(string)avc, width=(int)1920,
height=(int)1080,
codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind.GstPad:sink:
caps = video/x-h264, stream-format=(string)avc, width=(int)1920,
height=(int)1080,
codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80
/GstPipeline:pipeline0/GstDecodeBin:decodebin0.GstGhostPad:sink: caps
= video/x-h264, stream-format=(string)avc, width=(int)1920, height=(int)1080,
codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstH264Parse:h264parse1.GstPad:src:
caps = video/x-h264, stream-format=(string)byte-stream,
width=(int)1920, height=(int)1080, framerate=(fraction)50/1,
interlace-mode=(string)progressive, chroma-format=(string)4:2:0,
bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8,
parsed=(boolean)true, alignment=(string)au, profile=(string)baseline,
level=(string)4.2 Opening in BLOCKING MODE 0:00:00.823485062 834
0x55b9ce8a30 WARN v4l2
gstv4l2object.c:4408:gst_v4l2_object_probe_caps:
Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error
-1 0:00:00.823543397 834 0x55b9ce8a30 WARN v4l2 gstv4l2object.c:2370:gst_v4l2_object_add_interlace_mode:0x7f6807a640
Failed to determine interlace mode NvMMLiteOpen : Block : BlockType =
261 NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/nvv4l2decoder:nvv4l2decoder0.GstPad:sink:
caps = video/x-h264, stream-format=(string)byte-stream,
width=(int)1920, height=(int)1080, framerate=(fraction)50/1,
interlace-mode=(string)progressive, chroma-format=(string)4:2:0,
bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8,
parsed=(boolean)true, alignment=(string)au, profile=(string)baseline,
level=(string)4.2
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter1.GstPad:src:
caps = video/x-h264, stream-format=(string)byte-stream,
width=(int)1920, height=(int)1080, framerate=(fraction)50/1,
interlace-mode=(string)progressive, chroma-format=(string)4:2:0,
bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8,
parsed=(boolean)true, alignment=(string)au, profile=(string)baseline,
level=(string)4.2
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter1.GstPad:sink:
caps = video/x-h264, stream-format=(string)byte-stream,
width=(int)1920, height=(int)1080, framerate=(fraction)50/1,
interlace-mode=(string)progressive, chroma-format=(string)4:2:0,
bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8,
parsed=(boolean)true, alignment=(string)au, profile=(string)baseline,
level=(string)4.2 0:00:00.932089228 834 0x55b9ce8a30 WARN
v4l2
gstv4l2object.c:4408:gst_v4l2_object_probe_caps:
Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error
-1 0:00:00.932560124 834 0x55b9ce8a30 WARN v4l2 gstv4l2object.c:2370:gst_v4l2_object_add_interlace_mode:0x7f6807a640
Failed to determine interlace mode
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/nvv4l2decoder:nvv4l2decoder0.GstPad:src:
caps = video/x-raw(memory:NVMM), format=(string)NV12, width=(int)1920,
height=(int)1080, interlace-mode=(string)progressive,
multiview-mode=(string)mono,
multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono,
pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2,
colorimetry=(string)bt709, framerate=(fraction)50/1
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.065:
gst_mini_object_copy: assertion 'mini_object != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.066:
gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.066:
gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.066:
gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.066:
gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.066:
gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.067:
gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.067:
gst_mini_object_copy: assertion 'mini_object != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.068:
gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.068:
gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.068:
gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.068:
gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.068:
gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.069:
gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
0:00:00.942959420 834 0x55b9ce8a30 WARN GST_PADS
gstpad.c:4226:gst_pad_peer_query: could not send
sticky events 0:00:00.943568965 834 0x55b9ce8a30 WARN
v4l2videodec
gstv4l2videodec.c:1433:gst_v4l2_video_dec_decide_allocation:
Duration invalid, not setting latency 0:00:00.944316482 834
0x55b9ce8a30 WARN v4l2bufferpool
gstv4l2bufferpool.c:1054:gst_v4l2_buffer_pool_start:
Uncertain or not enough buffers, enabling copy threshold
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.073:
gst_mini_object_copy: assertion 'mini_object != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.074:
gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.074:
gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.074:
gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.074:
gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.074:
gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.074:
gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
0:00:00.948613871 834 0x55b9ce8a30 WARN basetransform
gstbasetransform.c:1355:gst_base_transform_setcaps:
transform could not transform video/x-h264,
stream-format=(string)byte-stream, width=(int)1920, height=(int)1080,
framerate=(fraction)50/1, interlace-mode=(string)progressive,
chroma-format=(string)4:2:0, bit-depth-luma=(uint)8,
bit-depth-chroma=(uint)8, parsed=(boolean)true, alignment=(string)au,
profile=(string)baseline, level=(string)4.2 in anything we support
0:00:00.948674601 834 0x55b9ce8a30 WARN basetransform
gstbasetransform.c:1415:gst_base_transform_reconfigure:
warning: not negotiated 0:00:00.948709446 834 0x55b9ce8a30 WARN
basetransform
gstbasetransform.c:1415:gst_base_transform_reconfigure:
warning: not negotiated WARNING: from element
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter1:
not negotiated Additional debug info: gstbasetransform.c(1415):
gst_base_transform_reconfigure ():
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter1:
not negotiated 0:00:00.971426937 834 0x7f70004a80 WARN
basesrc gstbasesrc.c:3055:gst_base_src_loop: error: Internal
data stream error. 0:00:00.971545793 834 0x7f70004a80 WARN
basesrc gstbasesrc.c:3055:gst_base_src_loop: error:
streaming stopped, reason not-negotiated (-4) 0:00:00.978535326 834
0x7f6807c8f0 WARN v4l2bufferpool
gstv4l2bufferpool.c:1518:gst_v4l2_buffer_pool_dqbuf:
Driver should never set v4l2_buffer.field to ANY
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105:
gst_mini_object_copy: assertion 'mini_object != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105:
gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105:
gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105:
gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105:
gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105:
gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105:
gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106:
gst_mini_object_copy: assertion 'mini_object != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106:
gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106:
gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106:
gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106:
gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106:
gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106:
gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
ERROR: from element /GstPipeline:pipeline0/GstRTMPSrc:rtmpsrc0:
Internal data stream error.
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106:
gst_mini_object_copy: assertion 'mini_object != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106:
gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106:
gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106:
gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.107:
gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.107:
gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.107:
gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
Additional debug info: gstbasesrc.c(3055): gst_base_src_loop ():
/GstPipeline:pipeline0/GstRTMPSrc:rtmpsrc0: streaming stopped, reason
not-negotiated (-4) ERROR: pipeline doesn't want to preroll. Setting
pipeline to NULL ...
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.107:
gst_mini_object_copy: assertion 'mini_object != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108:
gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108:
gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108:
gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108:
gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108:
gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108:
gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108:
gst_mini_object_copy: assertion 'mini_object != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108:
gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108:
gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108:
gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108:
gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.109:
gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.109:
gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
Freeing pipeline ...
所以正如Aswin所说，通过在timeoverlay之前添加convert解决了。
这是因为 timeoverlay 不能与 DMA 缓冲区一起使用（那是 (memory:NVMM) 的意思）
所以管道看起来像原来的，除了这个变化：
... decodebin ! nvvidconv ! 'video/x-raw' ! timeoverlay ! nvvidconv ! 'video/x-raw(memory:NVMM)
错误基本上是关于 timeoverlay 无法 link 到将缓冲区输出为 DMA 的 nvv4l2decoder。我们需要 nvvidconv 来复制这些缓冲区，以便我们可以在用户 space.
中使用它们
详细说明：
根据我对 DMA/CPU 缓冲区的模糊理解（请更正）- 所有 nv* 元素都可以使用 DMA 缓冲区（我猜更直接），从而加快了处理速度。但是在 nvv4l2decoder 和 nvv4l2h264enc 之间，我们放置了时间叠加层以将时间添加到视频中..
但不幸的是，timeoverlay 元素无法使用这些缓冲区。所以我们添加了 nvviddonv，它缓慢地将缓冲区（它通过 CPU）复制到用户 space，我们慢慢地进行更改，然后它慢慢地回到更快的 "DMA buffer space"。但我看不出有什么办法解决这个问题。除非有一些奇特的自定义元素可以直接使用缓冲区。
在过去的几年里，GStreamer 做了一些工作来更好地使用这些缓冲区 - 我看到了 ndufresne 的一些关于 DMA 栅栏（异步地使用 DMA 缓冲区）和零复制管道（不确定它是否适用）的介绍：
零拷贝：https://youtu.be/kNaa1fPv_uo
DMA 围栏：https://youtu.be/HpmzJGHqObs
一些一般的Linux文章：
https://01.org/linuxgraphics/gfx-docs/drm/driver-api/dma-buf.html
我需要向 rtmp 流添加时间叠加，并保存到磁盘。以下没有覆盖的管道工作正常：
gst-launch-1.0 -v \
rtmpsrc location=rtmp://192.168.x.x/live/0 do-timestamp=true ! queue2 ! flvdemux name=demux \
flvmux name=mux \
demux.video ! queue ! decodebin \
! nvvidconv \
! 'video/x-raw(memory:NVMM),width=1920,height=1080, format=(string)I420, framerate=50/1' \
! nvv4l2h264enc ! h264parse \
! mux.video \
demux.audio ! queue name="dmx_aud_q" ! mux.audio \
mux.src ! queue name="mux_q" ! filesink location="rtmp.flv"
一旦我添加了时间覆盖（甚至时钟覆盖），管道就不会 运行:
gst-launch-1.0 -v \
rtmpsrc location=rtmp://192.168.0.168/x.x do-timestamp=true ! queue2 ! flvdemux name=demux \
flvmux name=mux \
demux.video ! queue ! decodebin \
! timeoverlay \
! nvvidconv \
! 'video/x-raw(memory:NVMM),width=1920,height=1080, format=(string)I420, framerate=50/1' \
! nvv4l2h264enc ! h264parse \
! mux.video \
demux.audio ! queue name="dmx_aud_q" ! mux.audio \
mux.src ! queue name="mux_q" ! filesink location="rtmp.flv"
GST_DEBUG=3条日志在最后添加。管道图如下。
据我了解，在 decodebin 移交给 timeoverlay 时，上限协商出现了一些问题。我无法弄清楚如何使 timeoverlay 以管道可以继续多路复用的方式接受或输出数据。任何有助于理解这里发生的事情以及如何找到解决方案的帮助都会很棒。
这是在 Nvidia Jetson Nano 板上，运行宁 Ubuntu 18.0。 （以 "nv" 开头的插件通常可以替换为其他系统上的常规插件，我相信 - nvvidconv 与 videoconvert，nvv4l2h264enc 与 omxh264enc 等
Setting pipeline to PAUSED ... Opening in BLOCKING MODE 0:00:00.119491546 834 0x55b9d05600 WARN v4l2 gstv4l2object.c:2370:gst_v4l2_object_add_interlace_mode:0x55b9cf2360 Failed to determine interlace mode 0:00:00.119570298 834
0x55b9d05600 WARN v4l2 gstv4l2object.c:2370:gst_v4l2_object_add_interlace_mode:0x55b9cf2360 Failed to determine interlace mode 0:00:00.119623164 834
0x55b9d05600 WARN v4l2 gstv4l2object.c:2370:gst_v4l2_object_add_interlace_mode:0x55b9cf2360 Failed to determine interlace mode 0:00:00.119721552 834
0x55b9d05600 WARN v4l2 gstv4l2object.c:4408:gst_v4l2_object_probe_caps: Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1 Pipeline is PREROLLING ... 0:00:00.406155973 834 0x55b9a72370 WARN flvdemux gstflvdemux.c:659:gst_flv_demux_parse_tag_script: failed reading a tag, skipping /GstPipeline:pipeline0/GstQueue:dmx_aud_q.GstPad:sink: caps = audio/mpeg, mpegversion=(int)4, framed=(boolean)true, stream-format=(string)raw, rate=(int)44100, channels=(int)2, codec_data=(buffer)1210 /GstPipeline:pipeline0/GstQueue:dmx_aud_q.GstPad:src: caps = audio/mpeg, mpegversion=(int)4, framed=(boolean)true, stream-format=(string)raw, rate=(int)44100, channels=(int)2, codec_data=(buffer)1210 /GstPipeline:pipeline0/GstFlvMux:mux.GstFlvMuxPad:sink_1: caps = audio/mpeg, mpegversion=(int)4, framed=(boolean)true, stream-format=(string)raw, rate=(int)44100, channels=(int)2, codec_data=(buffer)1210 /GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, width=(int)1920, height=(int)1080, codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80 /GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = video/x-h264, stream-format=(string)avc, width=(int)1920, height=(int)1080, codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80 /GstPipeline:pipeline0/GstDecodeBin:decodebin0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-h264, stream-format=(string)avc, width=(int)1920, height=(int)1080, codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80 /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind.GstPad:src: caps = video/x-h264, stream-format=(string)avc, width=(int)1920, height=(int)1080, codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80 /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstH264Parse:h264parse1.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, width=(int)1920, height=(int)1080, codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80 /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, width=(int)1920, height=(int)1080, codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80 /GstPipeline:pipeline0/GstDecodeBin:decodebin0.GstGhostPad:sink: caps = video/x-h264, stream-format=(string)avc, width=(int)1920, height=(int)1080, codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80 /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstH264Parse:h264parse1.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, width=(int)1920, height=(int)1080, framerate=(fraction)50/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, alignment=(string)au, profile=(string)baseline, level=(string)4.2 Opening in BLOCKING MODE 0:00:00.823485062 834
0x55b9ce8a30 WARN v4l2 gstv4l2object.c:4408:gst_v4l2_object_probe_caps: Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1 0:00:00.823543397 834 0x55b9ce8a30 WARN v4l2 gstv4l2object.c:2370:gst_v4l2_object_add_interlace_mode:0x7f6807a640 Failed to determine interlace mode NvMMLiteOpen : Block : BlockType = 261 NVMEDIA: Reading vendor.tegra.display-size : status: 6 NvMMLiteBlockCreate : Block : BlockType = 261 /GstPipeline:pipeline0/GstDecodeBin:decodebin0/nvv4l2decoder:nvv4l2decoder0.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, width=(int)1920, height=(int)1080, framerate=(fraction)50/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, alignment=(string)au, profile=(string)baseline, level=(string)4.2 /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, width=(int)1920, height=(int)1080, framerate=(fraction)50/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, alignment=(string)au, profile=(string)baseline, level=(string)4.2 /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, width=(int)1920, height=(int)1080, framerate=(fraction)50/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, alignment=(string)au, profile=(string)baseline, level=(string)4.2 0:00:00.932089228 834 0x55b9ce8a30 WARN
v4l2 gstv4l2object.c:4408:gst_v4l2_object_probe_caps: Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1 0:00:00.932560124 834 0x55b9ce8a30 WARN v4l2 gstv4l2object.c:2370:gst_v4l2_object_add_interlace_mode:0x7f6807a640 Failed to determine interlace mode /GstPipeline:pipeline0/GstDecodeBin:decodebin0/nvv4l2decoder:nvv4l2decoder0.GstPad:src: caps = video/x-raw(memory:NVMM), format=(string)NV12, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)50/1(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.065: gst_mini_object_copy: assertion 'mini_object != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.066: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.066: gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.066: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.066: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.066: gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.067: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.067: gst_mini_object_copy: assertion 'mini_object != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.068: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.068: gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.068: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.068: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.068: gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.069: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed 0:00:00.942959420 834 0x55b9ce8a30 WARN GST_PADS gstpad.c:4226:gst_pad_peer_query: could not send sticky events 0:00:00.943568965 834 0x55b9ce8a30 WARN
v4l2videodec gstv4l2videodec.c:1433:gst_v4l2_video_dec_decide_allocation: Duration invalid, not setting latency 0:00:00.944316482 834
0x55b9ce8a30 WARN v4l2bufferpool gstv4l2bufferpool.c:1054:gst_v4l2_buffer_pool_start: Uncertain or not enough buffers, enabling copy threshold(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.073: gst_mini_object_copy: assertion 'mini_object != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.074: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.074: gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.074: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.074: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.074: gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.074: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed 0:00:00.948613871 834 0x55b9ce8a30 WARN basetransform gstbasetransform.c:1355:gst_base_transform_setcaps: transform could not transform video/x-h264, stream-format=(string)byte-stream, width=(int)1920, height=(int)1080, framerate=(fraction)50/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, alignment=(string)au, profile=(string)baseline, level=(string)4.2 in anything we support 0:00:00.948674601 834 0x55b9ce8a30 WARN basetransform gstbasetransform.c:1415:gst_base_transform_reconfigure: warning: not negotiated 0:00:00.948709446 834 0x55b9ce8a30 WARN
basetransform gstbasetransform.c:1415:gst_base_transform_reconfigure: warning: not negotiated WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter1: not negotiated Additional debug info: gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter1: not negotiated 0:00:00.971426937 834 0x7f70004a80 WARN
basesrc gstbasesrc.c:3055:gst_base_src_loop: error: Internal data stream error. 0:00:00.971545793 834 0x7f70004a80 WARN
basesrc gstbasesrc.c:3055:gst_base_src_loop: error: streaming stopped, reason not-negotiated (-4) 0:00:00.978535326 834 0x7f6807c8f0 WARN v4l2bufferpool gstv4l2bufferpool.c:1518:gst_v4l2_buffer_pool_dqbuf: Driver should never set v4l2_buffer.field to ANY(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105: gst_mini_object_copy: assertion 'mini_object != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105: gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105: gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_mini_object_copy: assertion 'mini_object != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed ERROR: from element /GstPipeline:pipeline0/GstRTMPSrc:rtmpsrc0: Internal data stream error.
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_mini_object_copy: assertion 'mini_object != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.107: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.107: gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.107: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed Additional debug info: gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstRTMPSrc:rtmpsrc0: streaming stopped, reason not-negotiated (-4) ERROR: pipeline doesn't want to preroll. Setting pipeline to NULL ...
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.107: gst_mini_object_copy: assertion 'mini_object != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_mini_object_copy: assertion 'mini_object != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.109: gst_structure_copy: assertion 'structure != NULL' failed
(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.109: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed Freeing pipeline ...
所以正如Aswin所说，通过在timeoverlay之前添加convert解决了。
这是因为 timeoverlay 不能与 DMA 缓冲区一起使用（那是 (memory:NVMM) 的意思）
所以管道看起来像原来的，除了这个变化：
... decodebin ! nvvidconv ! 'video/x-raw' ! timeoverlay ! nvvidconv ! 'video/x-raw(memory:NVMM)
错误基本上是关于 timeoverlay 无法 link 到将缓冲区输出为 DMA 的 nvv4l2decoder。我们需要 nvvidconv 来复制这些缓冲区，以便我们可以在用户 space.
中使用它们详细说明：
根据我对 DMA/CPU 缓冲区的模糊理解（请更正）- 所有 nv* 元素都可以使用 DMA 缓冲区（我猜更直接），从而加快了处理速度。但是在 nvv4l2decoder 和 nvv4l2h264enc 之间，我们放置了时间叠加层以将时间添加到视频中..
但不幸的是，timeoverlay 元素无法使用这些缓冲区。所以我们添加了 nvviddonv，它缓慢地将缓冲区（它通过 CPU）复制到用户 space，我们慢慢地进行更改，然后它慢慢地回到更快的 "DMA buffer space"。但我看不出有什么办法解决这个问题。除非有一些奇特的自定义元素可以直接使用缓冲区。
在过去的几年里，GStreamer 做了一些工作来更好地使用这些缓冲区 - 我看到了 ndufresne 的一些关于 DMA 栅栏（异步地使用 DMA 缓冲区）和零复制管道（不确定它是否适用）的介绍： 零拷贝：https://youtu.be/kNaa1fPv_uo DMA 围栏：https://youtu.be/HpmzJGHqObs
一些一般的Linux文章： https://01.org/linuxgraphics/gfx-docs/drm/driver-api/dma-buf.html