Gstreamer 从 USB 源播放原始 h.264 流(时间戳问题)
Gstreamer Playing a raw h.264 stream from a USB source (timestamp issues)
我正在处理通过 USB 输出原始 h.264 视频流的实时视频源(无人机无线视频接收器)。我的目标是将它集成到 Android 中的 QGroundStation,它有一个 GStreamer 管道。
我已经将接收到的 USB 数据的一部分转储到一个文件中,可以使用以下命令使用 vlc 完美播放:
vlc -c dump.bin --demux h264
但是,如果我使用此 GStreamer 管道播放它,播放速度太高(如 x10)
gst-launch-1.0.exe filesrc location="dump.bin" ! h264parse ! avdec_h264 ! autovideosink
我正在使用 appsrc 将 USB 数据推送到 QGroundControl 管道中。视频播放,但丢了很多帧,gstreamer 抱怨丢包是因为帧太迟了。
[...]
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.404134207)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.403025291)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.401385832)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.400435290)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.399607540)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.398911040)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.398131998)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.397308623)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.396620290)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.395761040)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.395125498)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.394197123)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.393461831)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.392803831)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.391983373)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.391033998)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.389664914)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.388862831)
W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 38639 will be dropped
W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 37460 will be dropped
W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 46566 will be dropped
W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 36055 will be dropped
W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 43397 will be dropped
[...]
仔细检查我的转储后,我意识到该流缺少 pts 和 dts 信息(这在基线 h.264 流中似乎很常见)
ffprobe -show_frames dump.bin
[PACKET]
codec_type=video
stream_index=0
pts=N/A
pts_time=N/A
dts=N/A
dts_time=N/A
duration=48000
duration_time=0.040000
convergence_duration=N/A
convergence_duration_time=N/A
size=1843
pos=0
flags=K_
[/PACKET]
[PACKET]
codec_type=video
stream_index=0
pts=N/A
pts_time=N/A
dts=N/A
dts_time=N/A
duration=48000
duration_time=0.040000
convergence_duration=N/A
convergence_duration_time=N/A
size=16851
pos=1843
flags=K_
[/PACKET]
但显然,持续时间信息在那里。
USB 端点读取 512 字节块(由于 USB 高速最大。批量端点的有效负载大小),并且一些传输更小(400 多个字节长)。我无法检测 NAL 的 beginning/end,因为它是一个不透明的连续字节流。 (video/x-h264, stream-format=(string)byte-stream, alignment=none)
所以我构建了一个 appsrc 来将视频流推送到管道,并尝试像这样盲目地为缓冲区添加时间戳:
void _startFeed(GstElement *source, guint size, gpointer pContext) {
(void)pContext;
GstBuffer *pBuffer;
GstFlowReturn ret;
GstMapInfo map;
guint8 *pRaw;
GstClock *pClk = gst_element_get_clock(gPipeline);
GstClockTime absolute = gst_clock_get_time(pClk);
GstClockTime base = gst_element_get_base_time(gPipeline);
gst_object_unref(pClk);
pBuffer = gst_buffer_new_and_alloc(bufferSize);
// Timestamp the buffers
GST_BUFFER_PTS(pBuffer) = absolute - base;
GST_BUFFER_DTS(pBuffer) = GST_BUFFER_PTS(pBuffer);
gst_buffer_map(pBuffer, &map, GST_MAP_WRITE);
pRaw = (guint8 *)map.data;
fifo_pull(pRaw, size); // This pulls up to 'size' bytes from the USB FIFO and copies it to pRaw
gst_buffer_unmap(pBuffer, &map);
g_signal_emit_by_name(gAppSrc, "push-buffer", pBuffer, &ret);
gst_buffer_unref(pBuffer);
}
...但仍然没有运气...
我使用以下管道将 h.264 流编码为 RTP 有效负载,然后使用指定目标帧速率的 caps 过滤器对其进行解码,但取得的成功有限:
gst-launch-1.0.exe filesrc location="dump.bin" ! video/x-h264, stream-format=(string)byte-stream, alignment=none ! h264parse ! rtph264pay ! rtpjitterbuffer ! rtph264depay ! video/x-h264, stream-format=byte-stream, alignment=nal, framerate=(fraction)30/1 ! h264parse ! avdec_h264 ! autovideosink
我可以在 C++ 中将它构建到 QGroundControl 中,但我认为这不是正确的方法,我不应该对目标帧速率做出任何假设,因为在这种情况下它是 30 fps,但它可能会动态变化。
所以,我的问题是:
- 让视频以正确的速度播放而不掉帧的正确方法是什么?
- 要求 GStreamer 使用标准管道根据数据包的持续时间信息生成 PTS/DTS(没有 B 帧,因此 PTS 应该等于 DTS)是否合理或可能?
更新::
- 我尝试了一种已知的从 DTS 插入 PTS 的解决方法 +
持续时间描述 here 但就我而言,PTS 和 DTS 都没有
在流中可用,因此没有任何区别。
- 此特定视频转储的正确帧速率应为 29.97 fps (NTSC)。我在视频图像中有一个运行计数器来验证它。
- 正如您在我的 ffprobe 输出中看到的,duration_time 信息显示为 0.04 秒,对应于 25 fps,这让我感到困惑。
- 令人惊讶的是,VLC 以正确的速度播放视频,并正确猜测帧速率为 29.97 fps(根据您按 Ctrl+J 得到的视频编解码器信息 window)
- 我注意到 rtph264pay 会生成丢失的时间戳,但猜测的帧速率是 25 fps,如数据包的 duration_time 字段所示
我使用了 h.264 流分析器并意识到这个特定的流在 SPS NAL 中没有 VUI 信息,(因此没有 time_tick 或 time_scale 信息可以被解析器使用计算 PTS)。
对我有用的是在接收器上设置 属性 "sync" = false 以在帧到达时立即渲染帧。不理想,但我可以忍受。
我正在处理通过 USB 输出原始 h.264 视频流的实时视频源(无人机无线视频接收器)。我的目标是将它集成到 Android 中的 QGroundStation,它有一个 GStreamer 管道。
我已经将接收到的 USB 数据的一部分转储到一个文件中,可以使用以下命令使用 vlc 完美播放:
vlc -c dump.bin --demux h264
但是,如果我使用此 GStreamer 管道播放它,播放速度太高(如 x10)
gst-launch-1.0.exe filesrc location="dump.bin" ! h264parse ! avdec_h264 ! autovideosink
我正在使用 appsrc 将 USB 数据推送到 QGroundControl 管道中。视频播放,但丢了很多帧,gstreamer 抱怨丢包是因为帧太迟了。
[...]
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.404134207)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.403025291)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.401385832)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.400435290)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.399607540)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.398911040)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.398131998)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.397308623)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.396620290)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.395761040)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.395125498)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.394197123)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.393461831)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.392803831)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.391983373)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.391033998)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.389664914)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.388862831)
W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 38639 will be dropped
W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 37460 will be dropped
W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 46566 will be dropped
W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 36055 will be dropped
W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 43397 will be dropped
[...]
仔细检查我的转储后,我意识到该流缺少 pts 和 dts 信息(这在基线 h.264 流中似乎很常见)
ffprobe -show_frames dump.bin
[PACKET]
codec_type=video
stream_index=0
pts=N/A
pts_time=N/A
dts=N/A
dts_time=N/A
duration=48000
duration_time=0.040000
convergence_duration=N/A
convergence_duration_time=N/A
size=1843
pos=0
flags=K_
[/PACKET]
[PACKET]
codec_type=video
stream_index=0
pts=N/A
pts_time=N/A
dts=N/A
dts_time=N/A
duration=48000
duration_time=0.040000
convergence_duration=N/A
convergence_duration_time=N/A
size=16851
pos=1843
flags=K_
[/PACKET]
但显然,持续时间信息在那里。
USB 端点读取 512 字节块(由于 USB 高速最大。批量端点的有效负载大小),并且一些传输更小(400 多个字节长)。我无法检测 NAL 的 beginning/end,因为它是一个不透明的连续字节流。 (video/x-h264, stream-format=(string)byte-stream, alignment=none)
所以我构建了一个 appsrc 来将视频流推送到管道,并尝试像这样盲目地为缓冲区添加时间戳:
void _startFeed(GstElement *source, guint size, gpointer pContext) {
(void)pContext;
GstBuffer *pBuffer;
GstFlowReturn ret;
GstMapInfo map;
guint8 *pRaw;
GstClock *pClk = gst_element_get_clock(gPipeline);
GstClockTime absolute = gst_clock_get_time(pClk);
GstClockTime base = gst_element_get_base_time(gPipeline);
gst_object_unref(pClk);
pBuffer = gst_buffer_new_and_alloc(bufferSize);
// Timestamp the buffers
GST_BUFFER_PTS(pBuffer) = absolute - base;
GST_BUFFER_DTS(pBuffer) = GST_BUFFER_PTS(pBuffer);
gst_buffer_map(pBuffer, &map, GST_MAP_WRITE);
pRaw = (guint8 *)map.data;
fifo_pull(pRaw, size); // This pulls up to 'size' bytes from the USB FIFO and copies it to pRaw
gst_buffer_unmap(pBuffer, &map);
g_signal_emit_by_name(gAppSrc, "push-buffer", pBuffer, &ret);
gst_buffer_unref(pBuffer);
}
...但仍然没有运气...
我使用以下管道将 h.264 流编码为 RTP 有效负载,然后使用指定目标帧速率的 caps 过滤器对其进行解码,但取得的成功有限:
gst-launch-1.0.exe filesrc location="dump.bin" ! video/x-h264, stream-format=(string)byte-stream, alignment=none ! h264parse ! rtph264pay ! rtpjitterbuffer ! rtph264depay ! video/x-h264, stream-format=byte-stream, alignment=nal, framerate=(fraction)30/1 ! h264parse ! avdec_h264 ! autovideosink
我可以在 C++ 中将它构建到 QGroundControl 中,但我认为这不是正确的方法,我不应该对目标帧速率做出任何假设,因为在这种情况下它是 30 fps,但它可能会动态变化。
所以,我的问题是:
- 让视频以正确的速度播放而不掉帧的正确方法是什么?
- 要求 GStreamer 使用标准管道根据数据包的持续时间信息生成 PTS/DTS(没有 B 帧,因此 PTS 应该等于 DTS)是否合理或可能?
更新::
- 我尝试了一种已知的从 DTS 插入 PTS 的解决方法 + 持续时间描述 here 但就我而言,PTS 和 DTS 都没有 在流中可用,因此没有任何区别。
- 此特定视频转储的正确帧速率应为 29.97 fps (NTSC)。我在视频图像中有一个运行计数器来验证它。
- 正如您在我的 ffprobe 输出中看到的,duration_time 信息显示为 0.04 秒,对应于 25 fps,这让我感到困惑。
- 令人惊讶的是,VLC 以正确的速度播放视频,并正确猜测帧速率为 29.97 fps(根据您按 Ctrl+J 得到的视频编解码器信息 window)
- 我注意到 rtph264pay 会生成丢失的时间戳,但猜测的帧速率是 25 fps,如数据包的 duration_time 字段所示
我使用了 h.264 流分析器并意识到这个特定的流在 SPS NAL 中没有 VUI 信息,(因此没有 time_tick 或 time_scale 信息可以被解析器使用计算 PTS)。
对我有用的是在接收器上设置 属性 "sync" = false 以在帧到达时立即渲染帧。不理想,但我可以忍受。