Qt+GStreamer:如何在播放直播视频流时抓拍
Qt+GStreamer: How to take a snapshot while playing live video stream
我开发了一个基于Qt 和QtGstreamer 的视频播放器。它用于播放实时流 (RTSP)。我必须增加用户在播放实时流时拍摄快照的可能性,而不会影响视频播放。
这是我制作的管道图:
-->queue-->autovideosink
uridecodebin-->videoflip-->tee--|
| -->queue->videoconvert-->pngenc-->filesink
|
|->audioconvert-->autoaudiosink
我使用来自 uridecodebin
的 pad-added
信号来添加和 link 动态地将我的元素添加到管道,接收到的上限的函数。
void Player::onPadAdded(const QGst::PadPtr &pad)
{
QGst::CapsPtr caps = pad->currentCaps();
if (caps->toString().startsWith("video/x-raw")) {
qDebug("Received 'video/x-raw' caps");
handleNewVideoPad(pad);
}
else if (caps->toString().startsWith("audio/x-raw")) {
qDebug("Received 'audio/x-raw' caps");
if (!m_audioEnabled) {
qDebug("Audio is disabled in the player. Ignoring...");
return;
}
handleNewAudioPad(pad);
}
else {
qWarning("Unsuported caps, arborting ...!");
return;
}
}
[...]
void Player::handleNewVideoPad(QGst::PadPtr pad)
{
m_player->videoTeeVideoSrcPad = m_player->videoTee->getRequestPad("src_%u");
// Add video elements
m_player->pipeline->add(m_player->videoFlip);
m_player->pipeline->add(m_player->videoTee);
m_player->pipeline->add(m_player->videoQueue);
m_player->pipeline->add(m_player->videoSink);
// Add snap elements
m_player->pipeline->add(m_player->snapQueue);
m_player->pipeline->add(m_player->snapConverter);
m_player->pipeline->add(m_player->snapEncoder);
m_player->pipeline->add(m_player->snapSink);
// Link video elements
m_player->videoFlip->link(m_player->videoTee);
m_player->videoQueue->link(m_player->videoSink);
// Link snap elements
m_player->snapQueue->link(m_player->snapConverter);
m_player->snapConverter->link(m_player->snapEncoder);
m_player->snapEncoder->link(m_player->snapSink);
// Lock snap elements
m_player->snapQueue->setStateLocked(true);
m_player->snapConverter->setStateLocked(true);
m_player->snapEncoder->setStateLocked(true);
m_player->snapSink->setStateLocked(true);
m_player->videoFlip->setState(QGst::StatePlaying);
m_player->videoTee->setState(QGst::StatePlaying);
m_player->videoQueue->setState(QGst::StatePlaying);
m_player->videoSink->setState(QGst::StatePlaying);
// Link pads
m_player->videoTeeVideoSrcPad->link(m_player->videoQueue->getStaticPad("sink"));
pad->link(m_player->videoSinkPad);
m_player->videoLinked = true;
}
快照方法:
void Player::takeSnapshot()
{
QDateTime dateTime = QDateTime::currentDateTime();
QString snapLocation = QString("/%1/snap_%2.png").arg(m_snapDir).arg(dateTime.toString(Qt::ISODate));
m_player->inSnapshotCaputre = true;
if (m_player->videoTeeSnapSrcPad) {
m_player->videoTee->releaseRequestPad(m_player->videoTeeSnapSrcPad);
m_player->videoTeeSnapSrcPad.clear();
}
m_player->videoTeeSnapSrcPad = m_player->videoTee->getRequestPad("src_%u");
// Stop the snapshot branch
m_player->snapQueue->setState(QGst::StateNull);
m_player->snapConverter->setState(QGst::StateNull);
m_player->snapEncoder->setState(QGst::StateNull);
m_player->snapSink->setState(QGst::StateNull);
// Link Tee src pad to snap queue sink pad
m_player->videoTeeSnapSrcPad->link(m_player->snapQueue->getStaticPad("sink"));
// Set the snapshot location property
m_player->snapSink->setProperty("location", snapLocation);
// Unlock snapshot branch
m_player->snapQueue->setStateLocked(false);
m_player->snapConverter->setStateLocked(false);
m_player->snapEncoder->setStateLocked(false);
m_player->snapSink->setStateLocked(false);
m_player->videoTeeSnapSrcPad->setActive(true);
// Synch snapshot branch state with parent
m_player->snapQueue->syncStateWithParent();
m_player->snapConverter->syncStateWithParent();
m_player->snapEncoder->syncStateWithParent();
m_player->snapSink->syncStateWithParent();
}
总线消息回调:
void Player::onBusMessage(const QGst::MessagePtr & message)
{
QGst::ElementPtr source = message->source().staticCast<QGst::Element>();
switch (message->type()) {
case QGst::MessageEos: { //End of stream. We reached the end of the file.
qDebug("Message End Off Stream");
if (m_player->inSnapshotCaputre) {
blockSignals(true);
pause();
play();
blockSignals(false);
m_player->inSnapshotCaputre = false;
}
else {
m_eos = true;
stop();
}
break;
}
[...]
}
问题是:
- 当我将
snapshot
属性 设置为 pngenc
元素的 true
时,我收到停止我的管道的 EOS
事件,所以我需要重新启动它,这会冻结视频播放大约半秒钟,这对我来说是不可接受的。
- 当我将
snapshot
属性 设置为 pngenc
元素的 false
时,我没有管道扰动,但我的 png 文件一直在增长,直到我再次调用Player::takeSnapshot()
方法。
我哪里错了?有更好的方法吗?
我尝试为我的 snapshot
分支创建 QGst::Bin
元素但未成功。焊盘探针呢?
提前致谢
您可以在任何水槽上采集最后一个样本 属性,例如您的视频接收器。这包含一个 GstSample,它有一个缓冲区,其中包含最新的视频帧。您可以将其作为快照,例如使用 gst_video_convert_sample() 或其异步变体,将其转换为 PNG/JPG/whatever.
见https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-libs/html/GstBaseSink.html#GstBaseSink--last-sample and https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-libs/html/gst-plugins-base-libs-gstvideo.html#gst-video-convert-sample
或者,您必须在第一帧之后关闭 filesink 快照管道。例如,通过 pad 探测来了解第一帧发生的时间,然后注入 EOS 事件以防止将更多 PNG 帧附加到同一文件。
感谢@sebastian-droge 的回答,我找到了解决方案,使用 gst_video_convert_sample
和我的视频接收器的 last-sample
属性。
我实施的解决方案是:
void Player::takeSnapshot()
{
QDateTime currentDate = QDateTime::currentDateTime();
QString location = QString("%1/snap_%2.png").arg(QDir::homePath()).arg(currentDate.toString(Qt::ISODate));
QImage snapShot;
QImage::Format snapFormat;
QGlib::Value val = m_videoSink->property("last-sample");
GstSample *videoSample = (GstSample *)g_value_get_boxed(val);
QGst::SamplePtr sample = QGst::SamplePtr::wrap(videoSample);
QGst::SamplePtr convertedSample;
QGst::BufferPtr buffer;
QGst::CapsPtr caps = sample->caps();
QGst::MapInfo mapInfo;
GError *err = NULL;
GstCaps * capsTo = NULL;
const QGst::StructurePtr structure = caps->internalStructure(0);
int width, height;
width = structure.data()->value("width").get<int>();
height = structure.data()->value("height").get<int>();
qDebug() << "Sample caps:" << structure.data()->toString();
/*
* { QImage::Format_RGBX8888, GST_VIDEO_FORMAT_RGBx },
* { QImage::Format_RGBA8888, GST_VIDEO_FORMAT_RGBA },
* { QImage::Format_RGB888 , GST_VIDEO_FORMAT_RGB },
* { QImage::Format_RGB16 , GST_VIDEO_FORMAT_RGB16 }
*/
snapFormat = QImage::Format_RGB888;
capsTo = gst_caps_new_simple("video/x-raw",
"format", G_TYPE_STRING, "RGB",
"width", G_TYPE_INT, width,
"height", G_TYPE_INT, height,
NULL);
convertedSample = QGst::SamplePtr::wrap(gst_video_convert_sample(videoSample, capsTo, GST_SECOND, &err));
if (convertedSample.isNull()) {
qWarning() << "gst_video_convert_sample Failed:" << err->message;
}
else {
qDebug() << "Converted sample caps:" << convertedSample->caps()->toString();
buffer = convertedSample->buffer();
buffer->map(mapInfo, QGst::MapRead);
snapShot = QImage((const uchar *)mapInfo.data(),
width,
height,
snapFormat);
qDebug() << "Saving snap to" << location;
snapShot.save(location);
buffer->unmap(mapInfo);
}
val.clear();
sample.clear();
convertedSample.clear();
buffer.clear();
caps.clear();
g_clear_error(&err);
if (capsTo)
gst_caps_unref(capsTo);
}
我创建了一个简单的测试应用程序,它实现了这个解决方案。该代码可在我的 Github
我开发了一个基于Qt 和QtGstreamer 的视频播放器。它用于播放实时流 (RTSP)。我必须增加用户在播放实时流时拍摄快照的可能性,而不会影响视频播放。
这是我制作的管道图:
-->queue-->autovideosink
uridecodebin-->videoflip-->tee--|
| -->queue->videoconvert-->pngenc-->filesink
|
|->audioconvert-->autoaudiosink
我使用来自 uridecodebin
的 pad-added
信号来添加和 link 动态地将我的元素添加到管道,接收到的上限的函数。
void Player::onPadAdded(const QGst::PadPtr &pad)
{
QGst::CapsPtr caps = pad->currentCaps();
if (caps->toString().startsWith("video/x-raw")) {
qDebug("Received 'video/x-raw' caps");
handleNewVideoPad(pad);
}
else if (caps->toString().startsWith("audio/x-raw")) {
qDebug("Received 'audio/x-raw' caps");
if (!m_audioEnabled) {
qDebug("Audio is disabled in the player. Ignoring...");
return;
}
handleNewAudioPad(pad);
}
else {
qWarning("Unsuported caps, arborting ...!");
return;
}
}
[...]
void Player::handleNewVideoPad(QGst::PadPtr pad)
{
m_player->videoTeeVideoSrcPad = m_player->videoTee->getRequestPad("src_%u");
// Add video elements
m_player->pipeline->add(m_player->videoFlip);
m_player->pipeline->add(m_player->videoTee);
m_player->pipeline->add(m_player->videoQueue);
m_player->pipeline->add(m_player->videoSink);
// Add snap elements
m_player->pipeline->add(m_player->snapQueue);
m_player->pipeline->add(m_player->snapConverter);
m_player->pipeline->add(m_player->snapEncoder);
m_player->pipeline->add(m_player->snapSink);
// Link video elements
m_player->videoFlip->link(m_player->videoTee);
m_player->videoQueue->link(m_player->videoSink);
// Link snap elements
m_player->snapQueue->link(m_player->snapConverter);
m_player->snapConverter->link(m_player->snapEncoder);
m_player->snapEncoder->link(m_player->snapSink);
// Lock snap elements
m_player->snapQueue->setStateLocked(true);
m_player->snapConverter->setStateLocked(true);
m_player->snapEncoder->setStateLocked(true);
m_player->snapSink->setStateLocked(true);
m_player->videoFlip->setState(QGst::StatePlaying);
m_player->videoTee->setState(QGst::StatePlaying);
m_player->videoQueue->setState(QGst::StatePlaying);
m_player->videoSink->setState(QGst::StatePlaying);
// Link pads
m_player->videoTeeVideoSrcPad->link(m_player->videoQueue->getStaticPad("sink"));
pad->link(m_player->videoSinkPad);
m_player->videoLinked = true;
}
快照方法:
void Player::takeSnapshot()
{
QDateTime dateTime = QDateTime::currentDateTime();
QString snapLocation = QString("/%1/snap_%2.png").arg(m_snapDir).arg(dateTime.toString(Qt::ISODate));
m_player->inSnapshotCaputre = true;
if (m_player->videoTeeSnapSrcPad) {
m_player->videoTee->releaseRequestPad(m_player->videoTeeSnapSrcPad);
m_player->videoTeeSnapSrcPad.clear();
}
m_player->videoTeeSnapSrcPad = m_player->videoTee->getRequestPad("src_%u");
// Stop the snapshot branch
m_player->snapQueue->setState(QGst::StateNull);
m_player->snapConverter->setState(QGst::StateNull);
m_player->snapEncoder->setState(QGst::StateNull);
m_player->snapSink->setState(QGst::StateNull);
// Link Tee src pad to snap queue sink pad
m_player->videoTeeSnapSrcPad->link(m_player->snapQueue->getStaticPad("sink"));
// Set the snapshot location property
m_player->snapSink->setProperty("location", snapLocation);
// Unlock snapshot branch
m_player->snapQueue->setStateLocked(false);
m_player->snapConverter->setStateLocked(false);
m_player->snapEncoder->setStateLocked(false);
m_player->snapSink->setStateLocked(false);
m_player->videoTeeSnapSrcPad->setActive(true);
// Synch snapshot branch state with parent
m_player->snapQueue->syncStateWithParent();
m_player->snapConverter->syncStateWithParent();
m_player->snapEncoder->syncStateWithParent();
m_player->snapSink->syncStateWithParent();
}
总线消息回调:
void Player::onBusMessage(const QGst::MessagePtr & message)
{
QGst::ElementPtr source = message->source().staticCast<QGst::Element>();
switch (message->type()) {
case QGst::MessageEos: { //End of stream. We reached the end of the file.
qDebug("Message End Off Stream");
if (m_player->inSnapshotCaputre) {
blockSignals(true);
pause();
play();
blockSignals(false);
m_player->inSnapshotCaputre = false;
}
else {
m_eos = true;
stop();
}
break;
}
[...]
}
问题是:
- 当我将
snapshot
属性 设置为pngenc
元素的true
时,我收到停止我的管道的EOS
事件,所以我需要重新启动它,这会冻结视频播放大约半秒钟,这对我来说是不可接受的。 - 当我将
snapshot
属性 设置为pngenc
元素的false
时,我没有管道扰动,但我的 png 文件一直在增长,直到我再次调用Player::takeSnapshot()
方法。
我哪里错了?有更好的方法吗?
我尝试为我的 snapshot
分支创建 QGst::Bin
元素但未成功。焊盘探针呢?
提前致谢
您可以在任何水槽上采集最后一个样本 属性,例如您的视频接收器。这包含一个 GstSample,它有一个缓冲区,其中包含最新的视频帧。您可以将其作为快照,例如使用 gst_video_convert_sample() 或其异步变体,将其转换为 PNG/JPG/whatever.
见https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-libs/html/GstBaseSink.html#GstBaseSink--last-sample and https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-libs/html/gst-plugins-base-libs-gstvideo.html#gst-video-convert-sample
或者,您必须在第一帧之后关闭 filesink 快照管道。例如,通过 pad 探测来了解第一帧发生的时间,然后注入 EOS 事件以防止将更多 PNG 帧附加到同一文件。
感谢@sebastian-droge 的回答,我找到了解决方案,使用 gst_video_convert_sample
和我的视频接收器的 last-sample
属性。
我实施的解决方案是:
void Player::takeSnapshot()
{
QDateTime currentDate = QDateTime::currentDateTime();
QString location = QString("%1/snap_%2.png").arg(QDir::homePath()).arg(currentDate.toString(Qt::ISODate));
QImage snapShot;
QImage::Format snapFormat;
QGlib::Value val = m_videoSink->property("last-sample");
GstSample *videoSample = (GstSample *)g_value_get_boxed(val);
QGst::SamplePtr sample = QGst::SamplePtr::wrap(videoSample);
QGst::SamplePtr convertedSample;
QGst::BufferPtr buffer;
QGst::CapsPtr caps = sample->caps();
QGst::MapInfo mapInfo;
GError *err = NULL;
GstCaps * capsTo = NULL;
const QGst::StructurePtr structure = caps->internalStructure(0);
int width, height;
width = structure.data()->value("width").get<int>();
height = structure.data()->value("height").get<int>();
qDebug() << "Sample caps:" << structure.data()->toString();
/*
* { QImage::Format_RGBX8888, GST_VIDEO_FORMAT_RGBx },
* { QImage::Format_RGBA8888, GST_VIDEO_FORMAT_RGBA },
* { QImage::Format_RGB888 , GST_VIDEO_FORMAT_RGB },
* { QImage::Format_RGB16 , GST_VIDEO_FORMAT_RGB16 }
*/
snapFormat = QImage::Format_RGB888;
capsTo = gst_caps_new_simple("video/x-raw",
"format", G_TYPE_STRING, "RGB",
"width", G_TYPE_INT, width,
"height", G_TYPE_INT, height,
NULL);
convertedSample = QGst::SamplePtr::wrap(gst_video_convert_sample(videoSample, capsTo, GST_SECOND, &err));
if (convertedSample.isNull()) {
qWarning() << "gst_video_convert_sample Failed:" << err->message;
}
else {
qDebug() << "Converted sample caps:" << convertedSample->caps()->toString();
buffer = convertedSample->buffer();
buffer->map(mapInfo, QGst::MapRead);
snapShot = QImage((const uchar *)mapInfo.data(),
width,
height,
snapFormat);
qDebug() << "Saving snap to" << location;
snapShot.save(location);
buffer->unmap(mapInfo);
}
val.clear();
sample.clear();
convertedSample.clear();
buffer.clear();
caps.clear();
g_clear_error(&err);
if (capsTo)
gst_caps_unref(capsTo);
}
我创建了一个简单的测试应用程序,它实现了这个解决方案。该代码可在我的 Github