使用 OpenCV 图像处理在 Project Tango 上呈现问题

Question

我遇到了一个问题，即在对其 YUV 缓冲区进行一些处理后渲染相机图像。

我正在使用示例 video-overlay-jni-example 并且在方法 OnFrameAvailable 中我正在使用 cv::Mat 创建一个新的帧缓冲区...

以下是我创建新帧缓冲区的方法：

cv::Mat frame((int) yuv_height_ + (int) (yuv_height_ / 2), (int) yuv_width_, CV_8UC1, (uchar *) yuv_temp_buffer_.data());

处理后，我将 frame.data 复制到 yuv_temp_buffer_ 以便在纹理上渲染它：memcpy(&yuv_temp_buffer_[0], frame.data, yuv_size_);

这很好用...

当我尝试执行 OpenCV 方法时出现问题 findChessboardCorners... 使用我之前创建的框架。

方法 findChessboardCorners 执行大约需要 90 毫秒 (11 fps)，但是，它的渲染速度似乎较慢。（它似乎在屏幕上以 ~0.5 fps 的速度呈现）。

这里是OnFrameAvailable方法的代码：

void AugmentedRealityApp::OnFrameAvailable(const TangoImageBuffer* buffer) {

    if (yuv_drawable_ == NULL){
        return;
    }

    if (yuv_drawable_->GetTextureId() == 0) {
        LOGE("AugmentedRealityApp::yuv texture id not valid");
        return;
    }

    if (buffer->format != TANGO_HAL_PIXEL_FORMAT_YCrCb_420_SP) {
        LOGE("AugmentedRealityApp::yuv texture format is not supported by this app");
        return;
    }

    // The memory needs to be allocated after we get the first frame because we
    // need to know the size of the image.
    if (!is_yuv_texture_available_) {
        yuv_width_ = buffer->width;
        yuv_height_ = buffer->height;
        uv_buffer_offset_ = yuv_width_ * yuv_height_;

        yuv_size_ = yuv_width_ * yuv_height_ + yuv_width_ * yuv_height_ / 2;

        // Reserve and resize the buffer size for RGB and YUV data.
        yuv_buffer_.resize(yuv_size_);
        yuv_temp_buffer_.resize(yuv_size_);
        rgb_buffer_.resize(yuv_width_ * yuv_height_ * 3);

        AllocateTexture(yuv_drawable_->GetTextureId(), yuv_width_, yuv_height_);
        is_yuv_texture_available_ = true;
    }

    std::lock_guard<std::mutex> lock(yuv_buffer_mutex_);
    memcpy(&yuv_temp_buffer_[0], buffer->data, yuv_size_);

    ///
    cv::Mat frame((int) yuv_height_ + (int) (yuv_height_ / 2), (int) yuv_width_, CV_8UC1, (uchar *) yuv_temp_buffer_.data());

    if (!stam.isCalibrated()) {
        Profiler profiler;
        profiler.startSampling();
        stam.initFromChessboard(frame, cv::Size(9, 6), 100);
        profiler.endSampling();
        profiler.print("initFromChessboard", -1);
    }
    ///

    memcpy(&yuv_temp_buffer_[0], frame.data, yuv_size_);
    swap_buffer_signal_ = true;
}

这里是方法的代码initFromChessBoard:

bool STAM::initFromChessboard(const cv::Mat& image, const cv::Size& chessBoardSize, int squareSize)
{
    cv::Mat rvec = cv::Mat(cv::Size(3, 1), CV_64F);
    cv::Mat tvec = cv::Mat(cv::Size(3, 1), CV_64F);

    std::vector<cv::Point2d> imagePoints, imageBoardPoints;
    std::vector<cv::Point3d> boardPoints;

    for (int i = 0; i < chessBoardSize.height; i++)
    {
        for (int j = 0; j < chessBoardSize.width; j++)
        {
            boardPoints.push_back(cv::Point3d(j*squareSize, i*squareSize, 0.0));
        }
    }

    //getting only the Y channel (many of the functions like face detect and align only needs the grayscale image)
    cv::Mat gray(image.rows, image.cols, CV_8UC1);
    gray.data = image.data;

    bool found = findChessboardCorners(gray, chessBoardSize, imagePoints, cv::CALIB_CB_FAST_CHECK);

#ifdef WINDOWS_VS
    printf("Number of chessboard points: %d\n", imagePoints.size());
#elif ANDROID
    LOGE("Number of chessboard points: %d", imagePoints.size());
#endif

    for (int i = 0; i < imagePoints.size(); i++) {
        cv::circle(image, imagePoints[i], 6, cv::Scalar(149, 43, 0), -1);
    }
}

有人在处理 YUV 缓冲区中的内容以在纹理上渲染后遇到同样的问题吗？

我使用其他设备而不是使用 camera2 的 Tango 项目进行了测试 API，屏幕上的渲染过程似乎与 OpenCV 函数过程本身的速率相同。

感谢任何帮助。

Answer 1

我有一个 . My app slowed down after using the copied yuv buffer and doing some image processing with OpenCV. I would recommand you to use the tango_support library 通过执行以下操作来访问 yuv 图像缓冲区：

在你的配置函数中：

int AugmentedRealityApp::TangoSetupConfig() {
   TangoSupport_createImageBufferManager(TANGO_HAL_PIXEL_FORMAT_YCrCb_420_SP, 1280, 720, &yuv_manager_);
}

在你的回调函数中：

void AugmentedRealityApp::OnFrameAvailable(const TangoImageBuffer* buffer) {
   TangoSupport_updateImageBuffer(yuv_manager_, buffer);
}

在您的渲染线程中：

void AugmentedRealityApp::Render() {
   TangoImageBuffer* yuv = new TangoImageBuffer();
   TangoSupport_getLatestImageBuffer(yuv_manager_, &yuv);
   cv::Mat yuv_frame, rgb_img, gray_img;
   yuv_frame.create(720*3/2, 1280, CV_8UC1);
   memcpy(yuv_frame.data, yuv->data, 720*3/2*1280);  // yuv image
   cv::cvtColor(yuv_frame, rgb_img, CV_YUV2RGB_NV21); // rgb image
   cvtColor(rgb_img, gray_img, CV_RGB2GRAY); // gray image
}

您可以与其他人共享 yuv_manger objects/threads 这样您就可以随时随地访问 yuv 图像缓冲区。

使用 OpenCV 图像处理在 Project Tango 上呈现问题

Rendering issue on Project Tango using OpenCV image processing

android

yuv

android-camera

google-project-tango