读取 .oni 文件会导致乱序帧

Question

我的问题与 this one 重复。那里没有解决办法。为了找到解决方案并详细说明我的具体设置，下面显示了用于从 .oni 文件读取帧的函数。如果此函数是运行且 Type == 2（即运行s for # of RGBD images，其中 Criteria 是 #），运行在 for 循环中使用此函数应该允许用户访问每个图像。但是，彩色图像和深度图像的索引不匹配并且乱序。这一直持续到 waitForAnyStream 对以下所有 IMG_pipeline::listen(...).

的调用超时

void IMG_pipeline::listen(int Type, int Criteria){
    int exitNumber;
    clock_t start = clock();
    double elapsedtime;
    openni::VideoFrameRef frame;
    int CurrentIMGCount=0;

    switch (Type){
    case 0:
    {
        exitNumber = -1;
        break;
    }
    case 1:
    {
        exitNumber = Criteria;
        break;
    }
    case 2:
    {
        exitNumber = -1;
        break;
    }
    }


    for (int i = 0;i!=exitNumber;i++){
        readyStream = -1;
        rc = openni::OpenNI::waitForAnyStream(streams, 2, &readyStream, SAMPLE_READ_WAIT_TIMEOUT);
        if (rc != openni::STATUS_OK)
        {
            printf("Wait failed! (timeout is %d ms)\n%s\n", SAMPLE_READ_WAIT_TIMEOUT, openni::OpenNI::getExtendedError());
            //break;
        }

        switch (readyStream)
        {
        case 0:
        {
            // Depth
            depth.readFrame(&frame);
            break;
        }
        case 1:
        {
            // Color
            color.readFrame(&frame);
            break;
        }
        default:
        {
            printf("Unexpected stream: %i\n", readyStream);
            continue;
        }
        }


        int Height = frame.getHeight();
        int Width = frame.getWidth();

        cvColor.release();
        cvX.release();
        cvY.release();
        cvZ.release();

        cvColor = cv::Mat(Height, Width, CV_8UC3);
        cvX = cv::Mat(Height, Width, CV_32F);
        cvY = cv::Mat(Height, Width, CV_32F);
        cvZ = cv::Mat(Height, Width, CV_32F);

        switch (frame.getVideoMode().getPixelFormat())
        {
        case openni::PIXEL_FORMAT_DEPTH_1_MM:
        case openni::PIXEL_FORMAT_DEPTH_100_UM:
        {
            openni::DepthPixel* pDepth = (openni::DepthPixel*)frame.getData();
            int k =0;
            for (int ri = 0; ri<Height; ri++)
            {
                for (int ci = 0; ci<Width; ci++)
                {
                    float pdepth_val = pDepth[k];
                    openni::CoordinateConverter::convertDepthToWorld(depth, (float)ri, (float)ci, pdepth_val, &cvX.at<float>(ri,ci), &cvY.at<float>(ri,ci), &cvZ.at<float>(ri,ci));
                    k++;
                }
            }
            TotalFrames[0]++;
            XYZCaptured = true;
            printf("Frame Index: %i \n", frame.getFrameIndex());
            printf("Depth Captured. \n");
            break;
        }
        case openni::PIXEL_FORMAT_RGB888:
        {
            cvColor.data = (uchar*)frame.getData();
            TotalFrames[1]++;
            ColorCaptured = true;
            printf("Frame Index: %i \n", frame.getFrameIndex());
            printf("Color Captured. \n");
            break;
        }
        default:
            printf("Unknown format \n");
        }

        printf("Frame extracted. \n");
        if (ColorCaptured && XYZCaptured){
            if (NewButNotRead == true){
                IMGsMissed++;


            }
            else
                NewButNotRead = true;

            ColorCaptured = false;
            XYZCaptured = false;
            RGBD_out.clear();
            RGBD_out.push_back(cvX);
            RGBD_out.push_back(cvY);
            RGBD_out.push_back(cvZ);
            RGBD_out.push_back(cvColor);
            CurrentIMGCount++;
            printf("Image overwritten. \n");

        }
        elapsedtime=(clock()-start)/((double)CLOCKS_PER_SEC);

        printf("Time since listen initiation: %f \n \n", elapsedtime);
        if (CurrentIMGCount ==Criteria && Type == 2)
            return;
        else if (elapsedtime>(double)Criteria && Type==0)
            return;

    }
    frame.release();

}

这是一个控制台输出示例：

Frame Index: 1 
Depth Captured. 
Frame extracted. 
Time since listen initiation: 0.004846 

Frame Index: 2 
Depth Captured. 
Frame extracted. 
Time since listen initiation: 0.011601 

Frame Index: 1 
Color Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.012640 

Depth frame count: 3 
Color frame count: 2 
Frame Index: 54 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000067 

Frame Index: 57 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.005878 

Depth frame count: 4 
Color frame count: 3 
Frame Index: 96 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000079 

Frame Index: 99 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.003628 

Depth frame count: 5 
Color frame count: 4 
Frame Index: 126 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000048 

Frame Index: 130 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.004782 

Depth frame count: 6 
Color frame count: 5 
Frame Index: 152 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000065 

Frame Index: 156 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.008294 

Depth frame count: 7 
Color frame count: 6 
Frame Index: 181 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000045 

Frame Index: 185 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.004095 

Depth frame count: 8 
Color frame count: 7 
Frame Index: 208 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000054 

Frame Index: 212 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.004242 

Depth frame count: 9 
Color frame count: 8 
Frame Index: 236 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000092 

Frame Index: 240 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.005918 

Depth frame count: 10 
Color frame count: 9 
Frame Index: 261 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000731 

Frame Index: 262 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000877 

Frame Index: 266 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.009347 

Depth frame count: 11 
Color frame count: 11 
Frame Index: 286 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000047 

Frame Index: 290 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.006080 

Depth frame count: 12 
Color frame count: 12 
Frame Index: 311 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000072 

Frame Index: 315 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.006453 

Depth frame count: 13 
Color frame count: 13 
Frame Index: 337 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000062 

Frame Index: 341 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.007485 

Depth frame count: 14 
Color frame count: 14 
Frame Index: 367 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000042 

Frame Index: 371 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.003758 

Depth frame count: 15 
Color frame count: 15 
Frame Index: 390 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000073 

Frame Index: 395 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.007917 

Depth frame count: 16 
Color frame count: 16 
Frame Index: 416 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000105 

Frame Index: 421 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.007554 

Depth frame count: 17 
Color frame count: 17 
Frame Index: 453 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000060 

Frame Index: 458 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.006150 

Depth frame count: 18 
Color frame count: 18 
Frame Index: 481 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000074 

Frame Index: 486 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.007169 

Depth frame count: 19 
Color frame count: 19 
Frame Index: 517 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000045 

Frame Index: 522 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.004196 

Depth frame count: 20 
Color frame count: 20 
Frame Index: 547 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000071 

Frame Index: 552 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.007375 

Depth frame count: 21 
Color frame count: 21 
Frame Index: 625 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000179 

Frame Index: 631 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.007922 

Depth frame count: 22 
Color frame count: 22 
Wait failed! (timeout is 2000 ms)
    waitForStreams: timeout reached

Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
    waitForStreams: timeout reached

Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
    waitForStreams: timeout reached

Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
    waitForStreams: timeout reached

Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
    waitForStreams: timeout reached

Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
    waitForStreams: timeout reached

Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
    waitForStreams: timeout reached

Unexpected stream: -1

这是对 IMG_pipeline::listen(...) 的调用：

IMG_pipeline pip_inst;
std::string FileName = "/home/derek/Test Data/RGBD/RGBD_S2_R1";
int Type = 2;
int Criteria = 1;
std::vector<cv::Mat> OUT;
int NumMissedIMGs;

int Start;
int Stop;

pip_inst.connect(FileName);

while (true)
{
  pip_inst.listen(Type, Criteria);
  if (pip_inst.IsNewIMG()){
    OUT = pip_inst.GetImage();
    cv::imshow("Current Frame", OUT.at(3));
    char c = cv::waitKey(0);
    if (c == 'f')
    {
      printf("Depth frame count: %i \n", pip_inst.GetDepthFrameCount());
      printf("Color frame count: %i \n", pip_inst.GetColorFrameCount());

    }
    else
    {
      Start = pip_inst.GetColorFrameCount();
      break;
    }
    cv::destroyWindow("Current Frame");

  }


}

彩色图像也是交替的 R、G、B 色调。但是，我确信数据在 cv::Mat 中的顺序存在问题。

更有趣的是，通过许多帧调用 IMG_pipeline::listen(...) 与运行ning IMG_pipeline::listen(...) 多个不同的索引结果次，通过 .oni 文件递增。

Answer 1

因此，由于编译器之间的 uchar 差异，RGB 信息没有被正确复制。所以RGB信息的case语句改成这样：

case openni::PIXEL_FORMAT_RGB888:
        {
            openni::RGB888Pixel* imgbuffer = (openni::RGB888Pixel*)frame.getData();
            //cvColor.data = (uchar*)imgbuffer;
            memcpy( cvColor.data, imgbuffer, 3*frame.getHeight()*frame.getWidth()*sizeof(uint8_t));
            cv::cvtColor(cvColor,cvColor,cv::COLOR_BGR2RGB);
            TotalFrames[1]++;
            ColorCaptured = true;
            printf("Frame Index: %i \n", frame.getFrameIndex());
            printf("Color Captured. \n");
            break;
        }

此外，为了确保我能够捕捉到每一帧，我放慢了文件播放速度。有趣的是，打开 .oni 文件的播放速度比抓取帧索引快（我只能假设这是捕获它的速度）。无论如何，这是用

完成的

Source.getPlaybackControl()->setSpeed(Ratio);

其中 Source 是我的设备，Ratio 是用户指定的浮点数。希望这对以后的人有帮助。

Answer 2

您可以使用 setSpeed command. Setting the speed to -1 will ensure that frames can be read manually in sequence from the oni stream i.e. each time you call waitForAnyStream you are guaranteed to get the next frame in the stream. See "Playback Speed" here 控制 oni 文件的播放速度以获取更多详细信息。

读取 .oni 文件会导致乱序帧

Reading from .oni file results in out of order frames

c++

linux

openni

asus-xtion