使用 openCV 和 Python 将对象从一个图像映射到另一个图像
Map an object from one image to another image using openCV and Python
这是一个关于使用 openCV(版本 4.5.1.48)和 Python(版本 3.8.5)进行立体校准和校正的问题。
我有两个相机放置在同一个轴上,如下图所示:
左(上)摄像头以640x480分辨率拍照,右(下)摄像头以320x240分辨率拍照。
目标是在右图 (320x240) 上找到一个对象,并在左图 (640x480) 上裁剪出相同的对象。换一种说法;将构成右图中对象的矩形转移到左图中。下面概述了这个想法。
在右图中发现一个红色物体,我需要将它的位置转移到左图中并将其裁剪掉。物体放置在距相机镜头 30 厘米的平面上。换一种说法;两个相机镜头到平面的距离(深度)是恒定的(30cm)。
这个主要问题是关于如何将一个位置从一个图像转移到另一个图像,当两个相机并排放置时,当图像具有不同的分辨率并且深度(相当)恒定时。不是找对象的问题
要解决这个问题,据我所知,必须使用立体校准,我发现了以下articles/code,其中包括:
- https://github.com/bvnayak/stereo_calibration/blob/master/camera_calibrate.py
- https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_calib3d/py_calibration/py_calibration.html
- https://python.plainenglish.io/the-depth-i-stereo-calibration-and-rectification-24da7b0fb1e0
以下是我使用的校准模式示例:
我有25张左右相机标定图案的照片。图案为 5x9,正方形尺寸为 40x40 毫米。
根据我的知识,我编写了以下代码:
import numpy as np
import cv2
import glob
CALIL = "path-to-left-images"
CALIR = "path-to-right-images"
# Termination criterias
criteria1 = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
criteria2 = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 1e-5)
# Chessboard parameters
checker_size = 40.0 # Square size in world units (mm)
checker_pattern = (5, 9) # 5 rows, 9 columns
# Flags
findChessboardCorners_flags = 0
#findChessboardCorners_flags |= cv2.CALIB_CB_ADAPTIVE_THRESH
#findChessboardCorners_flags |= cv2.CALIB_CB_NORMALIZE_IMAGE
#findChessboardCorners_flags |= cv2.CALIB_CB_FILTER_QUADS
#findChessboardCorners_flags |= cv2.CALIB_CB_FAST_CHECK
calibrateCamera_flags = 0
#calibrateCamera_flags |= cv2.CALIB_USE_INTRINSIC_GUESS
#calibrateCamera_flags |= cv2.CALIB_FIX_PRINCIPAL_POINT
#calibrateCamera_flags |= cv2.CALIB_FIX_ASPECT_RATIO
#calibrateCamera_flags |= cv2.CALIB_ZERO_TANGENT_DIST
#calibrateCamera_flags |= cv2.CALIB_FIX_K1 # K2, K3...K6
#calibrateCamera_flags |= cv2.CALIB_RATIONAL_MODEL
#calibrateCamera_flags |= cv2.CALIB_THIN_PRISM_MODEL
#calibrateCamera_flags |= cv2.CALIB_FIX_S1_S2_S3_S4
#calibrateCamera_flags |= cv2.CALIB_TILTED_MODEL
#calibrateCamera_flags |= cv2.CALIB_FIX_TAUX_TAUY
stereoCalibrate_falgs = 0
stereoCalibrate_falgs |= cv2.CALIB_FIX_INTRINSIC
#stereoCalibrate_falgs |= cv2.CALIB_USE_INTRINSIC_GUESS
#stereoCalibrate_falgs |= cv2.CALIB_USE_EXTRINSIC_GUESS
#stereoCalibrate_falgs |= cv2.CALIB_FIX_PRINCIPAL_POINT
#stereoCalibrate_falgs |= cv2.CALIB_FIX_FOCAL_LENGTH
#stereoCalibrate_falgs |= cv2.CALIB_FIX_ASPECT_RATIO
#stereoCalibrate_falgs |= cv2.CALIB_SAME_FOCAL_LENGTH
#stereoCalibrate_falgs |= cv2.CALIB_ZERO_TANGENT_DIST
#stereoCalibrate_falgs |= cv2.CALIB_FIX_K1 # K2, K3...K6
#stereoCalibrate_falgs |= cv2.CALIB_RATIONAL_MODEL
#stereoCalibrate_falgs |= cv2.CALIB_THIN_PRISM_MODEL
#stereoCalibrate_falgs |= cv2.CALIB_FIX_S1_S2_S3_S4
#stereoCalibrate_falgs |= cv2.CALIB_TILTED_MODEL
#stereoCalibrate_falgs |= cv2.CALIB_FIX_TAUX_TAUY
stereoRectify_flags = 0
stereoRectify_flags |= cv2.CALIB_ZERO_DISPARITY
# Prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
objp = np.zeros((1, checker_pattern[0] * checker_pattern[1], 3), np.float32)
objp[0, :, :2] = np.mgrid[0:checker_pattern[0],
0:checker_pattern[1]].T.reshape(-1, 2)*checker_size
# Arrays to store object points and image points from all the images.
objPoints = [] # 3d point in real world space
imgPointsL = [] # 2d points in image plane, left image (normal)
imgPointsR = [] # 2d points in image plane, right image (thermal)
# Get calibration images
# Get all left (normal) images from directory. Sort them
images_left = glob.glob(CALIL+'*')
images_left.sort()
# Get all right (thermal) images from directory. Sort them
images_right = glob.glob(CALIR+'*')
images_right.sort()
for left_img, right_img in zip(images_left, images_right):
# Left object points
imgL = cv2.imread(left_img)
grayL = cv2.cvtColor(imgL, cv2.COLOR_BGR2GRAY)
# Find the chessboard corners
retL, cornersL = cv2.findChessboardCorners(
grayL, (checker_pattern[0], checker_pattern[1]), findChessboardCorners_flags)
# Right object points
imgR = cv2.imread(right_img)
grayR = cv2.cvtColor(imgR, cv2.COLOR_BGR2GRAY)
# Find the chessboard corners
retR, cornersR = cv2.findChessboardCorners(
grayR, (checker_pattern[0], checker_pattern[1]), findChessboardCorners_flags)
if retL and retR:
# If found, add object points, image points (after refining them)
objPoints.append(objp)
# Left points
cornersL2 = cv2.cornerSubPix(
grayL, cornersL, (5, 5), (-1, -1), criteria1)
imgPointsL.append(cornersL2)
# Right points
cornersR2 = cv2.cornerSubPix(
grayR, cornersR, (5, 5), (-1, -1), criteria1)
imgPointsR.append(cornersR2)
shapeL = grayL.shape[::-1]
shapeR = grayR.shape[::-1]
# Calibrate each camera separately
retL, K1, D1, R1, T1 = cv2.calibrateCamera(
objPoints, imgPointsL, shapeL, None, None, flags=calibrateCamera_flags)
retR, K2, D2, R2, T2 = cv2.calibrateCamera(
objPoints, imgPointsR, shapeR, None, None, flags=calibrateCamera_flags)
# Stereo calibrate
ret, K1, D1, K2, D2, R, T, E, F = cv2.stereoCalibrate(
objPoints, imgPointsL, imgPointsR, K1, D1, K2, D2, shapeR, flags=calibrateCamera_flags, criteria=criteria2)
# Stereo rectify
R1, R2, P1, P2, Q, roi_left, roi_right = cv2.stereoRectify(
K1, D1, K2, D2, shapeR, R, T, flags=stereoRectify_flags, alpha=1)
# Undistort images
leftMapX, leftMapY = cv2.initUndistortRectifyMap(
K1, D1, R1, P1, shapeL, cv2.CV_32FC1)
rightMapX, rightMapY = cv2.initUndistortRectifyMap(
K2, D2, R2, P2, shapeR, cv2.CV_32FC1)
# Remap
left_rectified = cv2.remap(images_left[0], leftMapX, leftMapY,
cv2.INTER_LINEAR, cv2.BORDER_CONSTANT)
right_rectified = cv2.remap(images_right[0], rightMapX, rightMapY,
cv2.INTER_LINEAR, cv2.BORDER_CONSTANT)
但是我得到了一个糟糕的结果:
我尝试了不同的标志、alpha 参数,但没有任何效果...
问题:
- 当两个图像具有不同的分辨率时,甚至可以立体校准并解决这个问题吗?
- 一般的工作流程是正确的还是我遗漏了什么?旗帜?阿尔法参数?解决这个问题的其他方法?
编辑
在 Micha 的精彩评论之后,我发现透视单应性是(希望)解决这个问题的方法,而不是立体校准。这是因为需要找到的物体放置在距离两个相机镜头 (30cm) 恒定 length/depth 的平面上。
根据新的信息,我编写了以下代码,其中我使用了第一对图像来获取透视变换矩阵:
imgL = cv2.imread(images_left[0])
imgL = cv2.cvtColor(imgL, cv2.COLOR_BGR2GRAY)
imgR = cv2.imread(images_right[0])
imgR = cv2.cvtColor(imgR, cv2.COLOR_BGR2GRAY)
ret1, corners1 = cv2.findChessboardCorners(imgL, (checker_pattern[0], checker_pattern[1]))
cornersL2 = cv2.cornerSubPix(imgL, corners1, (5, 5), (-1, -1), criteria1)
ret2, corners2 = cv2.findChessboardCorners(imgR, (checker_pattern[0], checker_pattern[1]))
cornersR2 = cv2.cornerSubPix(imgR, corners2, (5, 5), (-1, -1), criteria1)
H, _ = cv2.findHomography(cornersL2, cornersR2)
基于透视变换矩阵H,我可以使用cv2.warpPerspective()
函数根据右图和校准板中的棋盘角扭曲左图。
但是,当我尝试扭曲它时,扭曲的图像(下图上方)相对于另一张(下方)图像有点偏右,如下图所示:
裁剪后的结果如下所示,其中的区域不匹配:
我想我需要调整变形图像的大小,使其与右图 (320x240) 的分辨率相同。扭曲图像的分辨率为 640x240。
问题:
- 是否应该将校准板放置在距相机镜头 30 厘米的位置,以优化透视变换矩阵的计算?
- 我有25张不同角度的标定板图片。是否需要使用所有图像,还是只使用一个图像?
- 我正在使用
cv2.warpPerspective()
功能,但裁剪不匹配。我应该使用其他功能吗?
我使用以下 openCV 函数解决了这个问题:
cv2.findChessboardCorners()
cv2.cornerSubPix()
cv2.findHomography()
cv2.warpPerspective()
我用距离30cm的标定板来计算透视变换矩阵H。因此,我可以将一个物体从右图映射到左图。虽然深度必须恒定(30 厘米),这有点问题,但在我的情况下是可以接受的。
感谢@Micka 的精彩回答。
这是一个关于使用 openCV(版本 4.5.1.48)和 Python(版本 3.8.5)进行立体校准和校正的问题。
我有两个相机放置在同一个轴上,如下图所示:
左(上)摄像头以640x480分辨率拍照,右(下)摄像头以320x240分辨率拍照。 目标是在右图 (320x240) 上找到一个对象,并在左图 (640x480) 上裁剪出相同的对象。换一种说法;将构成右图中对象的矩形转移到左图中。下面概述了这个想法。
在右图中发现一个红色物体,我需要将它的位置转移到左图中并将其裁剪掉。物体放置在距相机镜头 30 厘米的平面上。换一种说法;两个相机镜头到平面的距离(深度)是恒定的(30cm)。
这个主要问题是关于如何将一个位置从一个图像转移到另一个图像,当两个相机并排放置时,当图像具有不同的分辨率并且深度(相当)恒定时。不是找对象的问题
要解决这个问题,据我所知,必须使用立体校准,我发现了以下articles/code,其中包括:
- https://github.com/bvnayak/stereo_calibration/blob/master/camera_calibrate.py
- https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_calib3d/py_calibration/py_calibration.html
- https://python.plainenglish.io/the-depth-i-stereo-calibration-and-rectification-24da7b0fb1e0
以下是我使用的校准模式示例:
我有25张左右相机标定图案的照片。图案为 5x9,正方形尺寸为 40x40 毫米。
根据我的知识,我编写了以下代码:
import numpy as np
import cv2
import glob
CALIL = "path-to-left-images"
CALIR = "path-to-right-images"
# Termination criterias
criteria1 = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
criteria2 = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 1e-5)
# Chessboard parameters
checker_size = 40.0 # Square size in world units (mm)
checker_pattern = (5, 9) # 5 rows, 9 columns
# Flags
findChessboardCorners_flags = 0
#findChessboardCorners_flags |= cv2.CALIB_CB_ADAPTIVE_THRESH
#findChessboardCorners_flags |= cv2.CALIB_CB_NORMALIZE_IMAGE
#findChessboardCorners_flags |= cv2.CALIB_CB_FILTER_QUADS
#findChessboardCorners_flags |= cv2.CALIB_CB_FAST_CHECK
calibrateCamera_flags = 0
#calibrateCamera_flags |= cv2.CALIB_USE_INTRINSIC_GUESS
#calibrateCamera_flags |= cv2.CALIB_FIX_PRINCIPAL_POINT
#calibrateCamera_flags |= cv2.CALIB_FIX_ASPECT_RATIO
#calibrateCamera_flags |= cv2.CALIB_ZERO_TANGENT_DIST
#calibrateCamera_flags |= cv2.CALIB_FIX_K1 # K2, K3...K6
#calibrateCamera_flags |= cv2.CALIB_RATIONAL_MODEL
#calibrateCamera_flags |= cv2.CALIB_THIN_PRISM_MODEL
#calibrateCamera_flags |= cv2.CALIB_FIX_S1_S2_S3_S4
#calibrateCamera_flags |= cv2.CALIB_TILTED_MODEL
#calibrateCamera_flags |= cv2.CALIB_FIX_TAUX_TAUY
stereoCalibrate_falgs = 0
stereoCalibrate_falgs |= cv2.CALIB_FIX_INTRINSIC
#stereoCalibrate_falgs |= cv2.CALIB_USE_INTRINSIC_GUESS
#stereoCalibrate_falgs |= cv2.CALIB_USE_EXTRINSIC_GUESS
#stereoCalibrate_falgs |= cv2.CALIB_FIX_PRINCIPAL_POINT
#stereoCalibrate_falgs |= cv2.CALIB_FIX_FOCAL_LENGTH
#stereoCalibrate_falgs |= cv2.CALIB_FIX_ASPECT_RATIO
#stereoCalibrate_falgs |= cv2.CALIB_SAME_FOCAL_LENGTH
#stereoCalibrate_falgs |= cv2.CALIB_ZERO_TANGENT_DIST
#stereoCalibrate_falgs |= cv2.CALIB_FIX_K1 # K2, K3...K6
#stereoCalibrate_falgs |= cv2.CALIB_RATIONAL_MODEL
#stereoCalibrate_falgs |= cv2.CALIB_THIN_PRISM_MODEL
#stereoCalibrate_falgs |= cv2.CALIB_FIX_S1_S2_S3_S4
#stereoCalibrate_falgs |= cv2.CALIB_TILTED_MODEL
#stereoCalibrate_falgs |= cv2.CALIB_FIX_TAUX_TAUY
stereoRectify_flags = 0
stereoRectify_flags |= cv2.CALIB_ZERO_DISPARITY
# Prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
objp = np.zeros((1, checker_pattern[0] * checker_pattern[1], 3), np.float32)
objp[0, :, :2] = np.mgrid[0:checker_pattern[0],
0:checker_pattern[1]].T.reshape(-1, 2)*checker_size
# Arrays to store object points and image points from all the images.
objPoints = [] # 3d point in real world space
imgPointsL = [] # 2d points in image plane, left image (normal)
imgPointsR = [] # 2d points in image plane, right image (thermal)
# Get calibration images
# Get all left (normal) images from directory. Sort them
images_left = glob.glob(CALIL+'*')
images_left.sort()
# Get all right (thermal) images from directory. Sort them
images_right = glob.glob(CALIR+'*')
images_right.sort()
for left_img, right_img in zip(images_left, images_right):
# Left object points
imgL = cv2.imread(left_img)
grayL = cv2.cvtColor(imgL, cv2.COLOR_BGR2GRAY)
# Find the chessboard corners
retL, cornersL = cv2.findChessboardCorners(
grayL, (checker_pattern[0], checker_pattern[1]), findChessboardCorners_flags)
# Right object points
imgR = cv2.imread(right_img)
grayR = cv2.cvtColor(imgR, cv2.COLOR_BGR2GRAY)
# Find the chessboard corners
retR, cornersR = cv2.findChessboardCorners(
grayR, (checker_pattern[0], checker_pattern[1]), findChessboardCorners_flags)
if retL and retR:
# If found, add object points, image points (after refining them)
objPoints.append(objp)
# Left points
cornersL2 = cv2.cornerSubPix(
grayL, cornersL, (5, 5), (-1, -1), criteria1)
imgPointsL.append(cornersL2)
# Right points
cornersR2 = cv2.cornerSubPix(
grayR, cornersR, (5, 5), (-1, -1), criteria1)
imgPointsR.append(cornersR2)
shapeL = grayL.shape[::-1]
shapeR = grayR.shape[::-1]
# Calibrate each camera separately
retL, K1, D1, R1, T1 = cv2.calibrateCamera(
objPoints, imgPointsL, shapeL, None, None, flags=calibrateCamera_flags)
retR, K2, D2, R2, T2 = cv2.calibrateCamera(
objPoints, imgPointsR, shapeR, None, None, flags=calibrateCamera_flags)
# Stereo calibrate
ret, K1, D1, K2, D2, R, T, E, F = cv2.stereoCalibrate(
objPoints, imgPointsL, imgPointsR, K1, D1, K2, D2, shapeR, flags=calibrateCamera_flags, criteria=criteria2)
# Stereo rectify
R1, R2, P1, P2, Q, roi_left, roi_right = cv2.stereoRectify(
K1, D1, K2, D2, shapeR, R, T, flags=stereoRectify_flags, alpha=1)
# Undistort images
leftMapX, leftMapY = cv2.initUndistortRectifyMap(
K1, D1, R1, P1, shapeL, cv2.CV_32FC1)
rightMapX, rightMapY = cv2.initUndistortRectifyMap(
K2, D2, R2, P2, shapeR, cv2.CV_32FC1)
# Remap
left_rectified = cv2.remap(images_left[0], leftMapX, leftMapY,
cv2.INTER_LINEAR, cv2.BORDER_CONSTANT)
right_rectified = cv2.remap(images_right[0], rightMapX, rightMapY,
cv2.INTER_LINEAR, cv2.BORDER_CONSTANT)
但是我得到了一个糟糕的结果:
我尝试了不同的标志、alpha 参数,但没有任何效果...
问题:
- 当两个图像具有不同的分辨率时,甚至可以立体校准并解决这个问题吗?
- 一般的工作流程是正确的还是我遗漏了什么?旗帜?阿尔法参数?解决这个问题的其他方法?
编辑
在 Micha 的精彩评论之后,我发现透视单应性是(希望)解决这个问题的方法,而不是立体校准。这是因为需要找到的物体放置在距离两个相机镜头 (30cm) 恒定 length/depth 的平面上。
根据新的信息,我编写了以下代码,其中我使用了第一对图像来获取透视变换矩阵:
imgL = cv2.imread(images_left[0])
imgL = cv2.cvtColor(imgL, cv2.COLOR_BGR2GRAY)
imgR = cv2.imread(images_right[0])
imgR = cv2.cvtColor(imgR, cv2.COLOR_BGR2GRAY)
ret1, corners1 = cv2.findChessboardCorners(imgL, (checker_pattern[0], checker_pattern[1]))
cornersL2 = cv2.cornerSubPix(imgL, corners1, (5, 5), (-1, -1), criteria1)
ret2, corners2 = cv2.findChessboardCorners(imgR, (checker_pattern[0], checker_pattern[1]))
cornersR2 = cv2.cornerSubPix(imgR, corners2, (5, 5), (-1, -1), criteria1)
H, _ = cv2.findHomography(cornersL2, cornersR2)
基于透视变换矩阵H,我可以使用cv2.warpPerspective()
函数根据右图和校准板中的棋盘角扭曲左图。
但是,当我尝试扭曲它时,扭曲的图像(下图上方)相对于另一张(下方)图像有点偏右,如下图所示:
裁剪后的结果如下所示,其中的区域不匹配:
我想我需要调整变形图像的大小,使其与右图 (320x240) 的分辨率相同。扭曲图像的分辨率为 640x240。
问题:
- 是否应该将校准板放置在距相机镜头 30 厘米的位置,以优化透视变换矩阵的计算?
- 我有25张不同角度的标定板图片。是否需要使用所有图像,还是只使用一个图像?
- 我正在使用
cv2.warpPerspective()
功能,但裁剪不匹配。我应该使用其他功能吗?
我使用以下 openCV 函数解决了这个问题:
cv2.findChessboardCorners()
cv2.cornerSubPix()
cv2.findHomography()
cv2.warpPerspective()
我用距离30cm的标定板来计算透视变换矩阵H。因此,我可以将一个物体从右图映射到左图。虽然深度必须恒定(30 厘米),这有点问题,但在我的情况下是可以接受的。
感谢@Micka 的精彩回答。