如何使用 python、opencv、orb 描述符和来自 opencv 的 ransac 制作一系列图片的全景视图

Question

我需要从一系列图片（3张图片）中制作全景图。之后我创建了 orb，检测并计算了三张图片的关键点和描述符，我匹配了最可能相似的关键点：

图片 1 和图片 2
图片 2 和图片 3

然后我知道计算和找到只有 2 个图像之间的全景视图，我在 img1 和 img2 之间，以及 img2 和 img3 之间进行计算。但是然后对于最后一步，我想使用来自 opencv 的 Ransac 算法的仿射变换找到这 3 张图片的全景图。而且我不知道这样做（3张图片的全景图。所以，我当然必须选择图像2作为全景图的中心

我没有找到一个好的解释或足够好的解释来计算和细化这 3 张图片的全景图。有人可以帮我实现我需要的东西吗？

这是我在 img1 和 img2 之间以及 img2 和 img3 之间打印全景图的代码：

import numpy as np
import cv2 as cv
from matplotlib import pyplot as plt
from matplotlib.pyplot import imshow, show, subplot, title, axis


# draw matches
def draw_matches(img1, kpt1, img2, kpt2, matches):
    h1, w1 = img1.shape[:2]
    h2, w2 = img2.shape[:2]

    # Create a blank image with the size of the first image + second image
    new_img = np.zeros((max([h1, h2]), w1 + w2, 3), dtype='uint8')
    new_img[:h1, :w1, :] = np.dstack([img1, img1, img1])
    new_img[:h2, w1:w1 + w2, :] = np.dstack([img2, img2, img2])

    # extract the match keypoints
    for m in matches:
        (x1, y1) = kpt1[m.queryIdx].pt
        (x2, y2) = kpt2[m.trainIdx].pt

        # Draw circles on the keypoints
        cv.circle(new_img, (int(x1), int(y1)), 4, (0, 255, 0), 1)
        cv.circle(new_img, (int(x2) + w1, int(y2)), 4, (0, 255, 0), 1)

        # Connect the same keypoints
        cv.line(new_img, (int(x1), int(y1)), (int(x2) + w1, int(y2)), (255, 0, 255), 1)

return new_img


def warpImages(img1, img2, M):
    # get the corner coordinates of the "query" and "train" image
    h1, w1 = img1.shape[:2]
    h2, w2 = img2.shape[:2]
    pts_corners_src = np.float32([[0, 0], [0, h1], [w1, h1], [w1, 0]]).reshape(-1, 1, 2)
    pts_corners_temp = np.float32([[0, 0], [0, h2], [w2, h2], [w2, 0]]).reshape(-1, 1, 2)

    # When we have established a homography we need to warp perspective
    # perform perspective tranform using previously calculated matrix and the corners of  
    "query" image#
    # Change field of view
    pts_corners_dst = cv.perspectiveTransform(pts_corners_temp, M)

    ListOfPoints = np.concatenate((pts_corners_src, pts_corners_dst), axis=0)

    [x_min, y_min] = np.int32(ListOfPoints.min(axis=0).ravel() - 0.5)
    [x_max, y_max] = np.int32(ListOfPoints.max(axis=0).ravel() + 0.5)

    translation_dist = [-x_min, -y_min]

    H_translation = np.array([[1, 0, translation_dist[0]], [0, 1, translation_dist[1]], [0, 0, 
1]])

    new_img = cv.warpPerspective(img2, H_translation.dot(M), (x_max - x_min, y_max - y_min))
    new_img[translation_dist[1]:h1 + translation_dist[1], translation_dist[0]:w1 + translation_dist[0]] = img1

return new_img


def find_homography(kpt1, kpt2, matches):
    # Find an Homography matrix between two pictures
    # Transforming keypoints to list of points
    src_pts = np.float32([kpt1[m.queryIdx].pt for m in matches]).reshape(-1, 1, 2)
    dst_pts = np.float32([kpt2[m.trainIdx].pt for m in matches]).reshape(-1, 1, 2)

    # Compute a rigid transformation (without depth, only scale + rotation + translation) /affine transformation
    transformation_rigid_matrix, rigid_mask = cv.estimateAffinePartial2D(src_pts, dst_pts)
    affine_row = [0, 0, 1]
    transformation_rigid_matrix = np.vstack((transformation_rigid_matrix, affine_row))

return transformation_rigid_matrix


# Read images
img1 = cv.imread('1.jpg', 1)
img2 = cv.imread('2.jpg', 1)
img3 = cv.imread('3.jpg', 1)
img1 = cv.cvtColor(img1, cv.COLOR_BGR2GRAY)
img2 = cv.cvtColor(img2, cv.COLOR_BGR2GRAY)
img3 = cv.cvtColor(img3, cv.COLOR_BGR2GRAY)

# Initiate ORB detector, that it will be our detector object.
orb = cv.ORB_create()

# find the keypoints and compute the descriptors with ORB for images
kpts1, des1 = orb.detectAndCompute(img1, None)
kpts2, des2 = orb.detectAndCompute(img2, None)
kpts3, des3 = orb.detectAndCompute(img3, None)

# Create a BFMatcher object.
bf = cv.BFMatcher_create(cv.NORM_HAMMING)

# match descriptor
matches1to2 = bf.knnMatch(des1, des2, k=2)
matches2to3 = bf.knnMatch(des2, des3, k=2)

# draw matches
good1to2 = []
for m, n in matches1to2:
    if m.distance < 0.6 * n.distance:
        good1to2.append(m)

# draw matches
good2to3 = []
for m, n in matches2to3:
    if m.distance < 0.6 * n.distance:
        good2to3.append(m)

# find  affine transformation and panoramic view between 1 to 2
trans_affine_matrix1to2 = find_homography(kpts2, kpts1, good1to2)
img1to2 = warpImages(img1, img2, trans_affine_matrix1to2)

# find homography matrix and transformation 2 to 3
trans_affine_matrix2to3 = find_homography(kpts2, kpts3, good2to3)
img2to3 = warpImages(img3, img2, trans_affine_matrix2to3)

title1 = "panoramic between img1 and img2"
title2 = "panoramic between img2 and img3"
subplot(1, 2, 1)
imshow(img1to2)
axis('off')
title(title1)

subplot(1, 2, 2)
imshow(img2to3)
axis('off')
title(title2)
show()

谢谢大家的帮助。这是我的照片：

Answer 1

与其从图像 1 和 2 以及图像 2 和 3 创建全景图然后将其组合，不如尝试按顺序进行。有点像这样：

取图像 1 和 2，计算它们的匹配特征。
计算图像 2 与图像 1 的单应性。
使用此 Homography 扭曲图像 2 以将其与图像 1 拼接。这将为您提供中间结果 1。
将此中间结果 1 作为第一张图像，并对序列中的下一张图像重复步骤 1-3。

这里是一个不错的博客post，入门差不多：https://kushalvyas.github.io/stitching.html

要比较您的算法性能，您可以查看 opencv-stitcher class 的结果：

如何使用 python、opencv、orb 描述符和来自 opencv 的 ransac 制作一系列图片的全景视图

how make panoramic view of serie of pictures using python, opencv, orb descriptors and ransac from opencv

python

opencv

panoramas

affinetransform

orb