鱼眼畸变校正

概述

此示例应用程序使用同一相机/镜头拍摄的输入图像执行鱼眼镜头校准。然后，它使用 Remap 和校准数据来校正这些图像的鱼眼镜头畸变，并将结果保存到磁盘。用于畸变校正的映射是 VPI_FISHEYE_EQUIDISTANT，它将场景中的直线映射到校正图像中的直线。

镜头校准

镜头校准使用同一相机/镜头拍摄的一组图像，每个图像都显示一个位于不同位置的棋盘格图案，以便从整体上看，棋盘格几乎出现在整个视野中。图像越多，校准就越准确，但通常 10 到 15 张图像就足够了。

VPI 示例包含一组可以使用的输入图像。它们位于 /opt/nvidia/vpi3/samples/assets/fisheye 目录中。

注意

鱼眼 Python 示例仅适用于 OpenCV 版本 <= 4.9

校准图像示例

要为给定的镜头创建一组校准图像，请执行以下操作

1. 在纸上打印棋盘格图案。VPI 在示例的 assets 目录中提供了一个 10x7 棋盘格文件，可以使用，名为 checkerboard_10x7.pdf。
2. 将鱼眼镜头安装在相机上。
3. 在相机位置固定的情况下，拍摄几张照片，显示位于不同位置的棋盘格，覆盖视野的大部分区域。

说明

命令行参数如下

-c W,H [-s win] <image1> [image2] [image3] ...

其中

• -c W,H：指定棋盘格图案水平 (W) 和垂直 (H) 方向上的方格数。
• -s win：（可选）在棋盘格的每个内部顶点（4 个方格相交的点）周围的窗口宽度，用于顶点位置细化阶段。实际的顶点位置将在此窗口内搜索。如果省略此参数，将跳过细化阶段。
• imageN：校准图像集

这是一个调用示例

• C++

  ./vpi_sample_11_fisheye -c 10,7 -s 22 ../assets/fisheye/*.jpg

• Python

  python3 main.py -c 10,7 -s 22 ../assets/fisheye/*.jpg

这将校正包含的校准图像集，所有图像均使用也 包含 的棋盘格图案捕获。它在每个棋盘格内部顶点周围使用 22x22 窗口来细化顶点位置。

结果

以下是示例应用程序生成的一些输入和输出图像

输入	已校正

源代码

为了方便起见，以下是在 samples 目录中也安装的代码。

语言 C++ Python

import sys
import vpi
import numpy as np
from argparse import ArgumentParser
import cv2

# ============================
# Parse command line arguments

# Changes in fisheye camera calibration function in OpenCV 4.10, leads to corrupted output.
# Get the major and minor version numbers
version = cv2.__version__.split('.')
major = int(version[0])
minor = int(version[1])

# Check if the version is 4.10 or higher
if major * 100 + minor >= 410
 raise Exception("OpenCV >= 4.10 isn't supported")

parser = ArgumentParser()
parser.add_argument('-c', metavar='W,H', required=True,
 help='Checkerboard with WxH squares')

parser.add_argument('-s', metavar='win', type=int,
 help='Search window width around checkerboard verted used in refinement, default is 0 (disable refinement)')

parser.add_argument('images', nargs='+',
 help='Input images taken with a fisheye lens camera')

args = parser.parse_args();

# Parse checkerboard size
try
 cbSize = np.array([int(x) for x in args.c.split(',')])
except ValueError
 exit("Error parsing checkerboard information")

# =========================================
# Calculate fisheye calibration from images

# OpenCV expects number of interior vertices in the checkerboard,
# not number of squares. Let's adjust for that.
vtxCount = cbSize-1

# -------------------------------------------------
# Determine checkerboard coordinates in image space

imgSize = None
corners2D = []

for imgName in args.images
 # Load input image and do some sanity check
 img = cv2.imread(imgName)
 curImgSize = (img.shape[1], img.shape[0])

 if imgSize == None
 imgSize = curImgSize
 elif imgSize != curImgSize
 exit("All images must have the same size")

 # Find the checkerboard pattern on the image, saving the 2D
 # coordinates of checkerboard vertices in cbVertices.
 # Vertex is the point where 4 squares (2 white and 2 black) meet.
 found, corners = cv2.findChessboardCorners(img, tuple(vtxCount), flags=cv2.CALIB_CB_ADAPTIVE_THRESH + cv2.CALIB_CB_NORMALIZE_IMAGE)
 if found
 # Needs to perform further corner refinement?
 if args.s != None and args.s >= 2
 criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_COUNT, 30, 0.0001)
 imgGray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
 corners = cv2.cornerSubPix(imgGray, corners, (args.s//2, args.s//2), (-1,-1), criteria)
 corners2D.append(corners)
 else
 exit("Warning: checkerboard pattern not found in image {}".format(input))

# Create the vector that stores 3D coordinates for each checkerboard pattern on a space
# where X and Y are orthogonal and run along the checkerboard sides, and Z==0 in all points on
# checkerboard.
cbCorners = np.zeros((1, vtxCount[0]*vtxCount[1], 3))
cbCorners[0,:,:2] = np.mgrid[0:vtxCount[0], 0:vtxCount[1]].T.reshape(-1,2)
corners3D = [cbCorners.reshape(-1,1,3) for i in range(len(corners2D))]

# ---------------------------------------------
# Calculate fisheye lens calibration parameters
camMatrix = np.eye(3)
coeffs = np.zeros((4,))
rms, camMatrix, coeffs, rvecs, tvecs = cv2.fisheye.calibrate(corners3D, corners2D, imgSize, camMatrix, coeffs, flags=cv2.fisheye.CALIB_FIX_SKEW)

# Print out calibration results
print("rms error: {}".format(rms))
print("Fisheye coefficients: {}".format(coeffs))
print("Camera matrix:")
print(camMatrix)

# ======================
# Undistort input images

# Create an uniform grid
grid = vpi.WarpGrid(imgSize)

# Create undistort warp map from the calibration parameters and the grid
undist_map = vpi.WarpMap.fisheye_correction(grid,
 K=camMatrix[0:2,:], X=np.eye(3,4), coeffs=coeffs,
 mapping=vpi.FisheyeMapping.EQUIDISTANT)

# Go through all input images,
idx=0
for imgName in args.images
 # Load input image and do some sanity check
 img = cv2.imread(imgName)

 # Using the CUDA backend,
 with vpi.Backend.CUDA
 # Convert image to NV12_ER, apply the undistortion map and convert image back to RGB8
 imgCorrected = vpi.asimage(img).convert(vpi.Format.NV12_ER).remap(undist_map, interp=vpi.Interp.CATMULL_ROM).convert(vpi.Format.RGB8)

 # Write undistorted image to disk
 cv2.imwrite("undistort_python{}_{:03d}.jpg".format(sys.version_info[0],idx), imgCorrected.cpu())
 idx += 1

#include <opencv2/core/version.hpp>

#if CV_MAJOR_VERSION >= 3
# include <opencv2/imgcodecs.hpp>
#else
# include <opencv2/highgui/highgui.hpp>
#endif

// Changes in fisheye camera calibration function in OpenCV 4.10, leads to corrupted output.
#if CV_VERSION_MAJOR * 100 + CV_VERSION_MINOR >= 410
# error "OpenCV >= 4.10 isn't supported"
#endif

#include <opencv2/calib3d/calib3d.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <vpi/OpenCVInterop.hpp>

#include <vpi/Image.h>
#include <vpi/LensDistortionModels.h>
#include <vpi/Status.h>
#include <vpi/Stream.h>
#include <vpi/algo/ConvertImageFormat.h>
#include <vpi/algo/Remap.h>

#include <iostream>
#include <sstream>

#define CHECK_STATUS(STMT) \
 do \
 { \
 VPIStatus status = (STMT); \
 if (status != VPI_SUCCESS) \
 { \
 char buffer[VPI_MAX_STATUS_MESSAGE_LENGTH]; \
 vpiGetLastStatusMessage(buffer, sizeof(buffer)); \
 std::ostringstream ss; \
 ss << vpiStatusGetName(status) << ": " << buffer; \
 throw std::runtime_error(ss.str()); \
 } \
 } while (0);

static void PrintUsage(const char *progname, std::ostream &out)
{
 out << "Usage: " << progname << " <-c W,H> [-s win] <image1> [image2] [image3] ...\n"
 << " where,\n"
 << " W,H\tcheckerboard with WxH squares\n"
 << " win\tsearch window width around checkerboard vertex used\n"
 << "\tin refinement, default is 0 (disable refinement)\n"
 << " imageN\tinput images taken with a fisheye lens camera" << std::endl;
}

static char *my_basename(char *path)
{
#ifdef WIN32
 char *name = strrchr(path, '\\');
#else
 char *name = strrchr(path, '/');
#endif
 if (name != NULL)
 {
 return name;
 }
 else
 {
 return path;
 }
}

struct Params
{
 cv::Size vtxCount; // Number of internal vertices the checkerboard has
 int searchWinSize; // search window size around the checkerboard vertex for refinement.
 std::vector<const char *> images; // input image names.
};

static Params ParseParameters(int argc, char *argv[])
{
 Params params = {};

 cv::Size cbSize;

 for (int i = 1; i < argc; ++i)
 {
 if (argv[i][0] == '-')
 {
 if (strlen(argv[i] + 1) == 1)
 {
 switch (argv[i][1])
 {
 case 'h'
 PrintUsage(my_basename(argv[0]), std::cout);
 return {};

 case 'c'
 if (i == argc - 1)
 {
 throw std::invalid_argument("Option -c must be followed by checkerboard width and height");
 }

 if (sscanf(argv[++i], "%d,%d", &cbSize.width, &cbSize.height) != 2)
 {
 throw std::invalid_argument("Error parsing checkerboard information");
 }

 // OpenCV expects number of interior vertices in the checkerboard,
 // not number of squares. Let's adjust for that.
 params.vtxCount.width = cbSize.width - 1;
 params.vtxCount.height = cbSize.height - 1;
 break;

 case 's'
 if (i == argc - 1)
 {
 throw std::invalid_argument("Option -s must be followed by search window size");
 }
 if (sscanf(argv[++i], "%d", &params.searchWinSize) != 1)
 {
 throw std::invalid_argument("Error parsing search window size");
 }
 if (params.searchWinSize < 0)
 {
 throw std::invalid_argument("Search window size must be >= 0");
 }
 break;

 default
 throw std::invalid_argument(std::string("Option -") + (argv[i] + 1) + " not recognized");
 }
 }
 else
 {
 throw std::invalid_argument(std::string("Option -") + (argv[i] + 1) + " not recognized");
 }
 }
 else
 {
 params.images.push_back(argv[i]);
 }
 }

 if (params.images.empty())
 {
 throw std::invalid_argument("At least one image must be defined");
 }

 if (cbSize.width <= 3 || cbSize.height <= 3)
 {
 throw std::invalid_argument("Checkerboard size must have at least 3x3 squares");
 }

 if (params.searchWinSize == 1)
 {
 throw std::invalid_argument("Search window size must be 0 (default) or >= 2");
 }

 return params;
}

int main(int argc, char *argv[])
{
 // OpenCV image that will be wrapped by a VPIImage.
 // Define it here so that it's destroyed *after* wrapper is destroyed
 cv::Mat cvImage;

 # VPI objects that will be used
 VPIStream stream = NULL;
 VPIPayload remap = NULL;
 VPIImage tmpIn = NULL, tmpOut = NULL;
 VPIImage vimg = nullptr;

 int retval = 0;

 try
 {
 // First parse command line paramers
 Params params = ParseParameters(argc, argv);
 if (params.images.empty()) // user just wanted the help message?
 {
 return 0;
 }

 // Where to store checkerboard 2D corners of each input image.
 std::vector<std::vector<cv::Point2f>> corners2D;

 // 存储图像尺寸。所有输入图像必须具有相同尺寸。
 cv::Size imgSize = {};

 for (unsigned i = 0; i < params.images.size(); ++i)
 {
 // 加载输入图像并进行一些健全性检查
 cv::Mat img = cv::imread(params.images[i]);
 if (img.empty())
 {
 throw std::runtime_error("Can't read " + std::string(params.images[i]));
 }

 if (imgSize == cv::Size{})
 {
 imgSize = img.size();
 }
 else if (imgSize != img.size())
 {
 throw std::runtime_error("All images must have same size");
 }

 // 在图像上查找棋盘格图案，将棋盘格顶点的 2D
 // 坐标保存在 cbVertices 中。
 // 顶点是 4 个正方形（2 个白色和 2 个黑色）相遇的点。
 std::vector<cv::Point2f> cbVertices;

 if (findChessboardCorners(img, params.vtxCount, cbVertices,
 cv::CALIB_CB_ADAPTIVE_THRESH + cv::CALIB_CB_NORMALIZE_IMAGE))
 {
 // 需要执行进一步的角点细化吗？
 if (params.searchWinSize >= 2)
 {
 cv::Mat gray;
 cvtColor(img, gray, cv::COLOR_BGR2GRAY);

 cornerSubPix(gray, cbVertices, cv::Size(params.searchWinSize / 2, params.searchWinSize / 2),
 cv::Size(-1, -1),
 cv::TermCriteria(cv::TermCriteria::EPS + cv::TermCriteria::COUNT, 30, 0.0001));
 }

 // 将此图像的 2D 顶点保存在向量中
 corners2D.push_back(std::move(cbVertices));
 }
 else
 {
 std::cerr << "Warning: checkerboard pattern not found in image " << params.images[i] << std::endl;
 }
 }

 // 创建向量，用于存储空间中每个棋盘格图案的 3D 坐标
 // 其中 X 和 Y 是正交的，并沿着棋盘格边运行，并且在棋盘格上的所有点中 Z==0。
 // 棋盘格。
 std::vector<cv::Point3f> initialCheckerboard3DVertices;
 for (int i = 0; i < params.vtxCount.height; ++i)
 {
 for (int j = 0; j < params.vtxCount.width; ++j)
 {
 // 由于我们对外部相机参数不感兴趣，
 // 我们可以假设棋盘格正方形大小为 1x1。
 initialCheckerboard3DVertices.emplace_back(static_cast<float>(j), static_cast<float>(i), 0.0f);
 }
 }

 // 使用所有图像的初始棋盘格位置初始化向量
 std::vector<std::vector<cv::Point3f>> corners3D(corners2D.size(), initialCheckerboard3DVertices);

 // 相机内部参数，最初为单位矩阵（将通过校准过程估计）。
 using Mat3 = cv::Matx<double, 3, 3>;
 Mat3 camMatrix = Mat3::eye();

 // 存储鱼眼模型系数。
 std::vector<double> coeffs(4);

 // VPI 当前不支持相机矩阵上的倾斜参数，请确保
 // 校准过程将其固定为 0。
 int flags = cv::fisheye::CALIB_FIX_SKEW;

 // 运行校准
 {
 cv::Mat rvecs, tvecs; // 存储每个相机的旋转和平移，现在不需要。
 double rms = cv::fisheye::calibrate(corners3D, corners2D, imgSize, camMatrix, coeffs, rvecs, tvecs, flags);
 printf("rms 误差: %lf\n", rms);
 }

 // 输出校准结果。
 printf("鱼眼系数: %lf %lf %lf %lf\n", coeffs[0], coeffs[1], coeffs[2], coeffs[3]);

 printf("相机矩阵:\n");
 printf("[%lf %lf %lf; %lf %lf %lf; %lf %lf %lf]\n", camMatrix(0, 0), camMatrix(0, 1), camMatrix(0, 2),
 camMatrix(1, 0), camMatrix(1, 1), camMatrix(1, 2), camMatrix(2, 0), camMatrix(2, 1), camMatrix(2, 2));

 // 现在使用 VPI 来 undistort 输入图像：

 // 分配密集映射。
 VPIWarpMap map = {};
 map.grid.numHorizRegions = 1;
 map.grid.numVertRegions = 1;
 map.grid.regionWidth[0] = imgSize.width;
 map.grid.regionHeight[0] = imgSize.height;
 map.grid.horizInterval[0] = 1;
 map.grid.vertInterval[0] = 1;
 CHECK_STATUS(vpiWarpMapAllocData(&map));

 // 使用校准程序给出的系数初始化鱼眼镜头模型。
 VPIFisheyeLensDistortionModel distModel = {};
 distModel.mapping = VPI_FISHEYE_EQUIDISTANT;
 distModel.k1 = coeffs[0];
 distModel.k2 = coeffs[1];
 distModel.k3 = coeffs[2];
 distModel.k4 = coeffs[3];

 // 填充相机校准程序给出的相机内部参数。
 VPICameraIntrinsic K;
 for (int i = 0; i < 2; ++i)
 {
 for (int j = 0; j < 3; ++j)
 {
 K[i][j] = camMatrix(i, j);
 }
 }

 // 相机外参应为单位矩阵。
 VPICameraExtrinsic X = {};
 X[0][0] = X[1][1] = X[2][2] = 1;

 // 生成扭曲映射，以校正由鱼眼镜头拍摄的图像的失真
 // 使用上面计算的给定参数。
 vpiWarpMapGenerateFromFisheyeLensDistortionModel(K, X, K, &distModel, &map);

 // 创建 Remap 有效负载以根据上面生成的映射进行 undistortion。
 CHECK_STATUS(vpiCreateRemap(VPI_BACKEND_CUDA, &map, &remap));

 // 现在 remap 有效负载已创建，我们可以销毁扭曲映射。
 vpiWarpMapFreeData(&map);

 // 创建将要执行操作的流。我们正在使用 CUDA
 // 处理。
 CHECK_STATUS(vpiStreamCreate(VPI_BACKEND_CUDA, &stream));

 // NV12 格式的临时输入和输出图像。
 CHECK_STATUS(vpiImageCreate(imgSize.width, imgSize.height, VPI_IMAGE_FORMAT_NV12_ER, 0, &tmpIn));
 CHECK_STATUS(vpiImageCreate(imgSize.width, imgSize.height, VPI_IMAGE_FORMAT_NV12_ER, 0, &tmpOut));

 // 对于每个输入图像，
 for (unsigned i = 0; i < params.images.size(); ++i)
 {
 // 从磁盘读取它。
 cvImage = cv::imread(params.images[i]);
 assert(!cvImage.empty());

 // 将其包装到 VPIImage 中
 if (vimg == nullptr)
 {
 // 现在创建一个包装它的 VPIImage。
 CHECK_STATUS(vpiImageCreateWrapperOpenCVMat(cvImage, 0, &vimg));
 }
 else
 {
 CHECK_STATUS(vpiImageSetWrappedOpenCVMat(vimg, cvImage));
 }

 // 转换 BGR -> NV12
 CHECK_STATUS(vpiSubmitConvertImageFormat(stream, VPI_BACKEND_CUDA, vimg, tmpIn, NULL));

 // Undistorts 输入图像。
 CHECK_STATUS(vpiSubmitRemap(stream, VPI_BACKEND_CUDA, remap, tmpIn, tmpOut, VPI_INTERP_CATMULL_ROM,
 VPI_BORDER_ZERO, 0));

 // 将结果 NV12 转换回 BGR，写回输入图像。
 CHECK_STATUS(vpiSubmitConvertImageFormat(stream, VPI_BACKEND_CUDA, tmpOut, vimg, NULL));

 // 等待转换完成。
 CHECK_STATUS(vpiStreamSync(stream));

 // 由于 vimg 正在包装 OpenCV 图像，因此结果已在那里。
 // 我们只需要将其保存到磁盘。
 char buf[64];
 snprintf(buf, sizeof(buf), "undistort_%03d.jpg", i);
 imwrite(buf, cvImage);
 }
 }
 catch (std::exception &e)
 {
 std::cerr << "Error: " << e.what() << std::endl;
 PrintUsage(my_basename(argv[0]), std::cerr);

 retval = 1;
 }

 vpiStreamDestroy(stream);
 vpiPayloadDestroy(remap);
 vpiImageDestroy(tmpIn);
 vpiImageDestroy(tmpOut);
 vpiImageDestroy(vimg);

 return retval;
}