Camera Calibration in Python with OpenCV

Camera calibration is the process of estimating intrinsic and/or extrinsic parameters. Intrinsic parameters deal with the camera’s internal characteristics, such as its focal length, skew, distortion, and image center. Extrinsic parameters describe its position and orientation in the world.

 These Intrinsic and Extrinsic parameters describe the mapping between 3-D reference coordinates and 2-D image ones. Once we get to know these parameters we can remove the distortion from the images. Correction for image distortion in cameras is an important topic in order to obtain accurate information from the vision systems.

Intrinsic parameters

These intrinsic parameters define the properties of the camera produced by it in the real world.

Focal length :-

The focal length of an optical system is a measure of how strongly the system converges or diverges light. It is the inverse of the system’s optical power. A positive focal length indicates that a system converges light, while a negative focal length indicates that the system diverges light.

The formula for focal length is given by:-

1/f = 1/u +1/v

•  f – focal length of the lens
• u – object distance from the pole of the lens
• v – the distance between the image formed by the lens and the pole

Distortion:-

Lens distortion is any deformation that occurs in the images produced by a camera lens. Distortion can generally be described as when straight lines appear bent or curvy in photographs. Sometimes this effect is intended, other times it occurs as a result of an error.

Camera Calibration is performed to remove the distortion formed by the lens of the camera. There are two types of distortions known as Barrel Distortion, Pincushion distortion.

In Barel Distortion the curves along the edges appear to be diverging, while in Pincushion distortion the edges appear to be converging and these distortions appear to be as if seen by a fisheye.

Zhang’s method for Camera Calibration

Zhang model is a camera calibration method that uses traditional calibration techniques and self-calibration techniques( correspondence between the calibration points when they are in different positions ). To perform a full calibration by the zhang method at least three different images of the calibration target/gauge are required, either by moving the gauge or the camera itself. 

If some of the intrinsic parameters are given as data ( orthogonality of the image or optical center coordinates ) the number of images can be reduced to two.

In the first step, an approximation of the estimated projection matrix H between the calibration target and the image plane is determined using the DLT method. Subsequently, applying self-calibration techniques to obtain the image of the absolute conic matrix. 

The main contribution of the Zhang method is how to extract a constrained projective mapping M which maps the world coordinates with pixel coordinates.

Here M is a combination of Intrinsic parameter K and the extrinsic parameters R and T.

R and T are the extrinsic parameters that denote the coordinate system transformations from 3D world coordinates to 3D camera coordinates.

The Following are the steps followed to remove the distortion from the images:-

• Passing Multiple images with Known objects, in our example, we use a chessboard
• The known objects have to be at different positions in the images that we pass
• Different positions of the image such as rotated, translated, tilted.
• Computing the Camera matrix, distortion, rotational and translation vectors.
• Removing distortion and rebuilding the image

As the process begins with passing multiple images, instead of passing the paths of all the images manually, we can automate the process by using glob.glob()

glob.glob() 

The glob module generates lists of files matching given patterns, just like the Unix shell. The parameters it takes are the file extension that is required to be loaded into the program. The syntax is provided below:-

images = glob.glob('C:\images\calib\*.png')

In the above line of code, it searches for the images folder, once it enters the images folder it opens files having images since we have directed the function to do so by using *.png.

Once the list of images has been loaded there are certain parameters that have to be pre-defined while handling the corners of the object in the image. They are:-

Termination Criteria

Termination Criteria defines when to stop the iterative process of corner refinement. It is used while detecting the corners in the image and by default, the value is the same for every operation.

It terminates the iteration whenever the corner refinement exceeds TERM_CRITERIA_MAX_ITER or when it falls less than TERM_CRITERIA_EPS

criteria = (cv2.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001)

Now we need to find the corners of the object ( in our example chessboard ), using cv2.findChessboardCorners(). It accepts a gray image and the size of the object in the image. We can loop over the list of images generated by glob.glob().

for image in images:

    img = cv2.imread(image)
    gray = cv2.cvtColor(img, cv.COLOR_BGR2GRAY)

    # Find the chess board corners
    ret, corners = cv2.findChessboardCorners(gray, chessboardSize, None)

Once the corners are detected we need to refine those corners in the same loop and append the results of the particular image into image_points. And this has to be carried out only if the corners are detected, for this purpose we can set the ret condition to true.

if ret == True:
     objpoints.append(objp)
     corners2 = cv2.cornerSubPix(gray, corners, (11,11), (-1,-1), criteria)
     imgpoints.append(corners)

Now we can draw and display the corners in the chessboard

cv2.drawChessboardCorners(img, chessboardSize, corners2, ret)
cv2.imshow('img', img)

As shown in the above figure if we draw chessboard corners on a distorted image it looks this way. Since now we have the object_points and image_points taken from different images, but clicked from the same camera we can use these values to calibrate the camera’s intrinsic parameters.

cv2.calibrateCamera() accepts object_points and image_points as parameters and returns the camera matrix, distortion, rotational vectors, and translational vectors. Using these values we can calibrate the distortion and the error produced in the image by the camera.

ret, cameraMatrix, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, frameSize, None, None)

Now we have the internal parameters of the image, let’s try to remove the distortion from an image using these values. The figure shown below is a picture captured from the same camera, since we knew the distortion produced by it, let’s try removing distortion and produce a clean image.

img = cv2.imread('C:\Users\images\chess-demo.png')
cv2.imshow('img', img)

To remove distortion we need a newcamera intrinsic matrix. To calculate it we have an in-built function in OpenCV known as cv2.getOptionalCameraMatrix().

The function computes and returns the optimal new camera intrinsic matrix based on the free scaling parameter. By varying this parameter, you may retrieve only sensible pixels alpha=0, keep all the original image pixels if there is valuable information in the corners alpha=1, or get something in between.

cv2.getOptionalNewCameraMatrix(cameraMatrix, distCoeff, imageSize, alpha, newImageSize)

Parameters

• cameraMatrix – Input camera intrinsic matrix.
• distCoeff – Input vector of distortion coefficients. If the vector is NULL/empty, the zero distortion coefficients are assumed
• imageSize – Dimensions of the source image.
• alpha – Free scaling parameter between 0 (when all the pixels in the undistorted image are valid) and 1 (when all the source image pixels are retained in the undistorted image)
• newImageSize – Dimensions of the destination image

h,  w = img.shape[:2]
newCameraMatrix, roi = cv2.getOptimalNewCameraMatrix(cameraMatrix, dist, (w,h), 1, (w,h))

Now using the newCameraMatrix we can apply undistortion using cv2.undistort(). It corrects lens distortion for the given camera matrix and distortion coefficients

cv2.undistort(src, cameraMatrix, dist, destination, newCameraMatrix)

Parameters

• src – input distorted image
• destination – output corrected image that has the same type as of source.
• cameraMatrix – Input Camera Intrinsic matrix
• newCameraMatrix – Camera matrix of the distorted image. By default, it is the same as cameraMatrix but you may additionally scale and shift the result by using a different matrix
• distCoeff – Input vector of distortion coefficients

dst = cv.undistort(img, cameraMatrix, dist, None, newCameraMatrix)

Once we get a distorted image, it needs to be cropped using the ROI that we got in the previous step. the image is present only in the ROI region, and stores the coordinates of bounding rect in the image.

x, y, w, h = roi
cv.imwrite('caliResult0.png', dst)
dst = dst[y:y+h, x:x+w]
cv.imshow('img', img)
cv.imshow('Result1.png', dst)

As we can clearly notice the distortion in the image is completely removed in the right side image in the above figure.

Reprojection Error

The reprojection error is a geometric error corresponding to the image distance between a projected point and a measured one. It is used to quantify how closely an estimate of a 3D point recreates the point’s true projection.

The geometric error is calculated by normalizing the image points of the source and the projected 3D point and calculating the result by diving it with the number of points in the source image. So first we need to calculate the projected points in 3D of the source image.

cv2.projectPoints() function computes the 2D projections of 3D points to the image plane, given intrinsic and extrinsic camera parameters. Optionally, the function computes Jacobians -matrices of partial derivatives of image points coordinates (as functions of all the input parameters) with respect to the particular parameters, intrinsic and/or extrinsic.

cv2.projectPoints(objpoints, rvecs, tvecs, cameraMatrix, dist)

Parameters

• objpoints – Array of object points expressed wrt. the world coordinate frame
• rvecs – The rotation vector that, together with tvec, performs a change of basis from world to camera coordinate system
• tvecs – The translation vector
• cameraMatrix – Input Camera Intrinsic matrix
• distCoeff – Input vector of distortion coefficients

# Reprojection Error
mean_error = 0

for i in range(len(objpoints)):
    imgpoints2, _ = cv.projectPoints(objpoints[i], rvecs[i], tvecs[i], cameraMatrix, dist)
    error = cv.norm(imgpoints[i], imgpoints2, cv.NORM_L2)/len(imgpoints2)
    mean_error += error

print( "total error: {}".format(mean_error/len(objpoints)) )

>>> total error: 0.04000277131338806

The whole source code is given below

import numpy as np
import cv2 as cv
import glob

chessboardSize = (24,17)
frameSize = (1440,1080)

# termination criteria
criteria = (cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001)

# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
objp = np.zeros((chessboardSize[0] * chessboardSize[1], 3), np.float32)
objp[:,:2] = np.mgrid[0:chessboardSize[0],0:chessboardSize[1]].T.reshape(-1,2)

# Arrays to store object points and image points from all the images.
objpoints = [] # 3d point in real world space
imgpoints = [] # 2d points in image plane.

images = glob.glob('C:\images\calib\*.png')
for image in images:
    
    img = cv.imread(image)
    gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)

    # Find the chess board corners
    ret, corners = cv.findChessboardCorners(gray, chessboardSize, None)

    # If found, add object points, image points (after refining them)
    if ret == True:
        objpoints.append(objp)
        corners2 = cv.cornerSubPix(gray, corners, (11,11), (-1,-1), criteria)
        imgpoints.append(corners)
 
        # Draw and display the corners
        cv.drawChessboardCorners(img, chessboardSize, corners2, ret)
        
ret, cameraMatrix, dist, rvecs, tvecs = cv.calibrateCamera(objpoints, imgpoints, frameSize, None, None)
img = cv.imread('C:\images\chess-demo.png')
h,  w = img.shape[:2]
newCameraMatrix, roi = cv.getOptimalNewCameraMatrix(cameraMatrix, dist, (w,h), 1, (w,h))

# Undistort
dst = cv.undistort(img, cameraMatrix, dist, None, newCameraMatrix)

# crop the image
x, y, w, h = roi
cv.imwrite('caliResult0.png', dst)
dst = dst[y:y+h, x:x+w]
cv.imwrite('calibResult1.png', dst)

# Reprojection Error
mean_error = 0

for i in range(len(objpoints)):
    imgpoints2, _ = cv.projectPoints(objpoints[i], rvecs[i], tvecs[i], cameraMatrix, dist)
    error = cv.norm(imgpoints[i], imgpoints2, cv.NORM_L2)/len(imgpoints2)
    mean_error += error
print( "total error: {}".format(mean_error/len(objpoints)) )