CV / Pinehole Camera [Python]

This practical work focuses on Pinehole Camera Calibration using OpenCV and Python.

1. Preparation

The first dependency that is needed is OpenCV library that  can be installed with the command:

pip install opencv-python

or for conda installation:

conda install -c conda-forge opencv-python

The second dependency is the MatPlotLib library that can be installed with the command:

pip install matplotlib

or for conda installation:

conda install -c conda-forge matplotlib

The MatPlotLib library displays geometric rendering in an independent interactive window. Depending on the Python environment, this window may be rendered as a frozen image,
preventing user interaction. To correct this problem, here are a few solutions:

Jupyter Notebook:
Execute following code within the Jupyter Notebook: %matplotlib qt

PyCharm:
Go to Settings / Tool / Python Plot and uncheck the option Show plots in tool windows.

Spyder:
Go to Tools / Preferences / IPython console / Graphics / Backend:Inline and change "Inline" to "Automatic".

Click OK button and restart the IDE

2. Using OpenCV and MatPlot within Python program

OpenCV python binding relies on Numpy for the vector and matrix representation and on MatPlotLib for display. Using OpenCV within a Python program needs to import the three libraries:

import numpy as np
import cv2 as cv
import matplotlib.pyplot as plt

All Python program that uses OpenCV has to contain these imports.

Exercise 1.

Create a python file calibration_chessboard.py that contains the following program:

import numpy as np
import cv2 as cv
import glob
import matplotlib.pyplot as plt

def calibrate_chessboard(image_files):
	print('Chessboard calibration')


def main():
	calibrate_chessboard(())


if __name__ == "__main__":
	main()

Run the program to ensure that all the dependencies are installed.

 

3. Camera calibration

The pinhole camera model enables to describe the projection of a 3D point $P_{c}\ =\ (x_{c},\ y_{c},\ z_{c})$ onto an image $i$ as a 2D point $(u_{c}^{i},\ v_{c}^{i})$. More formally:

$$\left[{}\begin{array}{c}u_{c}^{i} \\ v_{c}^{i} \\ 1 \end{array}\right]{}\ =\ \left[{}\begin{array}{c} f_{u} & 0 & c_{u} \\ 0 & f_{v} & c_{v} \\ 0 & 0 & 1 \end{array}\right]\ \Delta{}\left({}\left[{}\begin{array}{c} \dfrac{r_{11}^{i}x_{c}\ +\ r_{12}^{i}y_{c}\ +\ r_{13}^{i}z_{c}\ +\ t_{x}^{i}}{r_{31}^{i}x_{c}\ +\ r_{32}^{i}y_{c}\ +\ r_{33}z_{c}\ +\ t_{z}^{i}} \\ \\ \dfrac{r_{21}^{i}x_{c}\ +\ r_{22}^{i}y_{c}\ +\ r_{23}^{i}z_{c}\ +\ t_{x}^{i}}{r_{31}^{i}x_{c}\ +\ r_{32}^{i}y_{c}\ +\ r_{33}z_{c}\ +\ t_{z}^{i}} \\ \\ 1 \end{array}\right]{}\right)$$

where $\Delta{}$ is the distortion computation:

$$\Delta{}\left({} \left[ \begin{array}{c} x_{p} \\ y_{p} \\ \alpha{} \end{array}\right]{}\right){}\ =\ \left[ \begin{array}{c} x_{p} \dfrac{1 + k_1 r^2 + k_2 r^4 + k_3 r^6}{1 + k_4 r^2 + k_5 r^4 + k_6 r^6} + 2 p_1 x_{p} y_{p} + p_2(r^2 + 2 x_{p}^2) + s_1 r^2 + s_2 r^4  \\ \\ y_{p} \dfrac{1 + k_1 r^2 + k_2 r^4 + k_3 r^6}{1 + k_4 r^2 + k_5 r^4 + k_6 r^6} + p_1 (r^2 + 2 y'^2) + 2 p_2 x_{p} y_{p} + s_3 r^2 + s_4 r^4 \\ \\ \alpha{} \end{array}\right],\ r\ =\ x_{p}^{2}+y_{p}^{2}$$

with:

  • $f_{u}$, $f_{v}$ respectively are the horizontal and vertical focal lengthes, expressed in pixels
  • $c_{u}$, $c_{v}$ are the coordinates of the principal point, expressed in pixels, within the image referential
  • $k_{1}$, $k_{2}$, $k_{3}$ are the radial distortion coefficients
  • $k_{4}$, $k_{5}$, $k_{6}$ are the radial distortion rational coefficients
  • $p_{1}$, $p_{2}$ are the tangential distortion coefficients
  • $s_{1}$, $s_{2}$, $s_{3}$, $s_{4}$ are the thin prism distortion coefficients
  • $(x_{c},\ y_{c},\ z_{c})$ are the coordinates of the reference point $c$ (i-e the 3D coordinates of the corner $c$)
  • $(u_{c}^{i},\ v_{c}^{i})$ are the coordinates of the projection of the reference point $c$ onto the image $i$
  • The image $i$ pose is defined such as: $$\left[{} \begin{array}{c} r_{11}^{i} & r_{12}^{i} & r_{13}^{i} & t_{x}^{i} \\ r_{21}^{i} & r_{22}^{i} & r_{23}^{i} & t_{y}^{i} \\ r_{31}^{i} & r_{32}^{i} & r_{33}^{i} & t_{z}^{i} \\ 0 & 0 & 0 & 1 \end{array} \right]$$

This computation shows that some of the parameters are the same for all the projection equations that are coming from a same camera:

  • $f_{u}$, $f_{v}$, $c_{u}$, $c_{v}$
  • The distortion parameters $k$, $p$ and $s$

Determining these parameters enable to simplify the projection computation. The process that compute their values is called camera calibration.

3.1. Preparation

The first steps before calibrating a camera are to ensure that all the needed dependencies are set up and that it is possible to load images from files.

Exercise 2.

Create a folder named data in the same directory as calibration_chessboard.py and download the images dataset from http://www.seinturier.fr/teaching/computer_vision/data/calibration-google_pixel_6.zip.

Unzip the calibration-google_pixel_6.zip file within the data directory. Now the directory structure should be data/google_pixel_6/chessboard/6x11 and should contains images. 

Modify the calibration_chessboard.py program as follows:

import numpy as np
import cv2 as cv
import glob
import matplotlib.pyplot as plt


def calibrate_chessboard(image_files):
	print('Image list:')
	for image_file in image_files:
		print('  - '+image_file)


def main():

	# Load images from data folder
	image_files = glob.glob('data/google_pixel_6/chessboard/6x11/*.jpg')

	calibrate_chessboard(image_files)


if __name__ == "__main__":
	main()

Run the program and ensure that are the images within data/google_pixel_6/chessboard/6x11 directory are listed. Expected result:

Image list:
  - data/google_pixel_6/chessboard/6x11/IMG00001.jpg
  - data/google_pixel_6/chessboard/6x11/IMG00002.jpg
  - data/google_pixel_6/chessboard/6x11/IMG00003.jpg
  - data/google_pixel_6/chessboard/6x11/IMG00004.jpg
  - data/google_pixel_6/chessboard/6x11/IMG00005.jpg
  - data/google_pixel_6/chessboard/6x11/IMG00006.jpg
  - data/google_pixel_6/chessboard/6x11/IMG00007.jpg
  - data/google_pixel_6/chessboard/6x11/IMG00008.jpg
  - data/google_pixel_6/chessboard/6x11/IMG00009.jpg
  - data/google_pixel_6/chessboard/6x11/IMG00010.jpg

Tip: The last file separator / may be a \ on window systems.

 

The first step of the calibration process is the detection of references points (the corners of the chessboard). The detection algorithm works with gray images so dealing with calibration process needs to first load images with their color and then, convert them into grayscale. The loading and convertion of images with OpenCV is performed with the instructions:

img = cv.imread(image_file)

gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)

The first line load an image within OpenCV format, the second line convert the images to grayscale.

After a successful load, images can be displayed within MatPlotLib (the library can handle OpenCV images) using the instruction:

plt.figure(image_file)
plt.imshow(img)
plt.show()

The first line create a windows named according to the image_file we want to display. The second and thirs lines are displaying the image within the created windows. Please note that if various images have to be displayed, the last line plt.show() has to be called only once at the end:

plt.figure(name1)
plt.imshow(img1)
...
plt.figure(nameN)
plt.imshow(imgN)

plt.show()

Exercise 3.

Modify the calibration_chessboard.py program as follows:

import numpy as np
import cv2 as cv
import glob
import matplotlib.pyplot as plt


def calibrate_chessboard(image_files):

	for image_file in image_files:

		# Load image
		img = cv.imread(image_file)

		# Convert image to gray
		gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)

		# Display image
		plt.figure(image_file) # Create a new window
		plt.imshow(img)   # Display the image within the new window
        
	plt.show()


def main():

	# Load images from data folder
	image_files = glob.glob('data/google_pixel_6/chessboard/6x11/*.jpg')

	calibrate_chessboard(image_files)


if __name__ == "__main__":
	main()

Ensure that when running the program, images are showing within MatPlotLib window.

 

3.2. Chessboard configuration

Camera calibration process aims at optimizing a set of reference points with known 3D coordinates with their 2D projections on the image. It is common to use chessboard as reference points set. A chessboard is made of corners that can be organized in rows and columns:

Each corner of the chessboard is a reference point with a known 3D coordinate. The reference points are listed from the chessboard top left to the chessboard bottom right (ie row by row from the top and column by column from the left). The coordinates of the reference point i are:

$$\left( \begin{array}{c}x_{i} \\ y_{i}\\ z_{i}\end{array} \right)\ =\ \left( \begin{array}{c}size\times{}((i-1)\ \text{div}\ cols) \\ size\times{}((i-1)\ \text{mod}\ cols) \\ 0 \end{array} \right)$$

where:

  • $rows$ and $cols$ are the number of rows and the number of columns of the chessboard respectively
  • $i$ is the index of the reference point, $1\leq{}\ i\ \leq{}rows\times{}cols$
  • $size$ is the length of a square

The previous calculus can also be written:

$$\left( \begin{array}{c}x_{r\times{}cols+c} \\ y_{r\times{}cols+c}\\ z_{r\times{}cols+c}\end{array} \right)\ =\ \left( \begin{array}{c}r\times{}size \\ c\times{}size) \\ 0 \end{array} \right),\ \text{where}\ (r, c)\text{ are the row and the column of the corner}$$

Exercise 4.

Modify the program calibration_chessboard.py in order to integrate chessboard characteristics. The function calibrate_chessboard(image_files) has now to start as follows:

def calibrate_chessboard(image_files):

	# Chessboard dimension
	rows = 6
	cols = 11
	size = 1.0

	# Reference points set up
	reference_points = np.zeros((rows * cols, 3), np.float32)

	for x in range(0, cols):
		for y in range(0, rows):
			reference_points[y * cols + x] = [x * size, y * size, 0]

	print('Reference points: ', reference_points)

	# Add here the code from previous exercise
	#  for image_file in image_files:
	# ...

Run the program to check the reference points. Expected output:

Reference points:  [[ 0.  0.  0.]
 [ 1.  0.  0.]
 [ 2.  0.  0.]
 [ 3.  0.  0.]
 [ 4.  0.  0.]
 [ 5.  0.  0.]
 [ 6.  0.  0.]
 [ 7.  0.  0.]
 [ 8.  0.  0.]
 [ 9.  0.  0.]
 [10.  0.  0.]
 [ 0.  1.  0.]
 [ 1.  1.  0.]
 [ 2.  1.  0.]
 [ 3.  1.  0.]
 [ 4.  1.  0.]
 [ 5.  1.  0.]
 [ 6.  1.  0.]
 [ 7.  1.  0.]
 [ 8.  1.  0.]
 [ 9.  1.  0.]
 [10.  1.  0.]
 [ 0.  2.  0.]
 [ 1.  2.  0.]
 [ 2.  2.  0.]
 [ 3.  2.  0.]
 [ 4.  2.  0.]
 [ 5.  2.  0.]
 [ 6.  2.  0.]
 [ 7.  2.  0.]
 [ 8.  2.  0.]
 [ 9.  2.  0.]
 [10.  2.  0.]
 [ 0.  3.  0.]
 [ 1.  3.  0.]
 [ 2.  3.  0.]
 [ 3.  3.  0.]
 [ 4.  3.  0.]
 [ 5.  3.  0.]
 [ 6.  3.  0.]
 [ 7.  3.  0.]
 [ 8.  3.  0.]
 [ 9.  3.  0.]
 [10.  3.  0.]
 [ 0.  4.  0.]
 [ 1.  4.  0.]
 [ 2.  4.  0.]
 [ 3.  4.  0.]
 [ 4.  4.  0.]
 [ 5.  4.  0.]
 [ 6.  4.  0.]
 [ 7.  4.  0.]
 [ 8.  4.  0.]
 [ 9.  4.  0.]
 [10.  4.  0.]
 [ 0.  5.  0.]
 [ 1.  5.  0.]
 [ 2.  5.  0.]
 [ 3.  5.  0.]
 [ 4.  5.  0.]
 [ 5.  5.  0.]
 [ 6.  5.  0.]
 [ 7.  5.  0.]
 [ 8.  5.  0.]
 [ 9.  5.  0.]
 [10.  5.  0.]]

 

3.3. Chessboard detection

Performing calibration relies on finding the reference points (chessboard corners) projections on all images. OpenCV provide a function that enable to detect corners of a chessboard:

ret, corners = cv.findChessboardCorners(image, patternSize, corners, flags)

The function attempts to determine whether the input image is a view of a chessboard pattern and locate the internal chessboard corners. The function returns a non-zero value if all of the corners are found and they are placed in a certain order (row by row, left to right in every row). Otherwise, if the function fails to find all the corners or reorder them, it returns 0. For example, a regular chessboard has $6\times{}10$ squares (black and white) and $5\times{}9$ internal corners, that is, points where the black squares touch each other. The function input parameters are:

  • image: the image in which the corners have to be detected. Most of the time obtained by imread()
  • patternSize: the number of corners that compose the chessboard expressed as (columns, rows)
  • corners: The detected corners as a list of 2D coordinates
  • flags: Specific computation flags for the underlying detection algorithm (see OpenCV documentation for further details)

For calibration processing, corners detection has to be processed on every image.

Exercise 5.

Modify the program calibration_chessboard.py in order to integrate chessboard detection on images. The function calibrate_chessboard(image_files) has to be modified as follows:

import numpy as np
import cv2 as cv
import glob
import matplotlib.pyplot as plt


def calibrate_chessboard(image_files):

	# Chessboard dimension
	rows = 6
	cols = 11
	size = 1.0

	# Reference points set up
	reference_points = np.zeros((rows * cols, 3), np.float32)

	for x in range(0, cols):
		for y in range(0, rows):
			reference_points[y * cols + x] = [x * size, y * size, 0]

	print('Reference points: ', reference_points)

	for image_file in image_files:

		print('  - Processing '+image_file)

		# Load image
		img = cv.imread(image_file)
        
		# Convert image to gray
		gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)

		# Find the chess board corners
		ret, corners = cv.findChessboardCorners(gray, (cols, rows), None)
	
		# If found, add object points, image points
		if ret is True:
			# Display image with corners
			plt.figure(image_file) # Create a new window
			plt.imshow(img)

			for corner in corners:
				plt.plot(corner[0][0], corner[0][1], marker='+', color='red')

		else:
			print('No chessboard corner found on image '+image_file)
        
	plt.show()


def main():

	# Load images from data folder
	image_files = glob.glob('data/google_pixel_6/chessboard/6x11/*.jpg')

	calibrate_chessboard(image_files)


if __name__ == "__main__":
	main()

Ensure that the program is running fine by checking that each image is displayed with detected corners on it (each corner is represented as a red cross):

 

3.4. Detection refinement

Chessboard detection can be optimized by enabling sub-pixel precision. The refinement of detected corners is performed using the OpenCV function:

cv.cornerSubPix(image, corners, winSize, zeroZone, criteria)

This function perform a sub-pixel accurate corner location that is based on the observation that every vector from the center to a point located within a neighborhood of is orthogonal to the image gradient at subject to image and measurement noise. Consider the expression:

$$\epsilon _i = {DI_{p_i}}^T  \cdot (q - p_i)$$

where ${DI_{p_i}}$ is an image gradient at one of the points $p_i$ in a neighborhood of $q$. The value of $q$ is to be found so that $\epsilon_i$ is minimized. A system of equations may be set up with $\epsilon_i$ set to zero:

$$\sum _i(DI_{p_i}  \cdot {DI_{p_i}}^T) \cdot q -  \sum _i(DI_{p_i}  \cdot {DI_{p_i}}^T  \cdot p_i)$$

where the gradients are summed within a neighborhood ("search window") of $q$. Calling the first gradient term $G$ and the second gradient term $b$ gives:

$$q = G^{-1}  \cdot b$$

The algorithm sets the center of the neighborhood window at this new center $q$ and then iterates until the center stays within a set threshold.

The function parameters are:

  • image: the input image (grayscaled)
  • corners: the detected corners (from cv.findChessboardCorners())
  • winSize: Half of the side length of the search window (in pixel). For example, if winSize=(5,5), a $(5*2+1) \times (5*2+1) = 11 \times 11$ search window is used.
  • zeroZone: Half of the size of the dead region in the middle of the search zone over which the computation is ignored. The value of (-1,-1) indicates that there is no such a size. 
  • criteria: Criteria for termination of the iterative process of corner refinement. That is, the process of corner position refinement stops either after criteria.maxCount iterations or when the corner position moves by less than criteria.epsilon on some iteration. 

Exercise 6.

Modify the program calibration_chessboard.py in order to integrate chessboard detection refinement. Within the function calibrate_chessboard(image_files), the existing code:

# If found, add object points, image points
if ret is True:
	# Display image with corners
	plt.figure(image_file) # Create a new window
	plt.imshow(img)

	for corner in corners:
		plt.plot(corner[0][0], corner[0][1], marker='+', color='red')

else:
	print('No chessboard corner found on image '+image_file)

has to be replaced with:

# If found, add object points, image points (after refining them)
if ret is True:
            
	# termination criteria
	criteria = (cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001)
    
	corners_subpix = cv.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria)
            
	# Display image with corners
	plt.figure(image_file) # Create a new window
	plt.imshow(img)

	for corner in corners_subpix:
		plt.plot(corner[0][0], corner[0][1], marker='+', color='red')

else:
	print('No chessboard corner found on image '+image_file)

Ensure that the program still detect correctly the chessboard.

 

3.5. Calibration computation

Camera calibration work with determining the parameters to pass from corners position on an image to a corner position in the 3D worlds. For this purpose, it is needed to express the correspondences between 2D and 3D points. For that, two arrays have to be created, containing for a same index I, the 3D coordinates in one array and the 2D coordinates of the corner in the second array.

Exercise 7.

Modify the program calibration_chessboard.py in order to integrate chessboard detection refinement. Within The function calibrate_chessboard(image_files), two arrays have to be declared at the beginning :

def calibrate_chessboard(image_files):

	# Chessboard dimension
	rows = 6
	cols = 11
	size = 1.0

	# Arrays to store object points and image points from all the images.
	points_2d = []  # 2d points in image plane
	points_3d = []  # 3d point in real world space

...

These arrays have to be updated after the corner detection of an image. For that, the code:

corners_subpix = cv.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria)
            
# Display image with corners
plt.figure(image_file) # Create a new window
plt.imshow(img)

for corner in corners_subpix:
	plt.plot(corner[0][0], corner[0][1], marker='+', color='red')

has to be replaced by:

corners_subpix = cv.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria)
            
# Update 2D / 3D corner coordinates
points_3d.append(reference_points)
points_2d.append(corners_subpix)
            
# Display image with corners
plt.figure(image_file) # Create a new window
plt.imshow(img)

for corner in corners_subpix:
	plt.plot(corner[0][0], corner[0][1], marker='+', color='red')

Ensure that the modifications have not broke your code.

 

The last information needed to perform the calibration is the size of the input images. This size can be obtained using OpenCV integrated functions.

Exercise 8.

Modify the program calibration_chessboard.py in order to perform image size retrieval. Within The function calibrate_chessboard(image_files), two variables have to be declared at the beginning of the function:

def calibrate_chessboard(image_files):

	# Chessboard dimension
	rows = 6
	cols = 11
	size = 1.0

	# Arrays to store object points and image points from all the images.
	points_2d = []  # 2d points in image plane
	points_3d = []  # 3d point in real world space

	# Image size
	image_width = -1
	image_height = -1

...

For each loaded image, its size can be obtained by replacing the code:

for image_file in image_files:

	# Load image
	img = cv.imread(image_file)

by:

for image_file in image_files:

	print('  - Processing '+image_file)

	# Load image
	img = cv.imread(image_file)

	# Get the image dimension
	(h, w) = img.shape[:2]

	if image_width == -1 and image_height == -1:
		image_width = w
		image_height = h
	elif w != image_width or h != image_height:
		print('All calibration images have to got same dimension, ignoring' + image_file)
		continue

Ensure that the program is running without error.

 

The pinhole camera model enables to describe the projection of a 3D point $P_{c}\ =\ (x_{c},\ y_{c},\ z_{c})$ onto an image $i$ as a 2D point $(u_{c}^{i},\ v_{c}^{i})$. More formally:

$$\left[{}\begin{array}{c}u_{c}^{i} \\ v_{c}^{i} \\ 1 \end{array}\right]{}\ =\ \left[{}\begin{array}{c} f_{u} & 0 & c_{u} \\ 0 & f_{v} & c_{v} \\ 0 & 0 & 1 \end{array}\right]\ \Delta{}\left({}\left[{}\begin{array}{c} \dfrac{r_{11}^{i}x_{c}\ +\ r_{12}^{i}y_{c}\ +\ r_{13}^{i}z_{c}\ +\ t_{x}^{i}}{r_{31}^{i}x_{c}\ +\ r_{32}^{i}y_{c}\ +\ r_{33}z_{c}\ +\ t_{z}^{i}} \\ \\ \dfrac{r_{21}^{i}x_{c}\ +\ r_{22}^{i}y_{c}\ +\ r_{23}^{i}z_{c}\ +\ t_{x}^{i}}{r_{31}^{i}x_{c}\ +\ r_{32}^{i}y_{c}\ +\ r_{33}z_{c}\ +\ t_{z}^{i}} \\ \\ 1 \end{array}\right]{}\right)$$

where $\Delta{}$ is the distortion computation:

$$\Delta{}\left({} \left[ \begin{array}{c} x_{p} \\ y_{p} \\ \alpha{} \end{array}\right]{}\right){}\ =\ \left[ \begin{array}{c} x_{p} \dfrac{1 + k_1 r^2 + k_2 r^4 + k_3 r^6}{1 + k_4 r^2 + k_5 r^4 + k_6 r^6} + 2 p_1 x_{p} y_{p} + p_2(r^2 + 2 x_{p}^2) + s_1 r^2 + s_2 r^4  \\ \\ y_{p} \dfrac{1 + k_1 r^2 + k_2 r^4 + k_3 r^6}{1 + k_4 r^2 + k_5 r^4 + k_6 r^6} + p_1 (r^2 + 2 y'^2) + 2 p_2 x_{p} y_{p} + s_3 r^2 + s_4 r^4 \\ \\ \alpha{} \end{array}\right],\ r\ =\ x_{p}^{2}+y_{p}^{2}$$

with:

  • $f_{u}$, $f_{v}$ respectively are the horizontal and vertical focal lengthes, expressed in pixels
  • $c_{u}$, $c_{v}$ are the coordinates of the principal point, expressed in pixels, within the image referential
  • $k_{1}$, $k_{2}$, $k_{3}$ are the radial distortion coefficients
  • $k_{4}$, $k_{5}$, $k_{6}$ are the radial distortion rational coefficients
  • $p_{1}$, $p_{2}$ are the tangential distortion coefficients
  • $s_{1}$, $s_{2}$, $s_{3}$, $s_{4}$ are the thin prism distortion coefficients
  • $(x_{c},\ y_{c},\ z_{c})$ are the coordinates of the reference point $c$ (i-e the 3D coordinates of the corner $c$)
  • $(u_{c}^{i},\ v_{c}^{i})$ are the coordinates of the projection of the reference point $c$ onto the image $i$
  • The image $i$ pose is defined such as: $$\left[{} \begin{array}{c} r_{11}^{i} & r_{12}^{i} & r_{13}^{i} & t_{x}^{i} \\ r_{21}^{i} & r_{22}^{i} & r_{23}^{i} & t_{y}^{i} \\ r_{31}^{i} & r_{32}^{i} & r_{33}^{i} & t_{z}^{i} \\ 0 & 0 & 0 & 1 \end{array} \right]$$

By default, OpenCV use a simple radial tangential distortion model where only $k_{1}$, $k_{2}$, $p_{1}$ and $p_{2}$ are computed (all other distortion parameters are set to $0$). In this case, the computation of $\Delta{}$ is:

$$\Delta{}\left({} \left[ \begin{array}{c} x_{p} \\ y_{p} \\ \alpha{} \end{array}\right]{}\right){}\ =\ \left[ \begin{array}{c} x_{p} (1 + k_1 r^2 + k_2 r^4) + 2 p_1 x_{p} y_{p} + p_2(r^2 + 2 x_{p}^2)  \\ \\ y_{p} (1 + k_1 r^2 + k_2 r^4) + p_1 (r^2 + 2 y'^2) + 2 p_2 x_{p} y_{p} \\ \\ \alpha{} \end{array}\right],\ r\ =\ x_{p}^{2}+y_{p}^{2}$$

OpenCV enable to compute calibration with the function cv.calibrateCamera. This function determine all the parameters involved within projection equations (including the optical distortion):

ret cv.calibrateCamera(objectPoints, imagePoints, imageSize, cameraMatrix, distCoeffs, rvecs, tvecs, flags = 0)

where its parameters are:

  • objectPoints: The 3D points (reference points) that compose the calibration reference (the chessboard in our case).
  • imagePoints: The 2D projection of the reference points for each image (the refined corners in our case)
  • cameraMatrix: a $3\times{}3$ matrix that contains camera intrinsics parameters: $$\left[ \begin{array}{c} f_{u} & 0 & c_{u} \\  0 & f_{v} & c_{v} \\ 0 & 0 & 1\end{array}\right]$$
  • distCoeffs: The coefficient of the distortion model. By default (see flags): $$\left[ \begin{array}{c} k_{1} & k_{2} & p_{1} & p_{2}\end{array}\right]$$
  • rvecs: A tuple of $i$ vectors that represent the images pose rotational component (i-e for an image $i$, the values $r_{xx}^{i}$)
  • tvecs: A tuple of $i$ vectors that represent the image pose translation $(t_{x}^{i},\ t_{y}^{i},\ t_{z}^{i})$
  • flags: Properties and parameters for the calibration process, 0 by default (see OpenCV documentation for further use)

With all these information, camera calibration can be performed.

Exercise 9.

Modify the program calibration_chessboard.py by modifying the function calibrate_chessboard(image_files) from:

for image_file in image_files:

	print('  - Processing '+image_file)

	...
        
plt.show()

to

for image_file in image_files:

	print('  - Processing '+image_file)

	...
    
# Camera calibration computation  
print('')    
print('Calibrating camera...', end='')
    
camera_matrix = np.zeros((3, 3, 1), np.float32)
dist_coefs = np.zeros((5, 1, 1), np.float32)

retval, camera_matrix, dist_coefs, rvecs, tvecs = cv.calibrateCamera(points_3d, points_2d, (image_height, image_width), camera_matrix, dist_coefs)

print(' [DONE]')

print('')
print('Camera matrix:')
print(camera_matrix)
print('')
print('Camera distortion:')
print(dist_coefs)
  
plt.show()

Execute the program and check the camera calibration values.

 

4. Calibration result

Once the calibration is computed, the results can be interpreted. For the sake of simplicity and a better software modularity, it is interesting to refactor the original code to separate the calibration computation and the processing of its results.

The first step is to add chessboard size (rows, columns) as parameters of the calibration function and to keep a list of the images that have been used during the calibration process.

Exercise 10.

Modify the program calibration_chessboard.py by modifying the function calibrate_chessboard(image_files) from:

def calibrate_chessboard(image_files):

	# Chessboard dimension
	rows = 6
	cols = 11
	size = 1.0

	# Arrays to store object points and image points from all the images.
	points_2d = []  # 2d points in image plane
	points_3d = []  # 3d point in real world space

	# Image size
	image_width = -1
	image_height = -1

to:

def calibrate_chessboard(image_files, chessboardDimension: tuple, chessBoardSize: float=1.0):
	"""Calibrate a camera using a chessboard as reference points set.

	:param image_files: The files of the images that have to be used
	:type image_files: list
	:param chessboardDimension: The dimension of the chessboard (i-e the number of corners by rows and by columns)
	:type chessboardDimension: tuple
	:param chessBoardSize: The size of a chessboard square (in terrain unit). By default set to 1.0
	:type chessBoardSize: float
	"""
    
	# Chessboard dimension
	rows = chessboardDimension[0]
	cols = chessboardDimension[1]
    
	size = chessBoardSize

	# Arrays to store object points and image points from all the images.
	points_2d = []  # 2d points in image plane
	points_3d = []  # 3d point in real world space

	# Image size
	image_width = -1
	image_height = -1

	used_images = []  # The image that have been used udring calibration

The used_image variable has to be updated with all images that are selected during the calibration process by modifying:

# If found, add object points, image points (after refining them)
if ret is True:
            
	# termination criteria
	criteria = (cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001)
    
	corners_subpix = cv.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria)
            
	# Update 2D / 3D corner coordinates
	points_3d.append(reference_points)
	points_2d.append(corners_subpix)

with:

# If found, add object points, image points (after refining them)
if ret is True:
            
	# termination criteria
	criteria = (cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001)
    
	corners_subpix = cv.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria)
            
	# Update 2D / 3D corner coordinates
	points_3d.append(reference_points)
	points_2d.append(corners_subpix)
            
	# Update used images
	used_images.append(image_file)

As the function calibrate_chessboard has been modified, the main function has to be updated by replacing:

def main():

	# Load images from data folder
	image_files = glob.glob('data/google_pixel_6/chessboard/6x11/*.jpg')

	calibrate_chessboard(image_files)

with:

def main():

	# Load images from data folder
	image_files = glob.glob('data/google_pixel_6/chessboard/6x11/*.jpg')

	calibrate_chessboard(image_files, (6, 11), 1.0)

Ensure that the modified program is still running.

 

The function that compute the calibration has to become an utility function that only make processing and returs results. The results that have been produces has to be returned to the main program.

Exercise 11.

Modify the program calibration_chessboard.py and add a return instruction to the  function calibrate_chessboard in order to return the computed results:

# Camera calibration computation  
print('')    
print('Calibrating camera...', end='')
    
camera_matrix = np.zeros((3, 3, 1), np.float32)
dist_coefs = np.zeros((5, 1, 1), np.float32)

retval, camera_matrix, dist_coefs, rvecs, tvecs = cv.calibrateCamera(points_3d, points_2d, (image_height, image_width), camera_matrix, dist_coefs)

plt.show()
    
return retval, used_images, camera_matrix, dist_coefs, rvecs, tvecs, reference_points, points_2d

The main function has to be modified to retrieve the returned parameters:

def main():

	# Load images from data folder
	image_files = glob.glob('data/google_pixel_6/chessboard/6x11/*.jpg')

	retval, used_images, camera_matrix, dist_coefs, rvecs, tvecs, reference_points, points_2d = calibrate_chessboard(image_files, (6, 11), 1.0)

 

With all results available from the main program, it is possible to modularize the displays. The first step is to externalize the display of the chessboard corners detection from the calibrate_chessboard function.

Exercise 12.

Modify the program calibration_chessboard.py by adding the function display_detection(image_files, projections) defined such as:

def display_detection(image_files, projections):
	"""Display the reference points projections on a set of images.

	:param image_files: The files of the images that hold projections
	:type image_files: list
	:param projections: A list of 2D points set that represents the projections accociated to an image
	:type projections: list
	"""
    
	image_index = 0 # The index of the current image
    
	for image_file in image_files:

		# Load image
		img = cv.imread(image_file)
        
		# Display image with corners
		plt.figure(image_file) # Create a new window
		plt.imshow(img)        # Display the image
        
		# Display the projections associated to the image
		for corner in projections[image_index]:
			plt.plot(corner[0][0], corner[0][1], marker='+', color='red')
        
		image_index = image_index + 1

This function reuse the display code that was originally within calibrate_chessboard. Now all the display related code present within the function can be deleted by modifying:

# If found, add object points, image points (after refining them)
if ret is True:
            
	# termination criteria
	criteria = (cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001)
    
	corners_subpix = cv.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria)
            
	# Update 2D / 3D corner coordinates
	points_3d.append(reference_points)
	points_2d.append(corners_subpix)
            
	# Update used images
	used_images.append(image_file)
            
	# Display image with corners
	plt.figure(image_file) # Create a new window
	plt.imshow(img)

	for corner in corners_subpix:
		plt.plot(corner[0][0], corner[0][1], marker='+', color='red')

else:
	print('No chessboard corner found on image '+image_file)
    
# Camera calibration computation  
print('')    
print('Calibrating camera...', end='')
    
camera_matrix = np.zeros((3, 3, 1), np.float32)
dist_coefs = np.zeros((5, 1, 1), np.float32)

retval, camera_matrix, dist_coefs, rvecs, tvecs = cv.calibrateCamera(points_3d, points_2d, (image_height, image_width), camera_matrix, dist_coefs)

plt.show()

 into:

# If found, add object points, image points (after refining them)
if ret is True:
            
	# termination criteria
	criteria = (cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001)
    
	corners_subpix = cv.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria)
            
	# Update 2D / 3D corner coordinates
	points_3d.append(reference_points)
	points_2d.append(corners_subpix)
            
	# Update used images
	used_images.append(image_file)

else:
	print('No chessboard corner found on image '+image_file)
    
# Camera calibration computation  
print('')    
print('Calibrating camera...', end='')
    
camera_matrix = np.zeros((3, 3, 1), np.float32)
dist_coefs = np.zeros((5, 1, 1), np.float32)

retval, camera_matrix, dist_coefs, rvecs, tvecs = cv.calibrateCamera(points_3d, points_2d, (image_height, image_width), camera_matrix, dist_coefs)

To maintain the display active, the main function has to be updated as follows:

def main():

	# Load images from data folder
	image_files = glob.glob('data/google_pixel_6/chessboard/6x11/*.jpg')

	retval, used_images, camera_matrix, dist_coefs, rvecs, tvecs, reference_points, points_2d = calibrate_chessboard(image_files, (6, 11), 1.0)

	# Display detection results
	display_detection(used_images, points_2d)

	# Display all the components
	plt.show()

Run the program and ensure that the chessboard detection on images is well displayed. 

 

The calibration function also return the 3D position of the reference points (within the reference_points variable) and the 3D positions of the images (stored within the tvecs variable). It is possible to display all these informations within MatPlotLib.

Before displaying the results, some computations have to be done has OpenCV:

  • Give the camera pose as a couple rotation vector (rvec), translation vector (tvec) instead a $4\times{}4$ transform
  • The camera pose represent the pass from camera referential to global referential, that is the inverse of the transform we are expecting

Computing the image position from its rvec and tvec values need first to convert the rotatrion vector into a $3\times{}3$ rotation matrix. This can be done using the code:

rotation_matrix = np.eye(3)
cv.Rodrigues(rvec, rotation_matrix)

The first line initialize a $3\times{}3$ matrix and the second line transform the rotation vector into a rotation matrix using Rodrigues transformation.

Once the rotation matrix is obtained, the image position (tvec) has to be inverted according the rotation matrix as OpenCV gives the inverse of the expectedtransform. This can be done using:

position = -rotation_matrix.transpose().dot(tvec)

It is now possible to display all the positions.

Exercise 13.

Modify the program calibration_chessboard.py by adding the function display_positions(reference_points, images_positions, image_rotations) defined such as:

def display_positions(reference_points, images_positions, image_rotations):
	"""Display the reference points and the positions of the images poses.

	:param reference_points: The 3D coordinates of the reference points
	:type reference_points: list
	:param images_positions: Yhe 3D coordinates of the images positions
	:type images_positions: list
	""" 

	# Initialize the MatPlotLib window
	plt.figure('3D positions')

	# Initializing 3D capabilities
	axes = plt.axes(projection="3d", proj_type='ortho')
    
	# Setting axis properties
	axes.set_xlim(-10, 10) # X Axis graduation
	axes.set_ylim(-10, 10) # Y Axis graduation
	axes.set_zlim(-10, 10) # Z Axis graduation
	axes.set_xlabel('X') # X Axis label
	axes.set_ylabel('Y') # Y Axis label
	axes.set_zlabel('Z') # Z Axis label
	axes.xaxis.label.set_color('red') # X Axis color
	axes.yaxis.label.set_color('green') # Y Axis color
	axes.zaxis.label.set_color('blue') # Z Axis color
	axes.tick_params(axis='x', colors='red') # X Axis graduation color
	axes.tick_params(axis='y', colors='green') # Y Axis graduation color
	axes.tick_params(axis='z', colors='blue') # Z Axis graduation color

	# Display reference points
	for reference_point in reference_points:
		plt.plot(reference_point[0], reference_point[1],reference_point[2], marker='+', color='red')
        
	# Display camera positions
	for index in range(len(images_positions)-1):

		image_position = images_positions[index]
		image_rotation = image_rotations[index]

		# Computing rotation matrix from toration vector
		rotation_matrix = np.eye(3)
		cv.Rodrigues(image_rotation, rotation_matrix)

		# Inverting 3D transform as OpenCV express transforms from camera to reference
		position = -rotation_matrix.transpose().dot(image_position)

		plt.plot(position[0], position[1], position[2], marker='*', color='green')

This function display within MatPlotLib the reference points (the chessboard corners) as red cross and the image positions as green stars.

It is pobbible to activate this display by modifying the main function as follows:

def main():

	# Load images from data folder
	image_files = glob.glob('data/google_pixel_6/chessboard/6x11/*.jpg')

	retval, used_images, camera_matrix, dist_coefs, rvecs, tvecs, reference_points, points_2d = calibrate_chessboard(image_files, (6, 11), 1.0)

	# Display detection results
	#display_detection(used_images, points_2d)
    
   	# Display reference points and camera positions
	display_positions(reference_points, tvecs, rvecs)

	# Show the images and the projections    
	plt.show()

Run the program and ensure that the reference points and the cameras positions are well displayed.

 

Camera calibration is the preliminary step in visual-based 3D reconstruction. Numerous tools are available for reconstructing scenes or objects in 3D from images, using camera calibration parameters as input data. It is therefore important to be able to save these parameters, for example within a file.

Exercise 14.

Modify the program calibration_chessboard.py by adding the function export_camera_parameters(camera_matrix, dist_coefs) defined such as:

def export_camera_parameters(camera_matrix, dist_coefs):
    
	with open('camera_parameters.txt', 'w') as file:
		dist_coefs_count = max(dist_coefs.shape)
		print('dist: ', dist_coefs_count)
		print('dist: ', dist_coefs)

		# Extract camera matrix values
		fx = camera_matrix[0][0]
		fy = camera_matrix[1][1]
		cx = camera_matrix[0][2]
		cy = camera_matrix[1][2]

		if dist_coefs_count == 4:
			file.write('# Type fx fy cx cy k1 k2 p1 p2\n')
			file.write('OPENCV '+str(fx)+' '+str(fy)+' '+str(cx)+' '+str(cy))
			file.write(' '+str(dist_coefs[0][0])+' '+str(dist_coefs[1][0])+' '+str(dist_coefs[2][0])+' '+str(dist_coefs[3][0])+'\n')

		elif dist_coefs_count == 5:
			file.write('# Type fx fy cx cy k1 k2 p1 p2 k3\n')
			file.write('OPENCV_RADIAL '+str(fx)+' '+str(fy)+' '+str(cx)+' '+str(cy))
			file.write(' '+str(dist_coefs[0][0])+' '+str(dist_coefs[1][0])+' '+str(dist_coefs[2][0])+' '+str(dist_coefs[3][0])+' '+str(dist_coefs[4][0])+'\n')

Perform a call to the function export_camera_parameters by modifying the main function as follows:

def main():

	# Load images from data folder
	image_files = glob.glob('data/google_pixel_6/chessboard/6x11/*.jpg')

	retval, used_images, camera_matrix, dist_coefs, rvecs, tvecs, reference_points, points_2d = calibrate_chessboard(image_files, (6, 11), 1.0)

	export_camera_parameters(camera_matrix, dist_coefs)

	# Display detection results
	display_detection(used_images, points_2d)
    
   	# Display reference points and camera positions
	display_positions(reference_points, tvecs, rvecs)

	# Show the images and the projections    
	plt.show()

Run the full program and check within the created camera_parameters.txt file the calibration parameters.

You have now a complete program that can perform a camera calibration.

5. Custom camera calibration

Using the calibration_chessboard.py program, it is possible to calibrate any camera according to a chessboard.

Exercise 15.

Choose a camera (for example a smartphone) and process to its calibration. For that:

  1. Ensure that all the camera settings are correct:
    • No auto focus
    • Fixed focal length
    • Fixed image orientation
  2. Choose an adapted chessboard
  3. Modify the calibration_chessboard.py main function according to the chessboard (the number of rows / cols)
  4. Run the program
Vision par Ordinateur