Camera calibration


Cameras have become commodity products - with inexpensive cameras selling from $30 or so to expensive cameras that can cost thousands of dollars. No matter how good a camera, when a camera captures a real life object into an image, some distortion gets introduced. The slide deck here gives an overview of the type of distortions caused during image capture, and the theory behind  camera calibration. Here we will look at sample code for camera calibration.

Calibration is the process of calculating the parameters that cause distortion. Remapping is the part where the parameters obtained from calibration can be used for a variety of purposes:

  • Correct the distortion that is caused when a real life object is captured by a less-than-optimal camera. For example, fish eye lenses can distort an image. This can be corrected to some extent using camera calibration.
  • Determine the real world coordinates (for example in millimeters or inches) corresponding to a certain object or measurement in the camera coordinates (pixels)  

The code and data for this demo can be downloaded from here. On unzipping the code, the following directory structure should be visible: 

acv@acv-vm:~/acv$ ls
bin  data  src
acv@acv-vm:~/acv$ 
Change to the bin folder and run cmake and make. The command line should show as follows: 
acv@acv-vm:~/acv/bin$ cmake ../src/
camera_calibration/ util/               
acv@acv-vm:~/acv/bin$ cmake ../src/camera_calibration/
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1") 
-- Found OpenCV: /usr/local (found version "4.4.0") 
-- Configuring done
-- Generating done
-- Build files have been written to: /home/acv/acv/bin
acv@acv-vm:~/acv/bin$ make
Scanning dependencies of target demo
[ 20%] Building CXX object CMakeFiles/demo.dir/CameraCalibration.cpp.o
[ 40%] Building CXX object CMakeFiles/demo.dir/home/acv/acv/src/util/BinaryHeap.cpp.o
[ 60%] Building CXX object CMakeFiles/demo.dir/home/acv/acv/src/util/ImageUtilities.cpp.o
[ 80%] Building CXX object CMakeFiles/demo.dir/home/acv/acv/src/util/LogManager.cpp.o
[100%] Linking CXX executable demo
[100%] Built target demo
 
The camera calibration process needs to estimate a transform from the real world coordinates (represented by matrix X in the image below) to the image pixel coordinates (represented by matrix U).

During camera calibration the matrix H is computed. In the openCV implementation, multiple correction matrices are computed instead of a single H. To get these matrices, a geometrical pattern can be printed and analyzed. A popular pattern is the checkerboard. Pixel positions of the known geometric pattern are stored in one array (For example, line 47, imageCoordinates2D in CameraCalibration.cpp). The real world coordinates of the geometrical pattern are stored in another array (For example, line 50, worldCoordinates3D in CameraCalibration.cpp). These are passed to a calibration function to get the required matrices for image correction. Depending on the camera used, the correction can be performed using functions suitable for a pinhole camera (Function cv::calibrateCamera in CameraCalibration.cpp), or for a fish eye lens camera (Function cv::fisheye::calibrate in CameraCalibration.cpp). 

Instead of a checkerboard pattern, any other geometric pattern or standard known image can be used. There are deep learning implementations that  can "learn" distortion parameters from an image. The checkerboard pattern is popular because corners of each block are easy to detect. The corners provide a large number of reliable points to obtain the calibration matrix. 

It is a good idea to use a checkerboard that has different height and width. For example 8X5, 9X6 etc. Most corner detection implementations will be able to detect all corners even when boards are placed at 90-degree angles or more. A large collection of checkerboards that you can use is here. The OpenCV implementation will detect checker board corners that are on the "inside". For example, in the checkerboard given in this file, the number of corners detected will be 8x6.

Once a checkerboard pattern is printed, multiple images can be taken using the camera that needs to be calibrated. Note that one has to input world coordinates for each corner. Typically, the distance between corners is assumed to be constant (unity) and the depth is also set to a constant value, even zero. See for example, lines 52-54 in CameraCalibration.cpp: 

for (int y = 0 ; y < checkerBoardHeight; y++)
	  for (int x =0; x < checkerBoardWidth; x++)
	    worldCoordinatesForOneCheckerboard.push_back(cv::Point3f(x, y, 0));

Once you have printed a checkerboard pattern, the kind of calibration images takes will depend on the target application. If the goal is to make precise measurements of objects, all calibration images need to be taken from a similar distance. If the goal is to correct an image across a range of distances (typically more difficult), images may be taken at those distance ranges.

There are five sets of sample calibration files in the folder "../data/camera_calibration/". The code can take images from any of these sets using command line arguments. For example, to use set 4, one would run the following command "./demo 4 false ../data/camera_calibration/set4/left04.jpg", and the result on the command prompt would be as shown: 

./demo 4 false ../data/camera_calibration/set4/left04.jpg
Checkerboard dimensions: width=6, height=9
Success 
Success 
Success 
Success 
Success 
Success 
Success 
Success 
Success 
Success 
Success 
Success 
Success 
rmsError : 0.408696
cameraMatrix : [536.0734367784613, 0, 342.3703824316232;
 0, 536.0163520809057, 235.5368541582433;
 0, 0, 1]
distCoeffs : [-0.265090110384994, -0.04674355191422817, 0.001833009318834493, -0.000314714821782175, 0.2523150938276316]
Rotation vector : [-0.08398745005176021, 0.3480280272277098, -1.542441254482061;
 -0.2258469406319553, 1.015511394982941, -2.794706289918831;
 -0.3469824864495253, -0.06738502637250668, -1.20088994646176;
 -0.2752733700049721, 0.1012335781634614, -1.562965657769406;
 -0.4735952746610879, 0.08970849079436399, -0.2260597730613408;
 0.06525918255397294, 0.4470184967726069, 0.1080001117573061;
 -0.1014161077054746, 0.3203483537936572, 0.3147293024132585;
 -0.3533906917454183, 0.2407187932109452, 0.2097002909307452;
 0.4828725027297215, -0.1703699921876566, -1.407403337518957;
 0.05280145647979654, -0.6017182096691464, -0.1845382236162483;
 -0.3746335993528633, 0.06982834946911369, -0.01937107941020187;
 0.4954233075152011, 0.1194882282179227, -0.2967596244234337;
 0.1972109021446256, -0.4200994448124338, -0.1949708029357463]
Translation vector : [-2.962181350251924, 0.5715900521471102, 16.83013788490948;
 2.533996722644027, 4.319992120307035, 13.71918906338706;
 -3.427434026609225, 0.4873823711674588, 11.56153537511424;
 -3.993878790789128, 2.27704403385266, 12.68878152538807;
 -2.51791607262065, -3.430690485057895, 12.85702179383783;
 2.207421037791208, -3.214465432621518, 15.60125335951144;
 -3.725851223232848, -4.310847843295919, 17.20439594753487;
 -1.590038546517381, -4.31771174613294, 14.01040703212592;
 -3.502645008398289, 1.615954616304321, 11.97222436416522;
 -2.168386116050941, -3.500111019153073, 10.73694879694808;
 -2.958485037056025, -3.944179153778983, 13.21423804939278;
 -3.405572983056168, -2.410422567525038, 12.58706851677992;
 -2.676427611382639, -3.189455566065201, 10.58262299550807]
The above result is giving an RMS error of 0.408696, which is a reasonably good measure. In order to test the code on images used with fish eye lenses, run the following command line argument and you should see the following result: 
./demo 5 true ../data/camera_calibration/set5/image-001.jpg 
Checkerboard dimensions: width=9, height=6
Success 
Success 
Success 
Success 
Success 
FishEye Lens, rmsError : 257.749
cameraMatrix : [456.032807281703, -49.09183958055493, 360.0115079494736;
 0, 482.9340404961708, 618.37469919714;
 0, 0, 1]
distCoeffs : [-0.07345406782823241;
 0.2658070458034077;
 -0.09650620904598614;
 0.02542927465181658]
Rotation vector : [0.4200834608359635, 0.5704444029835147, -0.009728899715657024;
 0.4651190222558045, -0.2649392718065864, -0.1257251172234883;
 -147703636.5185569, -269735612.0450727, 116273769.8290087;
 0.1017240669940113, 0.6448126913119389, 0.09617868129832943;
 0.8945465960172689, -0.1278554116836606, -0.285062442643213]
Translation vector : [1.232325282249292, -5.987205782927052, 6.871228174216364;
 -2.301634681349038, -5.272728286085483, 4.653899667341047;
 437890.7245563843, 1608301.740340468, -754766.2621698047;
 -0.1156795333850582, -5.710543338123093, 4.981837768192643;
 -4.03278546514446, -7.088518700948, 2.195026861144807]

Comments

Post a Comment

Popular posts from this blog

Discrete Wavelet Transform

Code and sample files for this demo can be downloaded from here . Code uses only libraries from the ACV-VirtualMachine.No extra libraries are needed. You can unzip the code to any folder on your system. All paths used in the code are relative. We are using the folder "~/acv/code/dwt_demo/" Compiling and running the code is fairly straight forward. You only need to run the following two commands: acv@acv-vm : ~/acv/code/dwt_demo/bin $ cmake ../src/wavelet_demo_others/ -- The C compiler identification is GNU 7.5.0 -- The CXX compiler identification is GNU 7.5.0 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - ...
Read more

Geometric image transforms

Image
 The sample code and data for this tutorial can be downloaded from here . The result files are here . The accompanying slide deck for this tutorial is here . After downloading the code, run the following commands:  acv@acv-vm : ~/vision/practice/bin $ cmake ..; make; ./homework transformDemo -- The C compiler identification is GNU 7.5.0 -- The CXX compiler identification is GNU 7.5.0 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1") -- Found OpenCV: /usr/local (found...
Read more

Testing image capture with high speed cameras

Image
In industrial systems, objects may be moving along conveyor belts or carried on pick and place machines. A camera based inspection solution for such situations needs to take into accoun that the object in question may not be stationery, or at a fixed distance from the camera. Ergo, the captured image may not contain the object in focus, or not have the object in view at all. One option would be to stop the conveyor belt to take a photo of the object:   However, stopping a conveyor belt is not always possible. You would need cameras that can capture a clear image even when objects are moving. Enthusiasts naiively assume that one can take a video and perform analysis on individual frames of the video. Unless the video has the correct frame rate and good resolution, the solution would not work. Assume that you need to implement an industrial imaging solution to monitor items moving on a conveyor belt. You should consider specifications such as these: Distance from camera t...
Read more