C#, OpenCV, and simple Camera/Robot Calibration

1. C#, OpenCV, and simple Camera/Robot Calibration Sebastian van Delden USC Upstate svandelden@uscupstate.edu

2. OpenCV • Open Source Computer Vision Library • OpenCV is Intel’s Open Source Computer Vision Library. • It is a collection of C functions and a few C++ classes that implement some popular Image Processing and Computer Vision algorithms. • EMGU provides wrapper class packages so that OpenCV can be used in C#.

3. Features • Image data manipulation • allocation, release, copying, setting, conversion • Image and video I/O • file and camera based input, image/video file output • Matrix and vector manipulation, and linear algebra routines • products, solvers, eigenvalues, SVD • Various dynamic data structures • lists, queues, sets, trees, graphs • Basic image processing • filtering, edge detection, corner detection, sampling and interpolation, color conversion, morphological operations, histograms, image pyramids

4. Features (cont.) • Structural analysis • connected components, contour processing, distance transform, various moments, template matching, Hough transform, polygonal approximation, line fitting, ellipse fitting, Delaunay triangulation • Camera calibration • finding and tracking calibration patterns, calibration, fundamental matrix estimation, homography estimation, stereo correspondence • Motion analysis • optical flow, motion segmentation, tracking • Object recognition • eigen-methods, HMM • Basic GUI • display image/video, keyboard and mouse handling, scroll-bars • Image labeling • line, conic, polygon, text drawing

5. Emgu CV • Emgu CVis a cross platform .Net wrapper to the Intel OpenCV image processing library. Allowing OpenCV functions to be called from .NET compatible languages such as C#, VB, VC++, IronPython etc. • Home Page: • http://www.emgu.com/wiki/index.php/Main_Page • Online Documentation: • http://www.emgu.com/wiki/files/2.2.1/document/Index.html • Download the executable version and install it. • Version 2.1.0 is on our course website: http://faculty.uscupstate.edu/svandelden

6. Getting Started… Create a new Windows Form Application

7. Add the Emgu Libraries as project References

8. Setting up the Window • Open your toolbox and add a SplitContainer to your Window. • Split it horizontally • This creates two “Panel”s where we can add elements. • Idea: we are going to add the video stream to the below panel and results/data etc to the top panel.

9. You can change the properties of any of the elements in the Windows form by clicking on the Properties to the right…

10. When you are building the Windows Form, you are in “Design” view… Right click on it and choose “View Code” to see the underlining code: Notice that this is a partial class. The InitializeComponent() method is automatically created and creates the Windows Form that you’ve been building.

11. To view the other partial class, click on the “designer” cs file in the Solution Explorer. If you expand this, you can see all the code that sets up the Windows Form. NOTE: Never modify this code directly at all, because it gets re-created every time you make changes to the design of the form!!

12. From your Toolbox, add an ImageBox to the lower panel. Drag he Panel sizes so that it looks like this. Note: if you are having problems selecting the SplitContainer, right click on the form and choose “Select SplitContainer”

13. Add a button labeled “Start Camera” to Panel1, then double click on it to create and view a button1_Click method (event handler)… Include the EMGU libraries. Create two data attributes : Capture _capture is the EMGU CV class for connecting to a camera The boolean will be used in toggling the camera off and on. Button Click event handler.

14. Add a user-defined method called ProcessFrame that will be doing our computer vision stuff. For now, just have it get the camera’s current frame and add it to the ImageBox on the Windows Form.

15. Add this code to the button’s click method: Instantiate the EMGU CV Camera object Toggle the camera on and off.

16. Try Running the program and click the Start Camera button.

17. Try applying a built-in EMGU CV computer vision algorithm to the image. • For example, convert to gray scale image and threshold (cut off 100 and include pixels up to 255:

18. Try it out..

19. Try Using the Inverted Threshold method (so that the background is black) and also do an “closing” – dilation followed by erosion.

20. Camera/Robot Calibration

21. Camera Calibration

22. Camera/Robot Calibration • We will calibrate the camera’s coordinate system with the robot’s world coordinate system. • We assume that the depth (Z) of the points on robot world system are known and equal. • Flat table top of the robot’s work area • This is significant because the robot world points form a plane in 3D space. • Since depth is known, we only need to calibrate camera (X,Y) pairs to robot (X,Y) pairs

23. 3x3 Perspective Transformation • Emgu has a built-in method that will compute the 3x3 perspective transformation based on at least 4 camera-robot (X,Y) pairs. • Need at least 4 points (8 values in total) because we need to recover 8 values: a b c d e f g h 1

24. The Transformation • The transformation captures/recovers: Translation: Scaling: Rotation: Shear:

25. The Emgu CV • HomographyMatrix PerM = CameraCalibration.GetPerspectiveTransform( PointF[ ] CameraPoints, PointF[ ] RobotPoints ); • NOTE: “Homography” – square, invertible matrix.

26. Example Program • Provide for you is a C# calibration program that • connects to the V+ robot via TCP/IP • displays the video from the camera • allows you to click on the image • When you do so, the picture X,Y along with the robot X,Y location of the tool frame are captured. • generate the calibration matrix based on the recovered points.

28. Activity • Use this program to calibrate your camera to your robot.