1 / 16

Mariolino De Cecco , Nicolo Biasi , Ilya Afanasyev

Reprojection of 3D points of Superquadrics Curvature caught by Kinect IR-depth sensor to CCD of RGB camera. Facoltà di Ingegneria. Mariolino De Cecco , Nicolo Biasi , Ilya Afanasyev. Trento, 2012. Content. Kinect installation. Kinect calibration. Kinect outputs.

azra
Download Presentation

Mariolino De Cecco , Nicolo Biasi , Ilya Afanasyev

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Reprojection of 3D points of Superquadrics Curvature caught by Kinect IR-depth sensor to CCD of RGB camera Facoltà di Ingegneria • Mariolino De Cecco, NicoloBiasi, Ilya Afanasyev • Trento, 2012

  2. Content • Kinect installation. • Kinect calibration. • Kinect outputs. • MATLAB Preprocessing Kinect data. • MATLAB Inputs. • SQ Curvature Software. • Tests. • Results. • Links.

  3. Kinect installation 1st attempt • 1. OpenNI backend: • Nicolas Burrus software [1]: Kinect RGBDemo v0.6.1. -> supports OpenNI/Nitebackends [2] and has the experimental infrared support with OpenNI (still buggy) -> I tried to check this work for OpenNI backend under Windows 64bit. • Install Sensor Kinect drivers under Windows 32 bit [3] and OpenNI/Nite modules (Win32) [2] according to the instruction [4] and verified that they work properly. • Use RGB-D Capture [1] to grab Kinect RGB and IR images with intensity.raw and depth.raw files. -> the software doesn’t grab IR images!! -> only RGB image, intensity.raw, depth.raw and calibration file (calibration.yml without distortion parameters)!! • “calibrate-openni-intrinsics --pattern-size 0.0405 grab1 calibration.yml” -> gives “openni_calibration.yml” with Intrinsic matrix and distortion coefficients for Kinect RGB camera !! • “calibrate-openni-depth.exe --pattern-size 0.0405 grab1” gives partly processed figures with a message about errors.

  4. The comparison of Kinect calibration results for - Libfreenect backend (under OS Linux / Ubuntu 10.10 - 32 bit) - OpenNI backend (under OS Windows 32bit) Libfreenect backend OpenNI backend The Depth Camera Calibration for OpenNI backend is NOT available!!! • The same figures of Intrinsic Matrixes for Kinect RGB and Depth Cameras

  5. The comparison of Kinect calibration results for - Libfreenect backend (under OS Linux / Ubuntu 10.10 - 32 bit) - OpenNI backend (under OS Windows 32bit) Libfreenect backend OpenNI backend The reprojection of 3D points from Kinect Depth Camera to RGB image for OpenNI is NOT possible!! • There is NO extrinsic mapping between Kinect Depth and RGB cameras for OpenNI backend

  6. Kinect installation 2nd attempt • 1. Libfreenect backend (Windows): • Nicolas Burrus software [1]: Kinect RGBDemo v0.6.1. -> supports Libfreenect backend [2] and has RGB and infrared support -> I tried to check the work under Windows 64bit. • Install Xbox NUI Motor drivers under Windows 32 bit [3] and OpenNI/Nite modules (Win32) [2] according to the instruction [4] and verified that they work properly. • Use RGB-D Capture [1] to grab Kinect RGB and IR images with intensity.raw and depth.raw files. -> the software doesn’t grab IR images!! -> only RGB image, intensity.raw, depth.raw and calibration file (calibration.yml without distortion paraeters)!! • “calibrate-openni-intrinsics --pattern-size 0.025 grab1 calibration.yml” -> gives “openni_calibration.yml” with Intrinsic matrix and distortion coefficients for Kinect RGB camera !! • “calibrate-openni-depth.exe --pattern-size 0.025 grab1” gives partly processed figures with a message about errors.

  7. Kinect installation • 1. OpenNI backend: • Nicolas Burrus software [1]: Kinect RGBDemo v0.6.1. -> supports OpenNI/Nitebackends [2] and has the experimental infrared support with OpenNI (still buggy) -> I tried to check this work for OpenNI backend under Windows 64bit. • Install Sensor Kinect drivers under Windows 32 bit [3] and OpenNI/Nite modules (Win32) [2] according to the instruction [4] and verified that they work properly. • Use RGB-D Capture [1] to grab Kinect RGB and IR images with intensity.raw and depth.raw files. -> the software doesn’t grab IR images!! -> only RGB image, intensity.raw, depth.raw and calibration file (calibration.yml without distortion paraeters)!! • “calibrate-openni-intrinsics --pattern-size 0.025 grab1 calibration.yml” -> gives “openni_calibration.yml” with Intrinsic matrix and distortion coefficients for Kinect RGB camera !! • “calibrate-openni-depth.exe --pattern-size 0.025 grab1” gives partly processed figures with a message about errors. New Idea

  8. 5. MATLAB Inputs. • Cube_Curvatura_20110925\main_reproject.m • -> color.png (480x640 pixels) • -> calibData.mat: • - matrixes of intrinsic parameters for IR camera (K_ir) and RGB camera (K_rgb) • - extrinsic mapping between ir (depth) and rgb Kinect camera (R,T) • - distortion coefficients (kc_ir, kc_rgb) • -> Points.mat (N x 6), where N – number of points with (x,y,z,R,G,B) info

  9. 6. MATLAB Software.6.1 Elimination of the ground. • RANSAC search of the ground by using the plane equation.

  10. 6.2 3D Points Reprojection. Transformation from IR (Depth) to RGB camera reference system: P_rgb = R * P_ir + T, where R,T - extrinsic mapping between ir and rgb Kinect cameras; P_rgb, P_ir – points in RGB and IR camera reference systems. Cube Reprojection without considering distortion (up figures). Cube Reprojection with considering distortion (bottom figures). • The distance from Kinect to the cube z = 0.5 m • ??? • Bad calibration?

  11. 6.2 3D Points Reprojection. Cube and Cylinder Reprojection without considering distortion (up figures). Cube and Cylinder Reprojection with considering distortion (bottom figures). • The distance from Kinect to the objects z = 0.6 m • ??? • Bad calibration?!

  12. 6.2 3D Points Reprojection. Transformation from IR (Depth) to RGB camera reference system: P_rgb = R * P_ir + T, where R,T - extrinsic mapping between ir and rgb Kinect cameras; P_rgb, P_ir – points in RGB and IR camera reference systems. Cube Reprojection with calibration parameters from Nicolo (upper figures). Cube Reprojection with calibration parameters from Alberto (bottom figures). • The distance from Kinect to the cube z = 0.5 m • Bad calibration?!

  13. 6.2 3D Points Reprojection. Camera Calibration and 3d Reconstruction [9] for so-called pinhole camera model. The scene view is formed by projecting 3D points into the image plane using a perspective transformation. where - are the coordinates of a 3D point in the world coordinate space. - are the coordinates of the projection point in pixels. A - matrix of intrinsic parameters (does not depend on the scene viewed). - a principal point (that is usually at the image center) - are the focal lengths expressed in pixel-related units. R(T) - the matrix of extrinsic parameters. Cube Reprojection with calibration parameters from Alberto (bottom figures). H A = K_rgb R(T) = [R, T] A R(T)

  14. 6.3 RANSAC fitting SQ to 3D data points. With using Levenberg-Marquardt algorithm of distance minimization from SQ to 3D points. • Red points – outliers. • Blue points – inliers. • Green points – SQ model.

  15. 6.3 Object structure creation and reprojection on image. Reprojection of the lines between the 3D points of cube vertexes to CCD of Kinect RGB camera (figure a) and to the image with reprojected 3D points (figure b). The information about vertex position was obtained in the previous stage of RANSAC cube pose estimation. • Red lines – the cube frameworks • Figure a • Figure b

  16. 9. References Nicolas Burrus. Kinect. RGBDemo, calibrate and visualize Kinect output. // http://nicolas.burrus.name/, 2011. OpenNI Modules. // www.openni.org, SensorKinect drivers. // github social coding. https://github.com/avin2/SensorKinect. How-to: Successfully install Kinect on Windows (OpenNI and NITE). // VangosPterneas blog: http://studentguru.gr/b/vangos/archive/2011/01/20/how-to-successfully-install-kinect-windows-openni-nite.aspx Install OpenKinect for Windows 7 and XP. // http://kinect.dashhacks.com/kinect-guides/2011/01/09/install-openkinect-windows-7-and-xp Camera Calibration and 3d Reconstruction. // OpenCV (Open Source Computer Vision) v2.1 documentation: http://opencv.willowgarage.com/documentation/cpp/camera_calibration_and_3d_reconstruction.html

More Related