3D Stereo Reconstruction using iPhone Devices

1. 3D Stereo Reconstruction using iPhone Devices • Performed By: • Ron Slossberg • Omer Shaked • Supervised By: • Aaron Wetzler Final Presentation 24/12/2012

2. Project’s Goal • Building a self-contained mobile 3D reconstruction system using two iPhone devices

3. Background – Pinhole Camera Model • The basic camera model • Transformation from 3D to 2D coordinates: • Distortion also taken into account

4. Background – Stereo Vision • Combine images from two cameras to generate depth image • Relative cameras’ positions in physical space (R, T) and image planes space (F) retrieved by stereo calibration process

5. Background – Stereo Correspondence • OpenCV offers a number of algorithms for stereo correspondence • We chose to use two algorithms that offer a good compromise between efficiency and quality: • Block Matching • Semi Global Block Matching

6. Background – Stereo Correspondence (Cont.) • Block Matching: • Looks at blocks of pixels along the epipolar lines and finds matches according to cross correlation. • Example of typical result:

7. Background – Stereo Correspondence (Cont.) • Semi Global Block Matching: • Adds on to the normal block matching algorithm by introducing global consistency constraints • The constraints are introduced by aggregating matching costs along several independent, one-dimensional paths across the image • Example of typical result:

8. Background – Reconstructed Scene • The matching algorithm produces a disparity map which is a gray scale image where every color corresponds to a certain disparity and thus a certain depth • Using a reprojection matrix on the disparity map we obtain a point in space corresponding to each pixel • We can render these points as a 3d mesh using the original picture colors for each vertex.

9. Programming Environment • iPhone app programming • Objective-C programming • Model-View-Controller design pattern • Main implemented features: • Displaying and controlling the views • Inter-device communication and time synchronization • Persistent storage of data

10. Programming Environment (cont.) • OpenCV libraries • C++ open-source code • Implement all the required algorithms for performing the calibration and reconstruction processes • Handle the interface with the iPhone’s camera • Main implemented features: • Integrating openCV functions into our iPhone app • Correct data flow into and out of every openCV function

11. Programming Environment (cont.) • OpenGL ES libraries • A lightweight version of the open-source OpenGL libraries, which includes an iOS API • Implement the framework for rendering 2D and 3D computer graphics • Main implemented features: • Displaying the reconstructed images as an interactive 3D surface

12. Software High-Level Design Main Menu Connect Devices • Generating a Bluetooth session between the devices

13. Software High-Level Design Main Menu Settings • Setting the right parameters for stereo calibration and reconstruction

14. Software High-Level Design Main Menu Calibration • Performing stereo calibration for the devices using a chessboard pattern

15. Software High-Level Design Main Menu Reconstruction • Performing 3D stereo reconstruction of the images captured by the devices

16. Software High-Level Design Main Menu Photo Album 3D Image Display • Interactive 3D color display of the images • Disparity map images shown within table

17. Time For LIVE Demo AND MOVIE

18. Calibration Process Flow Bluetooth session created Initial State Initial State Message delay calculated Validating message delay Calibration Calibration Initializing camera Capture Send Capture Order Wait Message Delay Send Capture Indication Capture Image Capture Image Exchange image corners data Extract Corners Extract Corners

19. Calibration Process Flow(Cont.) Performed separately at each device Calibrate Compute Intrinsic Parameters Process was separated to increase accuracy Compute Extrinsic Stereo Parameters Save Parameters

20. Reconstruction Process Flow Bluetooth Session Created Message delay calculated Initial State Initial State Calibration Performed Validating message delay Reconstruction Reconstruction Initializing camera Load parameters Load parameters Compute undistorted rectified bitmap Compute undistorted rectified bitmap Choose Reconstruction Algorithm Choose Reconstruction Algorithm

21. Reconstruction Process Flow(Cont.) Save Disparity Image Save Disparity Image Capture Send Capture Order Wait Message Delay Send Capture Indication Capture Image Capture Image Send and Receive Image Send Images Send and Receive Image Remap Images to get Rectified Images Remap Images to get Rectified Images Compute Disparity by stereo correspondence Compute Disparity by stereo correspondence

22. Implementation Issues • Simultaneous Photo Capture • Problem: need devices to capture images at the same time to gain good results • Solution: implemented RTT calculation algorithm that ignores anomalous results and performs update phases during operation • Other Solutions: Web Service, GPS • Advantage: messages traverse only short distance, no dependency on GPS signals • Main Disadvantage: GPS achieves better accuracy

23. Implementation Issues(Cont.) • Inter-Device Communication • Problem: need to pass messages and data between the two devices • Solution: Bluetooth communication • Other Solutions: Wi-Fi • Advantages: existing easy-to-use framework, simple protocol with low overhead • Main Disadvantage: smaller BW (affects the duration of the reconstruction process)

24. Summary • Very challenging and enjoyable project • Introduction with both computer vision and mobile app development • Final outcome is a stable, user-friendly app providing live results • Helpful documentation for future usage • Detailed webpage and demo movie • Finally – many thanks to Aaron for his guidance and support throughout this project !

25. References • Computer Vision course, Spring 2010, University of Illinois • http://www.cs.illinois.edu/~dhoiem/courses/vision_spring10/lectures/ • Developing Apps for iOS, Paul Hegarty, Stanford fall 2010 course (available on iTunes) • Multiple View Geometry in Computer Vision course, University of North Carolina • http://www.cs.unc.edu/~marc/mvg/slides.html • Modeling the Pinhole camera, course lecture, University of Central Florida • Computer Vision tutorial, GIP lab • Learning OpenCV, Gary Bradski & Adrian Kaehler • Stereo Vision using the OpenCV library by Sebastian Droppelmann, Moos Hueting, Sander Latour and Martijn van der Veen

26. The End Comments & Questions