Semi-automatic Range to Range Registration: A Feature-based Method

1. Semi-automatic Range to Range Registration: A Feature-based Method Chao Chen & Ioannis StamosComputer Science DepartmentGraduate Center, Hunter CollegeThe City University of New York

2. Texture Mapping Registration Motivation • Goal: highly accurate photo-realistic description of 3D world • Applications: urban planning, historical preservation, virtual reality 3D model Photo-realistic 3D Model Highly accurate Range scans Computational efficient Minimum human interaction 2D Pictures

3. Contribution • Related methods on range image registration • Iterative Closest Point algorithm [Besl & McKay, Chen & Medioni, Rusinkiewicz] • Require close initial registration • Spin images [Johnson & Hebert] • More suitable for curved surfaces; needs accurate normal • Previous feature-based algorithm [Stamos & Leordeanu] • Exhaustively searching line pairs; high complexity • Symmetric structures require manual registration • Our semi-automatic 3D registration system • No rough pre-registration required • Automated registration procedure: • Utilize global information to compute transformation • ICP algorithm to optimize registration • Context-sensitive user interface to: • Display registration result at each step • Conveniently adjust translation and rotation

4. Outline • Automated registration procedures • Previous exhaustive search approach • Improved automated registration • Global stitching process to register all images • User interface • Experimental results • Conclusions and future work

5. Exhaustive Search Approach • Range image segmentation Intersection line Each Segmented Planar Area is shown with different color for clarity Range sensing direction Segmented Planar Area Exterior and Interior borders shown Interior border

6. Exhaustive Search Approach • One pair of correctly matched lines provides rotation Line in left range image Line in right range image Plane normal in right image Plane normal in left image xright xleft zleft yright zright yleft Right range image coordinate system Left range image coordinate system

7. Exhaustive Search Approach • Two pairs of correctly matched lines provide exact translation xright xleft zright zleft yright yleft Rotated left range image coordinate system Right range image coordinate system

8. Exhaustive Search Approach • Find the two pairs of corresponding lines that maximizes the total number of line matches • Consider two corresponding line pairs • Compute transformation • Grade of computed transform: total number of line matches • Keep the transform with the highest grade • At the end refine best transform using all matched lines White lines (left scan) Blue lines (right scan) Red/Green lines (matches)

9. Exhaustive Search Approach • No initial registration needed • High computational complexity • Symmetry problem unsolved • Improvements • Extract object-based coordinate system • Context-sensitive user interface

10. Image1 line clustering Image2 line clustering Display registered pair Rotation estimation Wrong registration due to symmetry Translation estimation Rotation adjustment Transform refinement by ICP Translation adjustment Framework of New Solution Lines and planes from segmentation Correct registration Next pair of scans Global stitching Automated Registration User Interactions

11. Line Clustering • Line clustering • Line directions • Plane normals Building’s local coordinate system

12. y1 y2 x1 z1 x2 z2 Rotation Estimation • Rotation estimation R = [x2 y2 z2] T * [x1 y1 z1] 24 possible R’s?

13. Building y1 y2 -z1 -z2 x2 x1 x1 * R = x2 y1 * R = y2 z1 * R = z2 Scanner position 1 Scanner position 2 (0,1,0) * R = R11 : projection of y1 on y2 2 to 5 R’s R11> cos(45°) = 0.7 Return the R with the largest diagonal sum Sort by R00+R11+R22 Rotation Estimation • Heuristic: eliminate candidates based on observations • Scanner moves on the ground plane: y axis not change much • Overlapping images from close by viewpoints: smallest rotation candidate is chosen

14. y1 y1 y2 x1 z1 x1 z1 x2 z2 Translation Estimation • Translation estimation • Left and right axes parallel accordingly after rotation • Pick robust line pairs to estimate translation R

15. y1 y2 z1 z2 x1 x2 T Line pair 1 Line pair 2 Translation Estimation • One pair of matched lines provides an estimated translation • Two pairs with similar estimated translations provide translation candidate

16. T = (d1 + d2) / 2 T  linear system Translation Estimation • Two types of translation candidates y1 y2 z2 z1 x2 d1 x1 d2

17. Translation Estimation • Find the translation that maximizes the total number of line matches • Cluster all estimated T’s, pick 10 most frequently appeared T’s • For each T: • Find all matches, solve linear system to update R&T • Count matched line pairs again • Choose the R&T with the most number of matched line pairs • Refine transformation with ICP

18. Rotation wrong by 90° Need manual adjustment correct Display other rotations Display draggers save Adjust R/T, optimize Choose a new R, compute T save Registration System Flowchart start Read in all image pairs, form the transformation graph Read in one image pair Y Find pivot image, compute path Last pair? N Automated registration Compute transform from each image to pivot image Display to user 1 2 3 Rotation wrong by 90° Need manual adjustment correct Global optimization save Display other rotations Display draggers Global stitching Adjust R/T, optimize Choose a new R, compute T Display and save exit save

19. Correct Rotation & Translation Rotation Symmetry by 90deg. Need Manual Adjustment on R/T User Interface Display window: Points and lines of registered two scans

20. Rotation wrong by 90° Display other rotations Choose a new R, compute T User Interface start Read in all image pairs, form the transformation graph Read in one image pair Y Find pivot image, compute path Last pair? N Automated registration Compute transform from each image to pivot image Display to user 1 2 3 Rotation wrong by 90° Need manual adjustment correct Global optimization save Display other rotations Display draggers Global stitching Adjust R/T, optimize Choose a new R, compute T Display and save exit save

21. User Interface • Rotation wrong by 90 degrees: choose from other candidate rotations computed previously

22. Need manual adjustment Display draggers Adjust R/T, optimize save User Interface start Read in all image pairs, form the transformation graph Read in one image pair Y Find pivot image, compute path Last pair? N Automated registration Compute transform from each image to pivot image Display to user 1 2 3 Rotation wrong by 90° Need manual adjustment correct Global optimization save Display other rotations Display draggers Global stitching Adjust R/T, optimize Choose a new R, compute T Display and save exit save

23. User Interface • Adjusting rotation and translation based on the building’s coordinate system White draggers for translation Blue spheres for rotation

24. Correct Rotation & Translation Rotation Symmetry by 90deg. Need Manual Adjustment on R/T User Interface Display window: Points and lines of registered two scans Exhaustive Search Approach ICP Optimization

25. Global Stitching start Read in all image pairs, form the transformation graph Read in one image pair Y Find pivot image, compute path Last pair? N Automated registration Compute transform from each image to pivot image Display to user 1 2 3 Rotation wrong by 90° Need manual adjustment correct Global optimization save Display other rotations Display draggers Global stitching Adjust R/T, optimize Choose a new R, compute T Display and save exit save

26. Global Stitching • Global stitching for all images • Pivot image – the image with the most number of neighbors • Transform composition – along the strongest path from each image to the pivot image • Further improvement to consider • Global optimization to minimize registration error

27. Experimental Results • Thomas Hunter building, Hunter College • 14 scans, 15 pairs (13 automated, 2 manually adjusted) • 10~20 seconds per pair; a few minutes for entire registration

28. Registration Results • Shepard Hall, City College • 20 scans, 24 pairs (9 automated, 8 R symmetry, 7 adjust R/T) • 20~90 seconds per pair; 1 hour for entire registration video

29. Registration Results • Great Hall (interior of Shepard Hall) • 21 scans, 44 pairs (12 automated, 18 R symmetry, 13 adjust R/T ) • 20~90 seconds per pair; 1.5 hour for entire registration

30. Registration Results • Great Hall interior scene

31. Registration Results • Great Hall interior scene

32. Registration Results • Great Hall interior scene

33. Algorithm Performance Time for automated registration Average error of matching planes Number of matching lines Lines in two scans

34. Algorithm Performance

35. Conclusions • Semi-automatic registration system • Automated 3D registration routines • Context-sensitive user interface • Fast computation, accurate registration • Future work • Global optimization • Extract higher ordered curvatures from range data for faster and more accurate feature-based registration

36. Spin Image Representation Histogram of surface points about a rotation around surface normal at the sample Point at varying radii from the sample point A method of measuring shape and curvature with local support

37. Acknowledgement • NSF CAREER IIS-01-21239 • NSF MRI/RUI EIA-0215962 • Conference committee and all audiences Contact us: • http://www.cs.hunter.cuny.edu/~ioannis/Vision.htm • Ioannis Stamos, istamos@hunter.cuny.edu • Cecilia Chao Chen, cchen@gc.cuny.edu