Long-term image-based motion estimation

1. Long-term image-based motion estimation Dennis Strelow and Sanjiv Singh

2. On the web Related materials: • these slides • related papers • movies • VRML models at: http://www.cs.cmu.edu/~dstrelow/maryland

3. Introduction (1) micro air vehicle (MAV) navigation AeroVironment Black Widow AeroVironment Microbat

4. Introduction (2) mars rover navigation Mars Exploration Rovers (MER) Hyperion

5. Introduction (3) robotic search and rescue Center for Robot-Assisted Search and Rescue, U. of South Florida Rhex

6. Introduction (4) NASA ISS personal satellite assistant

7. Introduction (5) Each of these problems requires: • 6 DOF motion • in unknown environments • without GPS or other absolute positioning • over the long term …and some of the problems require: • small, light, and cheap sensors

8. Introduction (6) Monocular, image-based motion estimation is a good candidate In particular, simultaneous estimation of: • multiframe motion • sparse scene structure is the most promising approach

9. Outline Image-based motion estimation Improving image-based motion estimation Improving feature tracking Reacquisition

10. Outline Image-based motion estimation refresher difficulties Improving image-based motion estimation Improving feature tracking Reacquisition

11. Image-based motion estimation: refresher (1) A two-step process is typical… First, sparse feature tracking: • Inputs:raw images • Outputs: projections

12. Image-based motion estimation: refresher (2)

13. Image-based motion estimation: refresher (3) Second, estimation: • Input: • Outputs: • projections from tracker • 6 DOF camera position at the time of each image • 3D position of each tracked point

14. Image-based motion estimation: refresher (4)

15. Image-based motion estimation: refresher (5) Algorithms exist For tracking: • Lucas-Kanade (Lucas and Kanade, 1981)

16. Image-based motion estimation: refresher (6) For estimation: • SVD-based factorization (Tomasi and Kanade, 1992) • bundle adjustment (various, 1950’s) • Kalman filtering (Broida and Chellappa, 1990) • variable state dimension filter (McLauchlan, 1996)

17. Image-based motion estimation: difficulties (1) So, the problem is solved?

18. Image-based motion estimation: difficulties (2) • If so, where are the automatic systems for estimating the motion of: • in unknown environments? • from images in unknown environments?

19. Image-based motion estimation: difficulties (3) …and for automatically modeling • rooms • buildings • cities from a handheld camera?

20. Image-based motion estimation: difficulties (4) Estimation step can be very sensitive to: • incorrect or insufficient image feature tracking • camera modeling and calibration errors • outlier detection thresholds • sequences with degenerate camera motions

21. Image-based motion estimation: difficulties (5) …and for recursive methods in particular: • poor prior assumptions on the motion • poor approximations in state error modeling

22. Image-based motion estimation: difficulties (6) • 151 images, 23 points

23. Image-based motion estimation: difficulties (7)

24. Image-based motion estimation: difficulties (8) For long-term motion estimation, these errors accumulate

25. Outline Image-based motion estimation Improving image-based motion estimation overview image and inertial measurements Improving feature tracking Reacquisition

26. Improving image-based motion estimation: overview

27. Improving image-based motion estimation: overview

28. Improving image-based motion estimation: image and inertial (1) Image and inertial measurements are highly complimentary Inertial measurements can: • resolve the ambiguities in image-only estimates • establish the global scale

29. Improving image-based motion estimation: image and inertial (2) Images measurements can: • reduce the drift in integrating inertial measurements • distinguish between rotation, gravity, acceleration, bias, noise in accelerometer readings

30. Improving image-based motion estimation: image and inertial (3)

31. Improving image-based motion estimation: image and inertial (4)

32. Improving image-based motion estimation: image and inertial (5) • Other examples: • global scale typically within 5% • better convergence than image-only estimation

33. Improving image-based motion estimation: image and inertial (6) Many more details in: Dennis Strelow and Sanjiv Singh. Motion estimation from image and inertial measurements. IJRR, September 2004.

34. Outline Image-based motion estimation Improving image-based motion estimation Improving feature tracking Lucas-Kanade and real sequences The “smalls” tracker Reacquisition

35. Improving feature tracking: Lucas-Kanade and real sequences (1) • Lucas-Kanade is the “go to” sparse feature tracker: • iterative minimization of the intensity matching error function • applied at several image resolutions to handle large motions • features extracted based on image texture • feature death based on iteration convergence and correlation error

36. Improving feature tracking: Lucas-Kanade and real sequences (2) • Advantages: • fast • subpixel resolution • can handle some large motions well • uses general minimization, so easily extendible

37. Improving feature tracking: Lucas-Kanade and real sequences (3) 0.1 average pixel reprojection error!

38. Improving feature tracking: Lucas-Kanade and real sequences (4) • But, Lucas-Kanade has some flaws: • does not exploit the rigid scene • poor heuristics for: • large motions • extracting features • detecting feature mistracking

39. Improving feature tracking: Lucas-Kanade and real sequences (5)

40. Improving feature tracking: Lucas-Kanade and real sequences (6)

41. Improving feature tracking: Lucas-Kanade and real sequences (7)

42. Improving feature tracking: Lucas-Kanade and real sequences (7)

43. Improving feature tracking: Lucas-Kanade and real sequences (8)

44. Improving feature tracking: the “smalls” tracker (1) • smalls is a new sparse image feature tracker • designed to address these issues • i.e., designed for long-term motion estimation

45. Improving feature tracking: the “smalls” tracker (2) Leonard Smalls: tracker, lone biker of the apocalypse

46. Improving feature tracking: the “smalls” tracker (3) SIFT epipolar geometry features 1-D correlation matching along epipolar lines geometric mistracking detection feature death and birth to 6 DOF output estimation features

47. Improving feature tracking: the “smalls” tracker (4) • SIFT keypoints (Lowe, IJCV 2004): • image interest points • can be extracted despite of large changes in viewpoint • to subpixel accuracy • A keypoint’s feature vectors in two images usually match

48. Improving feature tracking: the “smalls” tracker (5) Epipolar geometry between adjacent images is determined using… SIFT epipolar geometry features • SIFT extraction and matching • two-frame bundle adjustment • RANSAC

49. Improving feature tracking: the “smalls” tracker (6) 1-D correlation matching along epipolar lines • initial search position from nearby SIFT matches • discrete SSD search (e.g.,  60 pixels) • 1-D Lucas-Kanade refines the match

50. Improving feature tracking: the “smalls” tracker (7) To check for mistracking, use only three-frame geometric consistency… geometric mistracking detection • …determined using: • three-frame bundle adjustment • RANSAC