CSCE 641 Computer Graphics: Image Registration

1. CSCE 641 Computer Graphics: Image Registration Jinxiang Chai

2. Review Image warping Image morphing

3. Image Warping Warping function - similarity, affine, projective etc Image warping - forward warping and two-pass 1D warping - backward warping Resampling filter - point sampling - bilinear filter - anisotropic filter x u Inverse y v forward T(u,v) S(x,y)

4. Image Morphing Point based image morphing Vector based image morphing

5. h h? Image Registration Image warping: given h and f, compute g g(x) = f(h(x)) g? f Image registration: given f and g, compute h g f

6. Why Image Registration? Lots of uses • Correct for camera jitter (stabilization) • Align images (mosaics) • View morphing • Special effects • Image based modeling/rendering • Etc. [Seitz 96]

7. Image Registration How do we align two images automatically? Two broad approaches: • Feature-based alignment • Find a few matching features in both images • compute alignment • Direct (pixel-based) alignment • Search for alignment where most pixels agree

8. Outline Image registration - feature-based approach - pixel-based approach

9. Readings Bergen et al.Hierarchical model-based motion estimation. ECCV’92, pp. 237–252. Shi, J. and Tomasi, C. (1994). Good features to track. In CVPR’94, pp. 593–600. Baker, S. and Matthews, I. (2004). Lucas-kanade 20 years on: A unifying framework. IJCV, 56(3), 221–255.

10. Outline Image registration - feature-based approach - pixel-based approach

11. Feature-based Alignment • Find a few important features (aka Interest Points) • Match them across two images • Compute image transformation function h

12. Feature-based Alignment • Find a few important features (aka Interest Points) • Match them across two images • Compute image transformation function h How to choose features • Choose only the points (“features”) that are salient, i.e. likely to be there in the other image • How to find these features?

13. Feature-based Alignment • Find a few important features (aka Interest Points) • Match them across two images • Compute image transformation function h How to choose features • Choose only the points (“features”) that are salient, i.e. likely to be there in the other image • How to find these features? • windows where has two large eigenvalues • Harris Corner detector

14. Feature Detection • Two images taken at the same place with different angles • Projective transformationH3X3

15. Feature Matching ? • Two images taken at the same place with different angles • Projective transformation H3X3

16. Feature Matching ? • Two images taken at the same place with different angles • Projective transformation H3X3 How do we match features across images? Any criterion?

17. Feature Matching ? • Two images taken at the same place with different angles • Projective transformation H3X3 How do we match features across images? Any criterion?

18. Feature Matching • Intensity/Color similarity • The intensity of pixels around the corresponding features should have similar intensity

19. Feature Matching • Intensity/Color similarity • The intensity of pixels around the corresponding features should have similar intensity • Cross-correlation, SSD

20. Feature Matching • Intensity/Color similarity • The intensity of pixels around the corresponding features should have similar intensity • Cross-correlation, SSD • Distance constraint • The displacement of features should be smaller than a given threshold

21. Feature-space Outlier Rejection bad Good

22. Feature-space Outlier Rejection Can we now compute H3X3 from the blue points?

23. Feature-space Outlier Rejection Can we now compute H3X3 from the blue points? • No! Still too many outliers…

24. Feature-space Outlier Rejection Can we now compute H3X3 from the blue points? • No! Still too many outliers… • What can we do?

25. Feature-space Outlier Rejection Can we now compute H3X3 from the blue points? • No! Still too many outliers… • What can we do? Robust estimation!

26. Robust Estimation: A Toy Example How to fit a line based on a set of 2D points?

27. Robust Estimation: A Toy Example How to fit a line based on a set of 2D points? RANSAC: an iterative method to estimate parameters of a mathematical model from a set of observed data which contains outliers

28. RANSAC: RANdom SAmple Consensus • Objective • Robust fit of model to data set S which contains outliers • Algorithm • Randomly select a sample of s data points from S and instantiate the model from this subset. • Determine the set of data points Si which are within a distance threshold t of the model. The set Si is the consensus set of samples and defines the inliers of S. • If the subset of Si is greater than some threshold T, re-estimate the model using all the points in Si and terminate • If the size of Si is less than T, select a new subset and repeat the above. • After N trials the largest consensus set Si is selected, and the model is re-estimated using all the points in the subset Si

29. RANSAC Repeat M times: • Sample minimal number of matches to estimate two view relation (affine, perspective, etc). • Calculate number of inliers or posterior likelihood for relation. • Choose relation to maximize number of inliers.

30. RANSAC Line Fitting Example Task: Estimate best line

31. RANSAC Line Fitting Example Sample two points

32. RANSAC Line Fitting Example Fit Line

33. RANSAC Line Fitting Example Total number of points within a threshold of line.

34. RANSAC Line Fitting Example Repeat, until get a good result

35. RANSAC Line Fitting Example Repeat, until get a good result

36. RANSAC Line Fitting Example Repeat, until get a good result

37. How Many Samples? Choose N so that, with probability p, at least one random sample is free from outliers. e.g. p=0.99

38. How Many Samples? Choose N so that, with probability p, at least one random sample is free from outliers. e.g. p=0.99 Affine transform

39. How Many Samples? Choose N so that, with probability p, at least one random sample is free from outliers. e.g. p=0.99 Projective transform

40. RANSAC for Estimating Projective Transformation RANSAC loop: • Select four feature pairs (at random) • Compute the transformation matrix H (exact) • Compute inliers where SSD(pi’, H pi) < ε • Keep largest set of inliers • Re-compute least-squares H estimate on all of the inliers

41. RANSAC

42. Feature-based Registration Works for small or large motion Model the motion within a patch or whole image using a parametric transformation model

43. Feature-based Registration Works for small or large motion Model the motion within a patch or whole image using a parametric transformation model How to deal with motions that cannot be described by a small number of parameters?

44. Outline Image registration - feature-based approach - pixel-based approach

45. Direct (pixel-based) Alignment : Optical flow Will start by estimating motion of each pixel separately Then will consider motion of entire image

46. Problem Definition: Optical Flow How to estimate pixel motion from image H to image I?

47. Problem Definition: Optical Flow How to estimate pixel motion from image H to image I? • Solve pixel correspondence problem • given a pixel in H, look for nearby pixels of the same color in I

48. Problem Definition: Optical Flow How to estimate pixel motion from image H to image I? • Solve pixel correspondence problem • given a pixel in H, look for nearby pixels of the same color in I Key assumptions • color constancy: a point in H looks the same in I • For grayscale images, this is brightness constancy • small motion: points do not move very far This is called the optical flow problem

49. Optical Flow Constraints Let’s look at these constraints more closely • brightness constancy: Q: what’s the equation?

50. Optical Flow Constraints Let’s look at these constraints more closely • brightness constancy: Q: what’s the equation? H(x,y) - I(x+u,v+y) = 0