Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics

1. Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics Class 3: Infinitesimal Motion CS329 Stanford University Amnon Shashua

2. Material We Will Cover Today • Infinitesimal Motion Model • Infinitesimal Planar Homography (8-parameter flow) • Factorization Principle for Motion/Structure Recovery • Direct Estimation

3. Infinitesimal Motion Model Rodriguez Formula:

4. Infinitesimal Motion Model

5. Reminder: Assume:

6. Infinitesimal Motion Model Let

7. Infinitesimal Motion Model

8. Infinitesimal Planar Motion (the 8-parameter flow)

9. Infinitesimal Planar Motion (the 8-parameter flow)

10. Infinitesimal Planar Motion (the 8-parameter flow) Note: unlike the discrete case, there is no scale factor

11. Reconstruction of Structure/Motion (factorization principle) Note: 2 interchanges 1 interchanges

12. Reconstruction of Structure/Motion (factorization principle)

13. Reconstruction of Structure/Motion (factorization principle) Let be the “flow” of point i at image j (image 0 is ref frame)

14. Reconstruction of Structure/Motion (factorization principle) Given W, find S,M (using SVD) Let for some Goal: find such that using the “structural” constraints on S

15. Reconstruction of Structure/Motion (factorization principle) Goal: find such that using the “structural” constraints on S Columns 1-3 of S are known, thus columns 1-3 of A can be determined. Columns 4-6 of A contain 18 unknowns: eliminate Z and one obtains 5 constraints

16. Reconstruction of Structure/Motion (factorization principle) Goal: find such that using the “structural” constraints on S Let because

17. Reconstruction of Structure/Motion (factorization principle) because Each point provides 5 constraints, thus we need 4 points and 7 views

18. Direct Estimation The grey values of images 1,2 Goal: find u,v per pixel

19. Direct Estimation Assume: We are assuming that (u,v) can be found by correlation principle (minimizing the sum of square differences).

20. Direct Estimation Taylor expansion:

21. Direct Estimation gradient of image 2 image 1 minus image 2

22. Direct Estimation “aperture problem”

23. Direct Estimation Estimating parametric flow: Every pixel contributes one linear equation for the 8 unknowns

24. Direct Estimation Estimating 3-frame Motion: Combine with:

25. Direct Estimation Let

26. Direct Estimation image 1 to image 2 image 1 to image 3 Each pixel contributes a linear equation to the 15 unknown parameters

27. Direct Estimation: Factorization Let be the “flow” of point i at image j (image 0 is ref frame)

28. Direct Estimation: Factorization

29. Direct Estimation: Factorization Recall:

30. Direct Estimation: Factorization

31. Direct Estimation: Factorization Rank=6 Rank=6 Enforcing rank=6 constraint on the measurement matrix removes errors in a least-squares sense.

32. Direct Estimation: Factorization Once U,V are recovered, one can solve for S,M as before.