Mobile Motion Tracking using Onboard Camera

1. Mobile Motion Tracking using Onboard Camera Supervisor Prof. LYU, Rung Tsong Michael Presented by Lam Man Kit, Wong Yuk Man

2. Outline 1. Motivation & Objective 2. Motion Input 3. Previous Work 4. Improvement & New Components 5. Applications 6. Experimental Result 7. Demo

3. Motivation • Rapid increase in the use of camera-phone. • Camera-phones’ processing power is increasing • Symbian OS makes programming on mobile phone possible

4. Objective • Develop motion input method on Symbian phone without requiring additional hardware • This acts as an innovative input method for applications such as: • Camera mouse to control the cursor • New input method for interactive games • Gesture input

5. What is Motion Input? A

6. Why Motion Input? • Motion input is quick and convenient • Common way to play game on mobile phone is to use joystick or keypad • Limited freedom of movement • Lack of maneuverability • Difficult to press the small-sized keypad • Motion input doesn’t have these problems!

7. Motion Tracking • Motion tracking is a process to find the motion vector of the current frame from the reference frame • We use block matching algorithm to find the motion vector

8. Block Matching Algorithm Comparison of blocks New Position The motion vector which corresponds to the best match (dx,dy)

9. Block Matching Algorithm • Evaluate the "goodness" of a match by • Sum of Absolute Difference • Select the candidate block with the lowest error

10. Motion Tracking • Block matching algorithm • Traditional block matching algorithm is slow • Our solution: Hybrid-type fast block matching algorithm • That is “Adaptive Spiral SEA PPNM SSD” algorithm • Adaptive Search Window • Spiral Scan Method • Successive Elimination Algorithm • Progressive Partial Norm Matching • Sum of Squared Difference

11. Previous Work • In the previous semester, • Developed a testing platform in both Window and Symbian OS • Developed our own block matching algorithm and feature selection algorithm • Built a real-time translational motion tracking engine • Developed some applications using the engine

12. Improvement on Feature Selection • A good feature block: • High variance -> complex block • Not in a repeated pattern • If a good feature block is found, the performance of the motion tracking will be improved • We have made improvements to our existing algorithm

13. squares Old Feature Selection • Divide the current frame into squares • Apply Feature Selection Algorithm to each square • However, the best feature block may appear between two squares Best feature block

14. New Feature Selection • More blocks are sampled • Sample 26x26 squares with size 15x15 in a 54x54 window • Old feature selection only samples 12x12 squares • Search in spiral way and stop searching when selection criteria are matched • Prefer to find a feature block in the center • Prevent out of bound problem (block appear out of screen)

15. New Feature Selection • If a feature block is found on the edge, the following will happen: • The tracking algorithm will fail to find the exact match • Solution: • Apply additional constraint

16. New Feature selection • In order to find a feature block that is not on the edge, we apply a checking algorithm • Important difference in all directions -> interest point

17. New Feature Selection • Now our newest feature selection algorithm becomes: • Find a block with a large variance which indicates the complexity of the block • Check if the block is on the edge or not by calculating the SAD between the candidate block and its 4 neighbors • If either one of the SAD is small -> the block is on edge -> reject • Else the block is not on edge

18. Rotation Tracking Engine • Observation and Motivation • As we rotate the phone, the object can still be tracked correctly (center of object always roughly equals center of green box)

19. Rotation Tracking Engine • First approach • Track one block by translational motion tracking engine, rotate the current block and then find which rotation of the current block best match with the previous block • Fail if tracking object is the same for different angle One-block approach

20. Rotation Tracking Engine • Our approach • If two blocks are tracked simultaneously, angle of line connecting the two blocks indicates the rotation angle of the phone • Use only translational motion tracking Two-block approach

21. Rotation Tracking Engine • Modification of adaptive search window method to suit rotation tracking • Phone’s motion can be both linear motion and circular motion • Linear Adaptive method used in translational motion tracking is not used in this engine • Why need adaptive search window method? • To increase performance based on the fact that during block matching, reaching the best candidate block earlier increases the elimination effect of SEA, PPNM, PDE

22. Rotation Tracking Engine • What to predict? • Predict the coordinates of the blocks in the next frame • Given the coordinates of the tracking blocks in the current frame and the previous frames • When to predict? • After each run of rotation tracking algorithm Line L1 Line L2 θ θ Line L3

23. Rotation Tracking Engine • Solution • Angle between L2 and L3 = θ • θ = Tan-1(slope(L2)) – Tan-1(slope(L3)) • Coordinates of the next tracking block (xL1, yL1) are calculated by multiplying the column matrix of coordinates of the previous block with a rotation matrix

24. Rotation Tracking Engine • Solution • The prediction of the positions of the next tracking blocks should also take the translational movement into account • Horizontal displacement = Tx • Tx = ( xL21 +xL22– xL31 -xL32 )/2 • Vertical displacement = Ty • Ty = ( yL21 +yL22– yL31 -yL32 )/2 • Coordinates of the next tracking block (xL1, yL1) is calculated by

25. Level 0 - 40° 0° 40° Level 1 Level -1 80° Level 2 120° Level 3 Rotation Tracking Engine • Reducing error by using level • Increase/decrease one level only when change is large • To give less sensitive but more desirable output for game • e.g. skiing game: skier face only to 7 directions • Reduce difficulty of the game, increase reliability of the engine

26. Rotation Tracking Engine • Some screenshot showing how rotation tracking is carried out and how the result is displayed

27. Rotation Tracking Engine A short video demo

28. Tailor-made background for both engines • Objective • Max. performance measurement • Increase usability • Condition • Feature selection can always find a good feature point • Within certain distance • No repeat pattern • Very distinct pattern • Pattern is nearly the same when rotated. E.g. Circle is good for rotation detection

29. Virtual Mouse • An application that make full use of the translational motion tracking engine • Remote control the mouse of PC by mobile phone using motion as input • Advantages: • It allows input in many directions and provides high levels of control • Many buttons left for other purposes • E.g. Joystick can be used as course adjustment of mouse cursor • E.g. Some buttons can be used as shortcuts of applications

30. Virtual Mouse

31. Virtual Mouse • Server • Configure Bluetooth device to provide RFComm Service and regards the Bluetooth transmission port as Comm. Port • Server receives message from that Comm. Port • Call function in MouseAction.h to make the mouse move and trigger mouse click event

32. Virtual Mouse • Client • Search for Bluetooth device nearby • Connect to the selected Bluetooth device • Every time motion tracking algorithm finishes, results are sent to server (14 times/sec) • Joystick and keypad can also be used to control mouse in PC (Multi-button mouse)

33. Car Racing Game • A “Car Racing Game” is developed using the motion tracking engine. • Game lags. It is because: • The engine takes time to find the motion vector • CPU speed of the Symbian phones is low • Solutions: • Double Buffering • Direct Screen Access

34. Skiing Game • We have developed a skiing game that makes use of the rotation tracking engine • Rotate the phone to control the angle of the skier • After using the rotation tracking engine: • The game becomes more interactive • The game has more degree of freedom for rotating the skier

35. Skiing Game

36. Skiing Game • The process of using the rotation tracking engine is very simple • Create an instance of the tracking engine • Call the function of the engine to find the motion vector • Use the motion vector for the game logic

37. Skiing Game

38. Experimental Result on Symbian phone • Testing Environment • Platform • Symbian Phone Nokia 6600 • Algorithm • Our final (hybrid-type) algorithm (Block matching algorithm featured with Adaptive Window, Spiral Scan, SEA, PPNM and PDE method) • Algorithm parameter • Block size         = 17 x 17 (pixels) • Search window size     = 16 x 16 (pixels) • Number of block to track in each run of algorithm = 1 block ONLY

39. Result

40. Experimental Result on Symbian phone • Testing Result (avg time to run) • Final algorithm • 7ms • Partial Spiral algorithm • 22ms • Full Exhaustive Search algorithm (the most simplest one) • 55ms

41. Result

42. Conclusion • We have developed: • The real-time motion tracking algorithm and feature selection method • The translational and rotational motion tracking engine on Symbian • A series of applications/games that make full use of our engines

43. Demo • Virtual Mouse

44. Demo • Act as an innovative input device for games

45. Q & A

46. Thank You !!

47. Experimental Result1 - Backgrounds

48. Experimental Result2 - Backgrounds

49. Stereo • If we want to track the movement in 3D, we have to: • Use two cameras • Know the focal length of the cameras • Know the distance between two cameras • Depth can be recovered with two images and triangulation

50. Stereo • Therefore, stereo has two steps • Finding matching points in the images • Then using them to compute depth. P P’1 P’2 O2 O1