Mobile Motion Tracking using Onboard Camera

1. Mobile Motion Tracking using Onboard Camera Supervisor: Prof. LYU, Rung Tsong Michael Prepared by: Lam Man Kit Wong Yuk Man

2. Agenda • Motivation & Objective • Previous Work • Improvement & New components • Sample Applications • Conclusion • Q&A

3. Motivation • Rapid increase in the use of camera-phone • Camera-phone can perform image processing tasks on the device itself • Enhance human-computer interaction by mobile phone

4. Objective • Implement real-time motion tracking on Symbian phone, without requiring additional hardware • Translational motion tracking • Rotational motion tracking • To develop a tracking engine for other mobile devices developers to use

5. Previous Work • A translational motion tracking engine • Blocking matching • SEA (Successive Elimination Algorithm) • PPNM • PDE (Partial Distortion Elimination) • Adaptive Window Search • Spiral Scan • Feature selection • A translational motion tracking engine

6. Block Matching Algorithm

7. Block Matching Algorithm • Evaluate the "goodness" of a match by Sum of Absolute Difference • Select the candidate block with the lowest error

8. Feature Selection • A good feature block: • High variance -> complex block • Not in a repeated pattern • If a good feature block is found, the performance of the motion tracking is improved • We have made improvements to our existing algorithm

9. squares Old Feature Selection • Divide the current frame into squares • Apply Feature Selection Algorithm to each squares • However, the best feature block may appear between two squares

10. New Feature Selection • More blocks are sampled • Sample 26x26 variances of blocks with size 15x15 in a 54x54 window • Search in spiral way and stop searching when selection criteria are matched • Prefer to find a feature block in the center • Prevent out of bound problem (block appear out of screen)

11. New Feature Selection • If a feature block is found on the edge, the following will happen: • The tracking algorithm will fail to find the exact match • Solution: • Apply additional constraint

12. New Feature selection • In order to find a feature block that is not on the edge, we apply a checking algorithm • Important difference in all directions => interest point

13. New Feature Selection • Now our newest feature selection algorithm becomes: • Find a block with a large variance -> which indicate the complexity of the block • Check if the block is on the edge or not by calculating the SAD between the candidate block and its 4 neighbors • If either one of the SAD is small -> the block is on edge -> reject • Else the block is not on edge

14. SSD is used instead of SAD • SSD = Sum of Squared Difference • Last term, SAD is used as matching criteria • SAD is commonly used because of its lower complexity (faster) • But we chose to use SSD finally

15. SSD is used instead of SAD • Experiment shows that SSD has better performance in accuracy and • Algorithm with SSD run as fast as that with SAD • Reason (of why as fast as SAD): • Elimination effects from SEA, PPNM & PDE become large when SSD is used Compensate the higher complexity of SSD

16. SSD is used instead of SAD • SEA and PPNM lower bound is adjusted using the inequality (12) proposed in the following paper J.J. Francis and G. de Jager. A Sum Square Error based Successive Elimination Algorithm for Block Motion Estimation (2002)

17. Experimental Result on Symbian phone • Frame rate • Maximum frame rate supported by Nokia 6600 is about 14 frames/sec • Running our algorithm (1/0.007 frames/sec) once only for each frame do not slow down the displaying frame rate, in other word, the frame rate can still be 14 frames/sec, our algorithm doesn’t lag the display

18. Rotation Tracking Engine • Observation and Motivation • As we rotate the phone, the object can still be tracked correctly (center of object roughly equals center of green box)

19. Rotation Tracking Engine • Our approach • If two blocks are tracked at the same time, angle of line connecting the two blocks reflect the tiling angle of the phone Two-block approach

20. Rotation Tracking Engine • Another approach • Track one block by simple motion tracking engine, then find which rotation gives the best match • Fail if tracking object is the same for different angle One-block approach

21. Rotation Tracking Engine • Modification of motion tracking algorithm to suit rotation tracking • Linear Adaptive method used in translation motion tracking is not used here because phone’s motion can be both linear motion and circular motion

22. Rotation Tracking Engine • What to predict? • Given the coordinates of the tracking blocks in the last 2 previous frames • Predict the coordinates of the blocks in the next frame Line L1 • A simplified mathematic problem (the following assumptions are approximately correct) • All three lines pass through the circle’s center • End points of the three lines lie on the circle Line L2 θ θ Line L3

23. Rotation Tracking Engine • Solution • Angle between L2 and L3 = θ • θ = Tan-1(slope(L2)) – Tan-1(slope(L3)) • Coordinates of the next tracking block (xL1, yL1) are calculated by multiplying the column matrix of coordinates of the previous block with a rotation matrix

24. Rotation Tracking Engine • Solution • The prediction of the position of the next tracking block should also take the translational movement into account • Horizontal displacement = Tx • Tx = ( xL21 +xL22– xL31 -xL32 )/2 • Vertical displacement = Ty • Ty = ( yL21 +yL22– yL31 -yL32 )/2 • Coordinates of the next tracking block (xL1, yL1) is calculated by

25. Rotation Tracking Engine • A simple drawing to show the detected rotation angle of the phone

26. Rotation Tracking Engine • Reducing error by using level • Apply threshold on output, increase/decrease one level only when change is large • To give less sensitive but more desirable output for game • e.g. skiing game: skier face only to 7 directions • Reduce difficulty, increase reliability • A small rotation or a small detection error will not increase/decrease one level

27. Conditions to be a good background for both engines • Objective • Max. performance measurement • Increase usability • Condition: • Feature selection can always find a good feature point • Within certain distance • No repeat pattern • Very distinct pattern • Pattern is nearly the same when rotated. E.g. Circle which is good for rotation detection

28. Virtual Mouse • An application that make full use of the translational motion detection engine • Remote control the mouse of PC by Symbian phone using motion tracking • Advantages: • It allows input in all directions • It provides high levels of control • Joystick can still be used as rough moving while camera input can be used as fine translation

29. Virtual Mouse • Server • In server side (Windows), the Bluetooth is configured to provide RFComm Service and server regards the Bluetooth transmission port as Comm. Port • Server receives message from that Comm. Port • Call function in MouseAction.h to make the mouse move and trigger mouse click event

30. Virtual Mouse • Client • Search for Bluetooth device nearby • Connect to the selected Bluetooth device • Every time motion tracking algorithm finishes, results are sent to server (12 times/sec) • Joystick and keypad can also be used to control mouse in PC (Multi-button mouse)

31. Car Racing Game • A “Car Racing Game” is developed using the motion tracking engine. • Game lags. It is because: • The engine takes time to find the motion vector • CPU speed of the Symbian phones are low • The game engine thread uses most of the CPU power while the thread for updating the screen cannot proceed • Solutions: • Double Buffering • Direct Screen Access • Single Thread

32. Frame 1 Computing …… Frame 2 Double Buffered Area • Double-buffering – two complete color buffers • One is displayed while the other is calculating next scene to be drawn, • When the drawing of the next frame is completed, the content of the back buffer is copied to the front screen Computing Frame 2 Front Screen Show Frame 2 Frame 2 ready Buffer 1 Buffer 2 copy

33. Direct Screen Access • In Symbian, traditional drawing requires the help of Window Server • Overhead in context switching • Lower speed • Access the screen directly in 3 ways: • Creating and using CfbsScreenDevice • Accessing screen memory directly. • Using CdirectScreenAccess. • Using DSA can bypass the Window Server • Get rid of context switching • Faster

34. Device Screen and Keypad Key Presses Update Display Window Server RWsSession RWsSession RWsSession Application 1 Application 1 Application 1 Traditional Graphic Programming

35. 1. Request to Window Server to work in Direct Screen Access mode Application Window Server 2. If successful, a region for drawing to is returned to the application. Direct Screen Access

36. Single Thread • Prevent context switching • No need to do synchronization

37. Car Racing Game • With the use of efficient graphic programming algorithms, the tracking engine will not affect the performance of the game • It shows that we can use the engine to develop other applications without performance degradation

38. Skiing Game • Sample program from the book “Developing Series 60 Applications: A Guide for Symbian Os C++ Developers” • Based on this game, we plug in our rotation tracking engine into it • The game becomes more interactive • Rotate the phone to control the angle of the skier

39. Skiing Game

40. Skiing Game • The process of using the engine is very simple • Create an instance of the tracking engine • Call the function of the engine to find the motion vector • Use the motion vector for the game logic

41. Skiing Game

42. Demo

43. Experimental Result on Symbian phone • Testing Environment • Platform • Symbian Phone Nokia 6600 • Algorithm • Our final (hybrid-type) Algorithm (Block matching algorithm featured with Adaptive Window, Spiral Scan, SEA, PPNM and PDE method) • Algorithm parameter • Block size = 17 x 17 (pixels) • Search window size = 16 x 16 (pixels) • Number of block to track in each run of algorithm = 1 block ONLY

44. Experimental Result on Symbian phone • Testing Result (avg time to run) • Final algorithm • 7ms • Final algorithm without adaptive window method and SEA, PPNM method • 22ms • Full Exhaustive Search algorithm (the most simplest one) • 55ms

45. Q & A