530 likes | 832 Views
Mobile Motion Tracking using Onboard Camera Supervisor Prof. LYU, Rung Tsong Michael Presented by Lam Man Kit, Wong Yuk Man Outline 1. Motivation & Objective 2. Motion Input 3. Previous Work 4. Improvement & New Components 5. Applications 6. Experimental Result 7. Demo Motivation
E N D
Mobile Motion Tracking using Onboard Camera Supervisor Prof. LYU, Rung Tsong Michael Presented by Lam Man Kit, Wong Yuk Man
Outline 1. Motivation & Objective 2. Motion Input 3. Previous Work 4. Improvement & New Components 5. Applications 6. Experimental Result 7. Demo
Motivation • Rapid increase in the use of camera-phone. • Camera-phones’ processing power is increasing • Symbian OS makes programming on mobile phone possible
Objective • Develop motion input method on Symbian phone without requiring additional hardware • This acts as an innovative input method for applications such as: • Camera mouse to control the cursor • New input method for interactive games • Gesture input
Why Motion Input? • Motion input is quick and convenient • Common way to play game on mobile phone is to use joystick or keypad • Limited freedom of movement • Lack of maneuverability • Difficult to press the small-sized keypad • Motion input doesn’t have these problems!
Motion Tracking • Motion tracking is a process to find the motion vector of the current frame from the reference frame • We use block matching algorithm to find the motion vector
Block Matching Algorithm Comparison of blocks New Position The motion vector which corresponds to the best match (dx,dy)
Block Matching Algorithm • Evaluate the "goodness" of a match by • Sum of Absolute Difference • Select the candidate block with the lowest error
Motion Tracking • Block matching algorithm • Traditional block matching algorithm is slow • Our solution: Hybrid-type fast block matching algorithm • That is “Adaptive Spiral SEA PPNM SSD” algorithm • Adaptive Search Window • Spiral Scan Method • Successive Elimination Algorithm • Progressive Partial Norm Matching • Sum of Squared Difference
Previous Work • In the previous semester, • Developed a testing platform in both Window and Symbian OS • Developed our own block matching algorithm and feature selection algorithm • Built a real-time translational motion tracking engine • Developed some applications using the engine
Improvement on Feature Selection • A good feature block: • High variance -> complex block • Not in a repeated pattern • If a good feature block is found, the performance of the motion tracking will be improved • We have made improvements to our existing algorithm
squares Old Feature Selection • Divide the current frame into squares • Apply Feature Selection Algorithm to each square • However, the best feature block may appear between two squares Best feature block
New Feature Selection • More blocks are sampled • Sample 26x26 squares with size 15x15 in a 54x54 window • Old feature selection only samples 12x12 squares • Search in spiral way and stop searching when selection criteria are matched • Prefer to find a feature block in the center • Prevent out of bound problem (block appear out of screen)
New Feature Selection • If a feature block is found on the edge, the following will happen: • The tracking algorithm will fail to find the exact match • Solution: • Apply additional constraint
New Feature selection • In order to find a feature block that is not on the edge, we apply a checking algorithm • Important difference in all directions -> interest point
New Feature Selection • Now our newest feature selection algorithm becomes: • Find a block with a large variance which indicates the complexity of the block • Check if the block is on the edge or not by calculating the SAD between the candidate block and its 4 neighbors • If either one of the SAD is small -> the block is on edge -> reject • Else the block is not on edge
Rotation Tracking Engine • Observation and Motivation • As we rotate the phone, the object can still be tracked correctly (center of object always roughly equals center of green box)
Rotation Tracking Engine • First approach • Track one block by translational motion tracking engine, rotate the current block and then find which rotation of the current block best match with the previous block • Fail if tracking object is the same for different angle One-block approach
Rotation Tracking Engine • Our approach • If two blocks are tracked simultaneously, angle of line connecting the two blocks indicates the rotation angle of the phone • Use only translational motion tracking Two-block approach
Rotation Tracking Engine • Modification of adaptive search window method to suit rotation tracking • Phone’s motion can be both linear motion and circular motion • Linear Adaptive method used in translational motion tracking is not used in this engine • Why need adaptive search window method? • To increase performance based on the fact that during block matching, reaching the best candidate block earlier increases the elimination effect of SEA, PPNM, PDE
Rotation Tracking Engine • What to predict? • Predict the coordinates of the blocks in the next frame • Given the coordinates of the tracking blocks in the current frame and the previous frames • When to predict? • After each run of rotation tracking algorithm Line L1 Line L2 θ θ Line L3
Rotation Tracking Engine • Solution • Angle between L2 and L3 = θ • θ = Tan-1(slope(L2)) – Tan-1(slope(L3)) • Coordinates of the next tracking block (xL1, yL1) are calculated by multiplying the column matrix of coordinates of the previous block with a rotation matrix
Rotation Tracking Engine • Solution • The prediction of the positions of the next tracking blocks should also take the translational movement into account • Horizontal displacement = Tx • Tx = ( xL21 +xL22– xL31 -xL32 )/2 • Vertical displacement = Ty • Ty = ( yL21 +yL22– yL31 -yL32 )/2 • Coordinates of the next tracking block (xL1, yL1) is calculated by
Level 0 - 40° 0° 40° Level 1 Level -1 80° Level 2 120° Level 3 Rotation Tracking Engine • Reducing error by using level • Increase/decrease one level only when change is large • To give less sensitive but more desirable output for game • e.g. skiing game: skier face only to 7 directions • Reduce difficulty of the game, increase reliability of the engine
Rotation Tracking Engine • Some screenshot showing how rotation tracking is carried out and how the result is displayed
Rotation Tracking Engine A short video demo
Tailor-made background for both engines • Objective • Max. performance measurement • Increase usability • Condition • Feature selection can always find a good feature point • Within certain distance • No repeat pattern • Very distinct pattern • Pattern is nearly the same when rotated. E.g. Circle is good for rotation detection
Virtual Mouse • An application that make full use of the translational motion tracking engine • Remote control the mouse of PC by mobile phone using motion as input • Advantages: • It allows input in many directions and provides high levels of control • Many buttons left for other purposes • E.g. Joystick can be used as course adjustment of mouse cursor • E.g. Some buttons can be used as shortcuts of applications
Virtual Mouse • Server • Configure Bluetooth device to provide RFComm Service and regards the Bluetooth transmission port as Comm. Port • Server receives message from that Comm. Port • Call function in MouseAction.h to make the mouse move and trigger mouse click event
Virtual Mouse • Client • Search for Bluetooth device nearby • Connect to the selected Bluetooth device • Every time motion tracking algorithm finishes, results are sent to server (14 times/sec) • Joystick and keypad can also be used to control mouse in PC (Multi-button mouse)
Car Racing Game • A “Car Racing Game” is developed using the motion tracking engine. • Game lags. It is because: • The engine takes time to find the motion vector • CPU speed of the Symbian phones is low • Solutions: • Double Buffering • Direct Screen Access
Skiing Game • We have developed a skiing game that makes use of the rotation tracking engine • Rotate the phone to control the angle of the skier • After using the rotation tracking engine: • The game becomes more interactive • The game has more degree of freedom for rotating the skier
Skiing Game • The process of using the rotation tracking engine is very simple • Create an instance of the tracking engine • Call the function of the engine to find the motion vector • Use the motion vector for the game logic
Experimental Result on Symbian phone • Testing Environment • Platform • Symbian Phone Nokia 6600 • Algorithm • Our final (hybrid-type) algorithm (Block matching algorithm featured with Adaptive Window, Spiral Scan, SEA, PPNM and PDE method) • Algorithm parameter • Block size = 17 x 17 (pixels) • Search window size = 16 x 16 (pixels) • Number of block to track in each run of algorithm = 1 block ONLY
Experimental Result on Symbian phone • Testing Result (avg time to run) • Final algorithm • 7ms • Partial Spiral algorithm • 22ms • Full Exhaustive Search algorithm (the most simplest one) • 55ms
Conclusion • We have developed: • The real-time motion tracking algorithm and feature selection method • The translational and rotational motion tracking engine on Symbian • A series of applications/games that make full use of our engines
Demo • Virtual Mouse
Demo • Act as an innovative input device for games
Stereo • If we want to track the movement in 3D, we have to: • Use two cameras • Know the focal length of the cameras • Know the distance between two cameras • Depth can be recovered with two images and triangulation
Stereo • Therefore, stereo has two steps • Finding matching points in the images • Then using them to compute depth. P P’1 P’2 O2 O1