Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context

1. Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context Felix Loew Master Thesis (TU München) 21.01.2005

2. Outline • Context-Awareness and Augmented Reality • Vision Based Tracking: Natural Feature Tracking • A „Magic Book“ User Study • Software Architecture (DWARF ) • Expected results and conclusions

3. Context-Awareness and Augmented Reality • A system has to adopt to the user´s behaviour! • A system has to learn how it is used! • New intuitive methods of interaction are required! • The system has to disappear (Marc Weiser)!

4. The „Magic Book“ in this context • User has to „learn“ how it is used • Very error-prone to movement • Movement of the handheld device is not considered in the tracking routine • Vision based tracking loses orientation • User has to reinitialize the tracking (looking on the marker again) • Immersion decreases!

5. Inertial Tracker: Intersense, Inertial Cube3 (~120FPS) Vision Based Tracking: NFT (~20 FPS) My Idea: Enhancing the Magic Book Movement

6. Outline • Context-Awareness and Augmented Reality • Vision Based Tracking: Natural Feature Tracking • A „Magic Book“ User Study • Software Architecture (DWARF ) • Summary, Conclusions & Limitations

7. Basics: Natural Feature Tracking Frame 1 Frame 2 Frame 4 Frame 3

8. Basics: Natural Feature Tracking • NFTARToolkit (Kato): 2D planar scenes • Estimation of feature position in next frame • It is not realistic that no movement occurs • Linear estimation • Kalman Filter • Within a window the “best match” with the template is searched (Template Matching) • For every pixel within window the similarity with the template is calculated (Normalized Cross Correlation)

9. Template Matching

10. Normalized Cross Correlation • Calculates similarity between template and area around search point • values [-1..1] • Best match is considered as feature point in current frame (threshold ~0.7)! • Computational expensive! • O (templateSize2 * searchSize2)

11. My approach • Consider movement as well! • Use additional tracking device (Inertial Tracker / 3DOF) • Naive approach: much movement large search window low movement small search window • Dynamic configuration during runtime

12. My approach What is much movement? Which movements actually occur? How does a user use such an application? USER STUDY ON MAGIC BOOK

13. Outline • Context-Awareness and Augmented Reality • Vision Based Tracking: Natural Feature Tracking • A „Magic Book“ User Study • Software Architecture (DWARF ) • Expected results and conclusions

14. User Study Setup • Goals • Detecting specific movement patterns • Derive Movement model • Estimation of a correlation between changes in orientation and feature points coordinates Orientation (Handheld, 3D) Position (FP-Coords, 2D)

15. Tracking Orientation and Position (6DOF) Magnetic Tracker: Flock of Birds Bird1: Magic Book Bird2: Handheld device Calibration to Magic Book coordinate system (Bird1) User Study Setup

16. Logging • Magnetic Tracker • Timestamp • Position P = (px,py,pz) • Orientation (quaternion) Q = (v,s) • NFT • Timestamp • 4 feature points per frame • Id • Screen-Position S = (sx,sy)

17. Logging • Magnetic Tracker pics\2_20_1.Case2.Flock.xls • NFT pics\2_20_1.Case2.Coords.xls

18. Comparable Tasks (Navigation Tasks) User has to answer certain questions (Overview, Focus and Search Tasks) „What is the girls hair color?“ „How many people do you see?“ Free Tasks User can use Magic Book freely User Tasks

19. Evaluation • Ideas for evaluation (Correlation) • Which angular offset (FOB) causes which coordinate offset (NFT)? • Which angular velocities occur? • Which rotations occur? • Which changes of position occur? • Is there a common movement profile or are there serious differences between different users?

20. Evaluation • Find mapping between: • Configure Software Architecture with this information search window orientation

21. Outline • Context-Awareness and Augmented Reality • Vision Based Tracking: Natural Feature Tracking • A „Magic Book“ User Study • Software Architecture (DWARF) • Expected results and conclusions

22. Software Architecture (DWARF) NFT ARToolkit

23. Dynamic Configuration

24. Outline • Context-Awareness and Augmented Reality • Vision Based Tracking: Natural Feature Tracking • A „Magic Book“ User Study • Software Architecture (DWARF) • Expected results and conclusions

25. Conclusions & Summary • Hybrid Tracking approach to consider context information as well • Movement context • 2DO: What about user context? • User Study on Magic Book (work in progress) • Expected conclusions on usage of Magic Book • Find correlation between movement and NFT properties • Process a configuration out of the study • Derive Movement model of the Magic Book • Limitation: Very task specific (features are not moving, only 2D)! • Software Architecture

26. Limitations & 2DO´s • Heaps of work • I won’t cover everything I want to do • Other “sensor-fusion” approaches might be more suited (Kalman-Filtering) • Evaluation of User Study will be hard • Is there an abstraction for NFT based applications?

27. Thanks • I would be glad if some of you want to join the study!