Accelerometer-Based Gestural Control for Browser Applications

1. Feature Extraction Spring Semester, 2010

2. Accelerometer Based Gestural Control of Browser Applications M. Kauppila et al., In Proc. of Int. Workshop on Real Field Identification, UCS 2007, pp. 2-17, 2007.

3. Outline • Motivation • Previous Work • H/W Architecture • Recognizer • Experiments • Discussion

4. Motivation • Large screen interface • Train, bus stations, marketing places, and other public places • Provide different kinds of interactions between the users and the displays

5. Previous Work • DBN, SVM • Samsung AIT, “Two-stage Recognition of Raw Acceleration Signals for 3-D Gesture-Understanding Cell Phones,” 2006. • Classify {0~9, O, X} by using DBN • SVM is used for the confusing pair of (6, O)

6. Communication Architecture • Scenario • Data flow

7. Communication Architecture • Accelerometers • Developed at our university • Send data at 50Hz • UPnP virtual sensor • Use a simple asynchronous Java API: decouple the recognizer and its clients

8. Browser Control • General browser / photo album app.

9. Recognizer • Segmentation • Preprocessing and normalization • Classification

10. Recognizer Segmentation • Feature vector for segmentation • Continuous acceleration signal: • Discretized acceleration signal: • Approximated derivative: • Approximated velocity:  • Two-state (non-gestural / gestural) HMM

11. Recognizer Segmentation Example

12. Recognizer Preprocessing • Sensor model • Dynamic component (gesture): ad(t) • Static component (gravity): as=(0, 0, g)T • Measured acceleration: • R: orthogonal matrix (Describing the orientation of the sensor) • Gravity estimation • Ras: Mean of the measured acceleration

13. Recognizer Preprocessing • Tilt Compensation • Let u = v/|v|, where v is the axis of rotation • Remember the measured acceleration • Finally,

14. Recognizer Normalization • Power normalization • Frobenius normalization • Tempo normalization • Rescale the gesture tempo so that all gestures have 30 samples • Downscaling: Box filtering • Upscaling: linear interpolation

15. Recognizer Classification • Training set • A single person, 16 samples per gesture • Recognizer • 12-state hidden Markov model per gesture • Choose the gesture class corresponding to the HMM with the highest score

16. Experiments • 11 Subjects • Confusion matrix

17. Experiments User Study • Three stages + a questionnaire (feedback) • Blind stage • Freely use the system without any prior training • Task stage • Solve a specific browsing task after training • Photo album stage

18. Experiments Subject Feedback • Five questions • Free-worded feedback • Stress of hands • Unintuitivity and learning overhead

19. Discussion • False positives still pose a problem • Rejection mechanism is needed • Recoiling problem • Avoiding the use of overly simplistic gestures

20. Human Activity Recognition with User-Free Accelerometers in the Sensor Networks S. Wang, et al., Int. Conf. Neural Networks and Brain, 2005. pp. 1212-1217, 2005.

21. Outline • Motivation • Feature Extraction • Classification • Experiments • Summary

22. Motivation • Human’s activities can be represented from three aspects • Movements of human bodies • Movements of the objects associated with the activities • Person-object interaction • Wearing the sensors is uncomfortable for users

23. Feature Extraction • Using a sliding window with 50% overlap • 19 features • Six features from each of the three axes • Acceleration, mean, standard deviation (stability), energy (data periodicity), frequency-domain entropy, correlation  normalized into [-1.1] • One feature represents vibration of the sensor (|ax2+ay2+az2-g2|)

24. Classification • Recognition algorithms • C4.5, MLP, SVM • Three types of tests • Self-consistency test: Training set = test set • Cross-validated test • Leave-one-subject-out validation

25. Experiments • System setup • Accelerometer: KXP74 (32Hz, -2g~+2g) • Fixed to the rear of the telephone receiver, base of the cup, and on the top of the pen • SVM-based feature selection

26. Experiments Data collection • Three activities was performed by four subjects • Drinking, phoning, and writing • Positive actions + negative actions • Positive: Write on the table or on the blackboard • Negative: Rotate the pen with fingers, … • Each lasted 5 minutes

27. Experiments Results • Self-consistency test: Accuracies > 95% • Cross-validated test • Leave-one-subject-out validation test

28. Experiments Discussion on Feature Selection • Attribute sequence • Drinking • Phoning • Writing • A: Acceleration, E: Mean, S: Stdev., G: Energy, P: Entropy, C: Correlation, Delta: Vibration

29. Summary • Accelerometer based gestural control of browser applications • Segmentation feature • Acceleration, derivative, and velocity • Tilt compensation • Human activity recognition with user-free accelerometers in the sensor networks • Feature extraction and selection