Activity Recognition and Biometric Identification Using Cell Phone Accelerometers

1. Activity Recognition and Biometric IdentificationUsing Cell Phone Accelerometers WISDM Project Department of Computer & Info. Science Fordham University

2. We are Interested in WISDM • WISDM: WIreless Sensor Data Mining • Powerful portable wireless devices are becoming common and are filled with sensors • Smart phones: Android phones, iPhone • Music players: iPod Touch • Sensors on smart phones include: • Microphone, camera, light sensor, proximity sensor, temperature sensor, GPS, compass, accelerometer

3. WISDM Data Mining Problems • Completed initial stages of research on 2 tasks: • Activity Recognition • What is the user doing? • Biometric Identification • Who is the user? Is the user who they claim to be? • Future tasks • GPS mining: learn about user routes & interactions • Use cell phones as a sensor network to learn about the environment

4. Accelerometer-Based Activity Recognition • The Problem: use accelerometer data to determine a user’s activity • Activities include: • Walking and jogging • Sitting and standing • Ascending and descending stairs • More activities to be added in future work

5. Applications of Activity Recognition • Health Applications • Generate activity profile to monitor overall type and quantity of activity • Parents can use it to monitor their children • Can be used to monitor the elderly • Make the device context-sensitive • Cell phone sends all calls to voice mail when jogging • Adjust music based on the activity • Broadcast (Facebook) your every activity

6. Accelerometer-Based Biometric Identification • The Problem: Use accelerometer data to identify an individual user • Identity prediction: map a user to one of a set of predetermined users • Authentication: determine whether a user is who they claim to be

7. Applications of Biometric Identification • Security & theft prevention of mobile devices • Automatic personalization of mobile devices • For example, send all calls to voicemail when jogging • Identify user and load proper settings • General Security Applications • Should the user be in this location? • Can be used as a second level of security

8. Our WISDM Platform • Platform based on Android cell phones • Android is Google’s open source mobile computing OS • Easy to program, free, will have a large market share • Android phones now outselling iPhones • Unlike most other work on activity recognition: • No specialized equipment • Single device naturally placed on body (in pocket)

9. Our WISDM Platform • Current research was conducted off-line • Data was collected and later analyzed off-line • Now updating our platform to operate in real-time • In June we released real-time sensor data collection app to Android marketplace • Currently collects accelerometer and GPS data and transmits it to our server

10. Accelerometers • Included in most smart phones & other devices • All Android phones, iPhones, iPod Touches, etc. • Tri-axial accelerometers that measure 3 dimensions • Initially included for screen rotation and advanced game play

11. Examples of Raw Data • Next few slides show data for one user over a few seconds for various activities • Cell phone is in user’s pocket • Earth’s gravity is registered as acceleration • Acceleration values relative to axes of the device, not Earth • In theory we can correct this given that we can determine orientation of the device

12. Standing

13. Sitting

14. Walking

15. Jogging

16. Descending Stairs

17. Ascending Stairs

18. Data Collection Procedure • User’s move through a specific course • Perform various activities for specific times • Data collected using Android phones • Activities labeled using our Android app • Data collection procedure approved by Fordham Institutional Review Board (IRB) • Collected data from ~35 users (will increase)

19. Data Preprocessing • Convert time series data into examples so we can use standard classifiers (e.g., decision trees) • Use a 10 second example duration/window • 3 acceleration values every 50 ms (600 total values) • Generate 43 total features • Ave. acceleration each axis (3) • Standard deviation each axis (3) • Binned/histogram distribution for each axis (30) • Time between peaks (3), Ave resultant acceleration (1)

20. Activity Recognition Results

21. Activity Recognition Final Data Set

22. Data Mining Step • Utilized three WEKA learning methods • Decision Tree (J48) • Logistic Regression • Neural Network • Results reported using 10-fold cross validation

23. Summary Results

24. J48 Confusion Matrix

25. Activity Recognition Conclusions • Able to identify activities with good accuracy • Hard to differentiate between ascending and descending stairs. To limited degree also looks like walking. • Can accomplish this with a cell phone placed naturally in pocket • Accomplished with simple features and standard data mining methods

26. Biometric Identification Results

27. Biometric Identification Data Sets • We evaluated 6 data sets (4 activities) • Aggregate (all 6 activities without class labels) • Walking • Jogging • Ascending Stairs • Descending Stairs • Aggregate-Oracle (all 6 activities with class labels) • The unlabeled aggregate data set is most realistic

28. # of Examples per User/Activity

29. Accuracy for Person Identification (based on 10-Second Examples) J48: Decision Tree Learner Straw Man: Strategy of always predicting the most common user

30. Aggregate Data Set Confusion Matrix (Results for first 14 users only)

31. We Can Do Better: Majority Scheme We know which records come from the same cell phone user So predict the users identity based on the identity predicted most often

32. Results Using Majority Voting

33. Ratio of Records Correctly Classified to Most Successful Imposter

34. Authentication Results Positive authentication rate: % of test examples coming from a user that are correctly classified as belonging to that user Negative authentication rate: % of test examples from an imposter that are correctly identified as not belonging to the user

35. Biometric Identification Conclusions Very accurate models for person identification using data mining of accelerometer data Generally perfect performance for identification when using majority scheme Can get good biometric results without knowing the specific activity the user is performing

36. Related Work • At least a dozen papers on activity recognition using multiple sensors, mainly accelerometers • Typically studies only 10-20 users • Activity recognition also done via computer vision • Actigraphy uses devices to study movement • Used by psychologists to study sleep disorders, ADD • A few recent efforts use cell phones • Yang (2009) used Nokia N95 and 4 users • Brezmes (2009) used Nokia N95 with real-time recognition • One model per user (requires labeled data from each user)

37. Future Work • Add more activities and users • Add more sophisticated features • Try time-series based learning methods • Deploy higher level applications: activity profiler • Can be used to encourage healthy behaviors • Can benefit the young and the elderly

38. Future Work (cont.) • Generate results in real time • Activity recognition • Uses a universal model so no need to train per user • Send results to server and get response back and on Web • Alternatively do everything on the phone • Biometric Identification • Need a model per user so would need to train model • Not too hard, just collect unlabelled activity data

39. Future Work (cont.) • GPS Data Mining • Find spatio-temporal patterns from GPS data • Location based on time of day, day of week • Identify friends and people you spend time with • Identify things about the environment • Where are the pedestrian paths on campus? • How busy is the cafeteria and when? • Where do people congregate • …

40. For More Information • See the WISDM web site: • http://storm.cis.fordham.edu/~gweiss/wisdm/ • See the two published papers on activity recognition and biometric identification • Meetings usually Thursday at 6:30 pm • Talk to me • There are quite a few benefits to undergraduate research!

41. Thank You Questions?