Mobile Phone based Inference Models using People-centric Features Nicholas D. Lane

1. Mobile Phone based Inference Models using People-centric Features Nicholas D. Lane Approach: Use features that leverage the ability of people to synthesize complex multivariate data Example: Spoken Words as features for Activity Recognition. Problem: Inferences about society and where we live are challenging with mobile phones. • 10^9 mobile phones are in daily use but with limited sensing capabilities (e.g., localization accelerometer, microphone). • Important inferences are difficult based on these sensors (e.g., What are people doing? Are they sick? Are they safe?). words selection Can I have a coffee? mobility patterns people and the environment Here is a coffee. non-verbal sounds (sneeze) Thanks for my coffee. behaviour Exploratory Experiment Recognition Process Audio Signals Hypothesis: Even when recognizing only a fraction of ambient spoken words it is possible to perform complex forms of activity recognition using only a simple bag-of-words model. MFCC feature vectors from audio frames LBG-based vector quantization Methodology: Build proof-of-concept iPhone-based prototype. Capture 19 hours of audio while doing different activities over 2 weeks. Isolated word based discrete HMMs Results Collection of Words Stemming & Stop Word Removal • With 17% of words recognized and using word only features mean activity recognition accuracy was 71%. coffee Activity class based bayesian “bag-of-word” models fast food • Recognizes different instances of classes (e.g., fast food) and does not confuse these with similar classes (e.g., restaurants). Activities Future Work • Evaluate other examples of People-centric features particularly those found in other modalities and across other time scales • Develop models that combine these examples with more conventional features. • Differentiates activity uses (e.g., coffee or book purchase) in the same physical space (e.g., bookstore). unknown class