Presenter 12-July-2014 KHU

1. Presenter 12-July-2014 KHU

2. Information Curation Layer

3. Low Level Context-awareness • Introduction • Motivation • Related Works • Architecture • Tools and Technologies • Development Timeline • Current Status

4. Introduction • Heterogeneous information source around people • Daily physical activities • Social interactions • Psychological states • Automatically collect and provide basic information to the system Physical Activities Social Interaction Emotional States

5. Information Introduction • Knowledge • Extract information directly from raw data • Play an important role to get useful knowledge from people • Daily habit • Behavior Physical Activities • Data Social Interaction Emotional States

6. Motivation • Collect, process different kind of data • Sensory data • Social data • Analyze and extract useful information at low-level • Physical activities • Social activities • Emotional states • Provide the interface to interact with higher level and database manager

7. Related Works • Activity Recognizer • [WiiRemote] The Wii Remote™ Plus controller is the heart of the motion gaming experience on your Wii console. • [Han2012] tried to overcome the limitation of accelerometer based activity recognition. Accelerometer is used to recognize walking, running, and stay, and audio, GPS and wifi are used to recognize bus and subway. • There lots of segmentation works such as graph-cut based segmentation by [Pourjam2013], and mean-shift algorithm by [Atefian2013] have been proposed for human body segmentation.

8. Low Level Context-awareness • Emotion Recognizer • [MITMindReader] MIT’s Mind Reader software can scan faces in a crowd to determine audience mood, a tool that may replace opinion polls and help public speakers tailor their words for maximum impact. • Various types of classifiers have been used for the task of speech emotion recognition such as HMM, GMM, SVM, etc. [Ayadi2011]. • Several emotion research works tried to separate the original complex multiple emotion classification problem by applying hierarchical approach with combination of different classifiers [Lee2011].

9. Architecture High Level Context-awareness Personal Information Low Level Context-awareness Emotion Recognizer SNS Interaction Analyzer Activity Recognizer Raw Data HDFS Data Access Interface

10. Architecture High Level Context-awareness Low Level Context-awareness Twitter Analyzer Activity Recognizer Emotion Recognizer System Data Wearable Sensor based AR Thesaurus Manager Classifying Attribute Thesaurus Morpheme Manager Feature Extraction Data Acquisition Training Models Analysis Sentiment Compilation Smartphone based AR Syntax Analyzer Positive, Negative Analyzer Decision Maker GPS Validation Morpheme Analyzer Feature Extraction Preprocessing Attribute-Emotion Mapping Module Attribute Extractor Emotion Extractor Audio based ER Video based ER Video based AR Decision Fusion Physiological sensor based ER Feature Extraction Non-Param Cumulative Sum Feature Extraction Classifying HMM Testing Classifying Auto Associate Neural Network Archive Listener • Synchronization Probability Computation Remote Control Request Module DMBS Connector Segmentation Data Acquisition Feature Extraction Preprocessing Face Detection • Statistical Feature • Extraction HMM Training Classification Tree Construction Activity Label (standing, sitting, running, …) Emotion Label (happy, angry, boredom, …) Sensory Data (Heart rate, Video, Audio) Personal Information (behavior, interest) Sensory Data (Acc, GPS, Video) Social Data (Twitter) HDFS Data Access Interface

11. Architecture • SNS Analyzer - Twitter • Take input from Twitter API in schema format • Analyze Twitter data in different contexts • Activity • Emotion • Behavior • Provide the output based on keyword High Level Context-awareness Low Level Context-awareness Twitter Analyzer Activity Recognizer Emotion Recognizer System Data Wearable Sensor based AR Thesaurus Manager Classifying Attribute Thesaurus Morpheme Manager Feature Extraction Data Acquisition Training Models Analysis Sentiment Compilation Smartphone based AR Syntax Analyzer Positive, Negative Analyzer Decision Maker GPS Validation Morpheme Analyzer Feature Extraction Preprocessing Attribute-Emotion Mapping Module Attribute Extractor Emotion Extractor Physiological sensor based ER Video based ER Audio based ER Video based AR Decision Fusion Feature Extraction Feature Extraction Non-Param Cumulative Sum Classifying HMM Testing Classifying Auto Associate Neural Network Archive Listener • Synchronization Probability Computation Remote Control Request Module DMBS Connector Segmentation Data Acquisition Feature Extraction Preprocessing Face Detection • Statistical Feature • Extraction HMM Training Classification Tree Construction Activity Label (standing, sitting, running, …) Emotion Label (happy, angry, boredom, …) Sensory Data (Heart rate, Video, Audio) Personal Information (behavior, interest) Sensory Data (Acc, GPS, Video) Social Data (Twitter) HDFS Data Access Interface

12. Architecture • Activity Recognizer • Take input from different sensors • Wearable sensors • Smartphone’s sensors • Video sensors • Recognize activities based on specific machine learning algorithms for each kind of data • Provide output as activity label in text format High Level Context-awareness Low Level Context-awareness Twitter Analyzer Activity Recognizer Emotion Recognizer System Data Wearable Sensor based AR Thesaurus Manager Classifying Attribute Thesaurus Morpheme Manager Feature Extraction Data Acquisition Training Models Analysis Sentiment Compilation Smartphone based AR Syntax Analyzer Positive, Negative Analyzer Decision Maker GPS Validation Morpheme Analyzer Feature Extraction Preprocessing Attribute-Emotion Mapping Module Attribute Extractor Emotion Extractor Physiological sensor based ER Video based ER Audio based ER Video based AR Decision Fusion Feature Extraction Feature Extraction Non-Param Cumulative Sum Classifying HMM Testing Classifying Auto Associate Neural Network Archive Listener • Synchronization Probability Computation Remote Control Request Module DMBS Connector Segmentation Data Acquisition Feature Extraction Preprocessing Face Detection • Statistical Feature • Extraction HMM Training Classification Tree Construction Activity Label (standing, sitting, running, …) Emotion Label (happy, angry, boredom, …) Sensory Data (Heart rate, Video, Audio) Personal Information (behavior, interest) Sensory Data (Acc, GPS, Video) Social Data (Twitter) HDFS Data Access Interface

13. Architecture • Emotion Recognizer • Take input from different sensors • Audio sensor • Video sensors • Physiological sensors • Recognize emotions based on specific machine learning algorithms for each kind of data • Apply Fusion technique to increase confident of predict output from different decisions. • Provide output as emotion label in text format High Level Context-awareness Low Level Context-awareness Twitter Analyzer Activity Recognizer Emotion Recognizer System Data Wearable Sensor based AR Thesaurus Manager Classifying Attribute Thesaurus Morpheme Manager Feature Extraction Data Acquisition Training Models Analysis Sentiment Compilation Smartphone based AR Syntax Analyzer Positive, Negative Analyzer Decision Maker GPS Validation Morpheme Analyzer Feature Extraction Preprocessing Attribute-Emotion Mapping Module Attribute Extractor Emotion Extractor Physiological sensor based ER Video based ER Audio based ER Video based AR Decision Fusion Feature Extraction Feature Extraction Non-Param Cumulative Sum Classifying HMM Testing Classifying Auto Associate Neural Network Archive Listener • Synchronization Probability Computation Remote Control Request Module DMBS Connector Segmentation Data Acquisition Feature Extraction Preprocessing Face Detection • Statistical Feature • Extraction HMM Training Classification Tree Construction Activity Label (standing, sitting, running, …) Emotion Label (happy, angry, boredom, …) Sensory Data (Heart rate, Video, Audio) Personal Information (behavior, interest) Sensory Data (Acc, GPS, Video) Social Data (Twitter) HDFS Data Access Interface

14. Tools and Technologies • Tools for development • MATLAB • Eclipse • Android SDK • Technologies • Machine Learning • Platforms • Microsoft Windows • Android OS

15. Development Timeline 1st Integration Phase Component Validation Interface Definition Validated Components Adapter Development 2nd Evaluation Phase Evaluate Components based on collected data Interface Report Evaluation Report Component Modification Component Adjustment Modified Components 1st year 2ndyear Final module Output

16. Current Status • Social Interaction Analyzer • Need to define the input and output to interact with Tapacross’s engine • Activity and Emotion Recognizer • Each individual module is available and ready for integration

17. References [WiiRemote] https://www.nintendo.com/wii/what-is-wii/#/controls [Han2012] Manhyung Han, La The Vinh, Young-Koo Lee and Sungyoung Lee, "Comprehensive Context Recognizer Based on Multimodal Sensors in a Smartphone", Journal of Sensors, vol. 12, no. 9, pp. 12588-12605, 2012 [Pourjam2013] Pourjam, E., Ide, I., Deguchi, D., & Murase, H. Segmentation of Human Instances Using Grab-cut and Active Shape Model Feedback. In proceddings of MVA2013 IAPR International Conference on Machine Vision Applications, pp. 77–80, May 20–23, 2013. [Atefian2013] Atefian, M., & Mahdavi-Nasab, H. (2013). A Robust Mean-Shift Tracking Using GMM Background Subtraction, J. Basic. Appl. Sci. Res., vol. 3, no. 4, 596–607, 2013. [MITMindReader]http://trac.media.mit.edu/mindreader/ [Ayadi2011] Ayadi, M.E., Kamel, M.S., Karray, F.: Survey on speech emotion recognition: Features, classification schemes, and databases. Pattern Recognition 44 (3), 572 - 587 (2011). [Lee2011] C.-C. Lee, E. Mower, C. Busso, S. Lee, and S. Narayanan. Emotion recognition using a hierarchical binary decision tree approach. Speech Commun., 53(9-10):1162-1171, Nov. 2011.

18. Thank You! lebavui@oslab.khu.ac.kr