ECE 494 Capstone Design Final Design Presentation Smartphone Based Human Behavioral Analysis

1. ECE 494 Capstone DesignFinalDesign PresentationSmartphone Based Human Behavioral Analysis Andrew Jackson Michael Armstrong Robbie Rosati Andy McWilliams Aaron Stewart April 18, 2014 Advisor:Dr. FeiHu

2. Outline Smartphone Based Human Behavioral Analysis Project Recap & Goal System Diagram Subsystem Breakdown & Team Roles Sensor Data Extraction & Analysis App Development & Activity Recognition DTW Algorithm Environment Sensors NN Algorithm Raw Memory Extraction & Analysis HMM Algorithm Administration

3. Project Recap & Goal Smartphone Based Human Behavioral Analysis • Create a system for tracking and detecting a user’s behavioral patterns though the phone’s sensors and internal memory logs. • Can be used in multiple areas: • Healthcare • Activity Monitoring • Homeland Security

4. System Diagram Smartphone Based Human Behavioral Analysis

5. Subsystem Breakdown & Team Roles Smartphone Based Human Behavioral Analysis

6. Sensor Data Extraction & Analysis Aaron Stewart

7. Android, iOS, or Windows Phone We chose to go with Android. • No development license fees • Majority of market • Many sensors available • Open file system • Previous Android programming experience • Cheaper phone prices Smartphone Based Human Behavioral Analysis

8. Samsung Galaxy S4 Mini Smartphone Based Human Behavioral Analysis New phone with many sensors Fairly inexpensive for an unlocked phone Smaller size great for testing in pockets

9. Sensors Smartphone Based Human Behavioral Analysis

10. Sensor Types Smartphone Based Human Behavioral Analysis

11. Sensor Types Smartphone Based Human Behavioral Analysis

12. Behavior Analysis App Version 1.0 Smartphone Based Human Behavioral Analysis Shows values from each sensor in real-time as the sensor updates Can save the data to a file on phone Move file to computer to analyze with MATLAB Graph the data Validate that it makes sense

13. Testing with App Smartphone Based Human Behavioral Analysis Most sensor data comes in a set of 3 points Few have single values Tests related to typical smartphone user behaviors Used MATLAB for data graphing and filtering

14. Light Sensor Test (Different Exposure) Smartphone Based Human Behavioral Analysis

15. Accelerometer Test (Answer Call) X Y Z Smartphone Based Human Behavioral Analysis

16. App Development and Activity Recognition Andrew Jackson

17. Smartphone Based Human Behavioral Analysis

18. Behavior Analysis App Version 2.0 Smartphone Based Human Behavioral Analysis Integrates DTW machine learning algorithm Allows training of the algorithm from the phone – no computer required Once trained, will recognize where the phone is with almost 100% accuracy Visual Upgrades All features from previous app rolled over

19. Further App Improvements Smartphone Based Human Behavioral Analysis • Increased accuracy of activity recognition by • Increasing sensor sensitivity • Polling each sensor more times a second • Tweaking DTW algorithm code • Using multiple sensors at the same time • Continued adding new activities • Added speech recognition By demo day: • Add in GPS coordinates • Further explore light, sound, and proximity sensor possibilities

20. Accelerometer, Light, and Proximity Smartphone Based Human Behavioral Analysis Chose to work with accelerometer first because it can give some of the most useful data about the phone Applying DTW on accelerometer data Next, applied light and proximity sensors to get better results

21. Recognized Activities Last Presentation Now Smartphone Based Human Behavioral Analysis Walking (with phone in hand) Talking on phone Sitting on table Holding in hand Walking (with phone in pocket) Running Stairs Sitting Driving

22. App Demonstration Smartphone Based Human Behavioral Analysis

23. Dynamic Time Warping Andrew Jackson

24. Dynamic Time Warping Measures similarity between two sequences which may vary in time or speed Calculates an optimal match between the two given sequences or time series A distance-like quantity is measured between the two series Smartphone Based Human Behavioral Analysis

25. DTW Time Series Cost: 3.3084e+05 Cost: 2.7239e+06 Smartphone Based Human Behavioral Analysis Time series can be accelerated/decelerated as much as necessary to give an optimal match.

26. Applying DTW to the Project Smartphone Based Human Behavioral Analysis Allows training for different activities with disregard to time Supports three-dimensional data Provides extremely accurate results Runs in O(n2) time

27. FastDTW Smartphone Based Human Behavioral Analysis Open-source algorithm based on original DTW Runs in O(n) time Slightly less accurate Easy to port to Android

28. Environmental Sensors Robbie Rosati

29. neural networks Proximity Sensor Smartphone Based Human Behavioral Analysis Gives proximity in cm Most phones only return binary values “near” and “far” Ability to check whether phone is pressed against ear, in pocket, etc.

30. Light Sensor Smartphone Based Human Behavioral Analysis Detects ambient luminosity in lux Useful for indoor/outdoor detection Could be included in gestures, used with DTW

31. GPS Smartphone Based Human Behavioral Analysis Can get user’s latitude and longitude Could improve detection for if user is driving Worst sensor with battery life Need to only use it occasionally

32. Microphone Smartphone Based Human Behavioral Analysis Speaker recognition via NN algorithm Passive or active detection Could use to detect loudness of rooms Also see things in sound waveform – like snoring

33. Neural Networks Robbie Rosati

34. Transition from Support Vector Machine to Neural Networks Smartphone Based Human Behavioral Analysis Implemented SVM into our Behavior Analysis app Ran too inefficiently for phone hardware Difficult to train with the sound sensor Therefore, decided to use an alternative algorithm that would fit our needs. Decided on Neural Networks, a popular machine learning algorithm for speech recognition

35. Neural Networks Smartphone Based Human Behavioral Analysis Algorithm used for machine learning and pattern recognition Inspired by the way the brain recognizes objects and sound Presented as systems of interconnected neurons that can compute values Difficult to train in a short time so used an API from Google to offload processing from the phone Integrates speech recognition into the app

36. Google Speech API Smartphone Based Human Behavioral Analysis Open source API for speech recognition Could be coded into our Behavior Analysis app Uses NN to interpret speech Can be presented in text with further coding However, the app requires an internet connection for this function since it streams audio to remote servers

37. App Demonstration Smartphone Based Human Behavioral Analysis

38. Raw Memory Extraction & Analysis Michael Armstrong

39. Smartphone Based Human Behavioral Analysis

40. Overview Smartphone Based Human Behavioral Analysis • Retrieving the physical image of a device is our goal. • Immense variety of phones with an array of OS and applications. • Current solutions are time consuming and/or very expensive. • Access to deleted data • A logical image is easier to obtain, but it omits deleted data, and logical extraction interfaces usually enforce access rules and may modify data upon access.

41. Samsung Galaxy S3 Mini Smartphone Based Human Behavioral Analysis Relatively inexpensive Compatible with team’s SIM cards SD card slot Has some sensors in case we need it as back up for sensor testing

42. Testing Options Smartphone Based Human Behavioral Analysis • Flashing is interpreted as a dump of the phones memory into a format that is either hexadecimal or binary. • Backup Software • Flashing box • Linux Forensics Software • dc3dd

43. dc3dd Smartphone Based Human Behavioral Analysis Terminal based utility for Linux Parses a partition bit-by-bit and creates binary image Advantages: Exactly what we need, easy to use Disadvantage: Parses through empty space and fills it with zeroes, creating a very large image file full of nothing however, there is a workaround for this issue

44. Steps for Extraction Smartphone Based Human Behavioral Analysis Gain root access to Android using simple utility Extract a copy of the database file Shrink the partition of a USB flash drive to the smallest possible size and copy the database file to the partition Use dc3dd to parse the partition and create the binary image for the database file

45. Viewing the Image File Smartphone Based Human Behavioral Analysis Any hex viewer application should able to view the data inside the image Chose GHex because it is easy to obtain with Ubuntu and has search functions for finding the data we are looking for

46. Finding Desired Data Smartphone Based Human Behavioral Analysis Messages and numbers are encoded as ASCII Locate corresponding hex data using the search function in Ghex Verify by comparing the located hex data to ASCII values Process should work for any type of file because the ASCII data will there regardless of file type

47. Example: Text Message Smartphone Based Human Behavioral Analysis

48. ‘Strings’ and ‘Grep’ Commands Smartphone Based Human Behavioral Analysis Strings - Captures groups of letters separated by spaces in a file Grep - Searches for an expression matching the user’s input expression

49. Hidden Markov Model Andy McWilliams

50. Hidden Markov Model Smartphone Based Human Behavioral Analysis Way of predicting the next possible output in a sequence of events Equivalent to recognizing or interpreting that particular sequence Slightly complicated because there can be multiple ways to produce the same observed output