Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

1. Darwin Phones: The Evolution of Sensing and Inference on Mobile PhonesPresented By: Brandon Ochs EmilianoMiluzzo, Cory T. Cornelius, AshwinRamaswamy, TanzeemChoudhury, Zhigang Liu, Andrew T. Campbell, "Darwin phones: the evolution of sensing and inference on mobile phones," In Proc. of 8th ACM Conference on Mobile Systems, Applications, and Services (MobiSys), 2010, pp. 5-20.

2. What does Darwin do? • A Smartphone platform for urban sensing • Proof of concept model uses microphone • Communicates with other local devices to improve inference accuracy (collaborative inference) • Framework can be expanded to gather information using a range of sensor data

3. What about battery life? • Communicates with backend server to do the CPU-intensive machine learning algorithms • Local devices share models rather than re-computing them • Sensing is enabled/disabled as the system sees fit

4. Common Urban Sensing Challenges • Human burden of training classifiers • Ability to perform reliably in different environments (indoor vs outdoor) • The ability to scale to a large number of phones without hurting usability and battery life. • Darwin overcomes all of these through classifier/model evolution, model pooling, and collaborative inference

5. Types of Learning • Supervised: Given a fully-labeled training set • Semi-Supervised: Given a small training set that is evolved • Unsupervised: No training set is given

6. Darwin Steps • Evolution, Pooling, and Collaborative Inference • These represent Darwin’s novel evolve-pool-collaborate model implemented on mobile phones

7. Classifier Evolution • Automated approach to updating models over time • Needs to account for variability in sensing conditions and settings • Variability in background noise and phone location require separate models

8. Model Pooling • Reuses models that have already been built and evolved on other phones • Exchange classification models whenever the model is available from another phone • Classifiers do not need to be retrained, which increases scalability • Can pool models from backend servers

9. Collaborative Inference • Combines results from multiple phones • Run inference algorithms in parallel on the same classifiers • System is more robust to degradation in sensing quality • Increases accuracy

10. Darwin Design: Computation • Reduces the on-the-phone computation by offloading some of the work to backend servers • Backend server uses a machine learning algorithm to compute a Gaussian Mixture Model (2 hours for 15 seconds of audio) • Feature vectors are computed locally

11. Darwin Design: Context • Context (in/out of pocket, in/out of bag) will impact the sensing and inference capability • Classifier evolution makes sure the classifier of an event is robust across different environments

12. Darwin Design: Co-location • Accounts for a group of co-located phones running the same classification algorithm and sensing the same event but computing different inference results • Phones pool classification models when collocated or from backend servers • Compares against its own model and the co-located model • Drastically reduces classification latency • Exploits diversity of different phone sensing context viewpoints

13. Speaker Recognition • Attempts to identify a speaker by analyzing the microphone’s audio stream • Suppresses silence, low amplitude audio, and chunks that do not contain human voice • Reduce false positives by pre-processing in 32ms blocks

14. Speaker Modeling • Feature vector consisting of Mel Frequency Cepstral Coefficients • Each speaker is modeled with 20 Gaussians • An initial speaker model is built by collecting a short training sample

15. Classifier Evolution: Training Step • Short training phase (30 seconds) used to build a model which is later evolved • First 15 seconds used as the training set • Last 15 seconds used as baseline for evolution

16. Classifier Evolution: Evolution Step • Semi-supervised learning strategy • If the likelihood of the incoming audio stream is much lower than any of the baselines then a new model is evolved

18. Collaborative Inference • Local inference phase can be broken into three steps: • Local inference operated by each individual phone • Propagation of the result of the local inference to the neighboring phones • Final inference based on the neighboring mobile phones local inference results • Each node individually operates inference on the sensed event • Results and confidence broadcasted

20. Privacy and Trust • Raw sensor data is not stored on or leaves the mobile phone • The content of a conversation or raw audio data is never disclosed • Users can choose to opt out of Darwin

21. Experimental Results • Tested using a mixture of five N97 and iPhones used by eight people over a period of two weeks • Audio recorded in different locations • Classifier trained indoors

22. Experiment 1 Parameters • Three people walk along a sidewalk of a busy road and engage in conversation • The speaker recognition application without the Darwin components runs on each of the phones carried by the people

23. Experiment 1 Results: Without Evolution

24. Experiment 2 Parameters • Meeting setting in an office environment where 8 people are involved in conversation • The phones are located at different distances from people in the meeting, some on the table and some in people’s pockets

25. Experiment 2 Results

26. Experiment 2 Results

27. Experiment 3 Parameters • Five phones in a noisy restaurant • Three of the five people are engaged in conversation • Two of the five phones are placed on the table • Phone 4 Is the closest phone to speaker 4 and also the closest phone to another group of people having a loud conversation

28. Experiment 3 Results

29. Experiment 3 Results

30. Experiment 3 Results

31. Experiment 3 Results

32. Experiment 4 Parameters • Five people walk along a sidewalk and three of them are talking • The greatest improvement is observed by speaker 1, whose phone is clipped to their belt

33. Experiment 4 Results

34. Experiment 4 Results

35. Time and Energy Measurements • Baselines for power use determined • Measurements performed using the Nokia Energy Profiler tool • No data gathered for the iPhone • Smart duty cycling required later to save battery life

36. Time and Energy Measurements

37. Possible Applications • Virtual square application • Social application for a group of friends • Place discovery application • Use collaborative inference to determine location • Friend Tagging application • Exploit face recognition to tag friends on pictures

38. Future Work • Duty cycling for improved battery life • Simplified classification techniques

39. Improvements On The Paper • Studies don’t show conclusive evidence; there should be separate control models for each of the scenarios

40. Conclusion • The Darwin system combines classifier evolution, model pooling, and collaborative inference • Results indicate that the performance boost offered by Darwin off sets problems with sensing context • The Darwin system provides a scalable framework that can be used for other urban sensing applications

41. References • [1] Emiliano Miluzzo, Cory T. Cornelius, Ashwin Ramaswamy, Tanzeem Choudhury, Zhigang Liu, Andrew T. Campbell, "Darwin phones: the evolution of sensing and inference on mobile phones," In Proc. of 8th ACM Conference on Mobile Systems, Applications, and Services (MobiSys), 2010, pp. 5-20.  • [2] H. Ezzaidi and J. Rouat. Pitch and MFCC Dependent GMM Models for Speaker Identification systems. In Electrical and Computer Engineering, 2004. Canadian Conference on, volume 1, 2004 • [3] H. Ezzaidi and J. Rouat. Pitch and MFCC Dependent GMM Models for Speaker Identification systems. In Electrical and Computer Engineering, 2004. Canadian Conference on, volume 1, 2004.