Intelligent sensor and learning challenges for context aware appliances

1. Intelligent sensorand learning challengesfor context aware appliances Intelligent sensorand learning challengesfor context aware appliances >>Stéphane Canu scanu@insa-rouen.fr asi.insa-rouen.fr/~scanu INSA Rouen, France - EU Laboratoire PSI

2. 1984: La souris et leMacintoch 200X : la nouvelle rupture "break through"

3. La technologie d'aujourd'hui • Loi de Moore • Communication "sans fil" • L'ère des données Quelles applications ?

4. Wearable

5. IHM Olympus Optical Co., Ltd. is pleased to announce its new wearable user interface technologies. Employing gestures and other hand movements for input, the system is an ideal match for new wearable PCs.

6. Wearable http://www.redwoodhouse.com/wearable/index.html http://wearables.cs.bris.ac.uk/public/wearables/esleeve.htm http://www.ices.cmu.edu/design/streetware/

7. Reasearch on wearable

8. Wearable

9. context aware appliances The mediacup (calm version of the active badge) Phone by night http://mediacup.teco.edu/overview/engl/m_what.html

10. General Motors and CMU The car -drives together - informs you -in a parking… GM/CMU Companion driver interface system

11. Oops! Where is my car? • Old fashion software design: process • Match the sentence • Send the query to the satellite • Satellite send query to the car on its own frequency • Car answers… • Tell the computer what to do (where is the switch) • Distributed software design: interaction • Software agents talk together • Future way: Programming by Example • Show the computer what to do • Today's solution: Louis my 3 years old son Disappearing computer >>Your Wish is My Command: Programming by Example Henry Lieberman, editor, Published by Morgan Kaufmann, 2001.

12. Calm technology • Ubiquitous computing • One people - many computer • Technology at our service • Reactive to what user do • Proactive - Prepare what to do next • Situated – sharing context (Hans Gellersen, Sensing in Ubiquitous Computing) • Adapted to our needs • New functionalities and new behaviors • New way of communicating • Learn to adapt Machines have to know their context >> M. Weiser "The Computer for the 21st Century." Scientific American, September 1991

13. Context input Explicit Input sensors Explicit Output actuators Context-aware application Context output What is the context? • user • activity (available/meeting) • location, • identity, profile • environment monitoring • time, day/night, temperature, weather, • resources (networks, services…) • appliance - proprioception • usage - functionalities • maintenance • resources (energy…) Adapted From Henry Lieberman and Ted Selker, Out of Context: Computer Systems That Adapt To, and Learn From, Context, IBM Systems Journal 39, 2000. + history… Abstract representation of the situation Knowledge? How to find it from data?

14. Sensing context from theenvironmentpresentation roadmap • Data • Representation • Information retrieval • Context evolution • User interaction >> Kristof Van Laerhoven, Kofi Aidoo: Teaching Context to Applications In Personal and Ubiquitous Computing, Volume 5 Issue 1 (2001) pp 46-49

15. Context from data >> Data Representation Information retrieval Context evolution • Unbelievable capacity • Moore’s law • New sensors • Artificial nose • Bio sensor • “Personal” data • humor: affective computing Data Era! http://www-stat.stanford.edu/~donoho/lectures.html

16. Biological sensors >> Data Representation Information retrieval Context evolution How areyou? http://www.teco.edu/tea/sensors.html

17. Expression recognition >> Data Representation Information retrieval Context evolution Machine Perception LabFace Detection and Expression Recognition http://markov.ucsd.edu/~movellan/mplab/index.html

18. Too much informationkills information >> Data Representation Information retrieval Context evolution "We are drowning in information and starving for knowledge." - Rutherford D. Roger • Critic of the "Data Era" • Data smog • Non measurable things • Ethical consequences • the Orwellian future Filter data!

19. Intelligent sensors >> Data Representation Information retrieval Context evolution • Requirements: • Data • Accuracy and confidence • Self diagnostic • Self calibration • How to do it? • Uncertainty management • Learning ability • Network + database • Adaptation ability • Fault detection mechanism Associated software sensors >>S. Canu et al., "Black-box Software Sensor Design for Environmental Monitoring" , in International Conference on Artificial Neural Networks , Skovde, Sweden. Sep 2-4, 1998 (and related work on data validation within the EM2S project)

20. Data validation >> Data Representation Information retrieval Context evolution • Mono sensor validation • Static validation • Mean, variance • Dynamic validation • Cusum (control charts) • Trend analysis • Multisensor validation • Residual analysis • Fusion: Joint probability estimation • Prior knowledge: Balanced relations • Hierarchical validation • Multisensor perception Interactive matrix of smart sensors >>http://www.accenture.com/xd/xd.asp?it=enWeb&xd=services\technology\research\tech_sensor_matrix.xml >> K. Van Laerhoven, A. Schmidt and H.-W. Gellersen. "Multi-Sensor Context-Aware Clothing". In Proceedings of the 6th International Symposium on Wearable Computers, 2002

21. Software sensor >> Data Representation Information retrieval Context evolution • Value + confidence interval + validity domain • How to build it? • From a model: tracking = Kalman filter • When no model is available: learn it! Raw datav(t) Raw data x(t) environment Raw data y(t) Raw data z(t) learning = Black box modeling

22. Towards proprioceptors >> Data Representation Information retrieval Context evolution • Learn • How to learn? • Gaussian mixture + EM • Include prior: Bayesian networks • Deal with uncertainty: Evidence framework • Use to: • Detect non nominal situations • Replace missing data d = Curse of dimensionality (Belman) • >> E. Petriu et al., "Sensor based information appliances",

23. What is data? >> Data Representation Information retrieval Context evolution • Individuals or measurements • Associated variables • Data set (matrix) • line = measurements • column = variable • Data: point clouds • Data exploration: recognize patterns too many data: SUMARIZE

24. Summarize data Data Representation Information retrieval Context evolution >> • Non linear components analysis • Feature space: kernel (PCA or ICA) • Local linear • Quantisation (SOM) • Relevant distance • Select features • Local adapted representation • Feature selection • Select relevant situations • Sparse learning • Kernel learning Kernel representation >> J. Mäntyjärvi, J. Himberg, P. Korpipää, H. .Mannila,"Extracting the Context of a Mobile Device User", 8thSymposium on Human-Machine Systems-HMS,Kassel,Germany,2001.

25. Data' Data Influence map Kernel representationDistance maps j i Example in 2 dimension of the influence map of the "black circle". Red color denotes a high influence while the low influence zones are in blue. Analyze data proximity through the kernel map >> B. Scholkopf and A. Smola, "Leaning with Kernels", MIT Press, 2001

26. Example of kernel map Class 1 Class 2 Class 2 Data clouds in two dimensions Associated kernel map Even in d dimensions you can visualize >>S. Canu and al., "Functionnal learning through kernels", invited lecture at the NATO institute in Leuven, 2002

27. Looking for hiden shapes Data Representation Information retrieval Context evolution >> • Data point = information + noise • Principal curve • Non linear PCA • Independent curve • Non linear ICA Kernel representation + linear analysis >> Balázs Kégl http://www.iro.umontreal.ca/~kegl/research/pcurves/

28. Navigatein high dimensional space Data Representation Information retrieval Context evolution >> >> J. B. Tenenbaum, V. de Silva and J. C. Langford http://isomap.stanford.edu/handfig.html

29. Information retrieval Data Representation Information retrieval Context evolution >> • What for • User profiling • User identification • Battery discharge rate • Sequence induction… • Classification problem • Decision theory • Example based programming • Learning machine Select relevant cases

30. A brief historical perspectiveof machine learning Data Representation Information retrieval Context evolution >> • Before machines • Statistics: PCA, DA, regression, CART, kNN • 70's - Learning is logic • Grammatical inference in expert systems • 80's - Learning is human • Neural networks: backprop • 90's - Learning is a problem: COLT • Kernel machines: SVM • Mixture of experts: adaboost What is the learning problem? >> T. Hastie, R. Tibshirani and J. Friedman, "The elements of statistical learning", Springer, 2001

31. Fit Summarize data data What is learning? • Data • Training set • Test point looking for such that • Learning is balancing • Hypothesis set (Neural networks, Kernels) • Fitting criterion (least square, absolute value) • Compression criterion (penalization, Margin) • Balancing mechanism (cross validation,generalization) Learning is summarizing

32. + + + + + + Linear discriminationseparable case Data Representation Information retrieval Context evolution >> How to correctly classify all points? Occam Razor's wx+ b=0 (w,b) ??? + + + + + + + Use hyperplane

33. + + + + + + Linear discriminationseparable case Data Representation Information retrieval Context evolution >> How to correctly classify all points? wx+ b=0 + + + + + + + Be sparse

34. + + + + + + + + + + + + + The classifier Margin Data Representation Information retrieval Context evolution >> How to correctly classify all points? Margin wx+ b=0 Margin Be sparse

35. + + + + + + + + + + + + + Maximize the marginBe sparse Data Representation Information retrieval Context evolution >> How to correctly classify all points? wx+ b=1 Margin wx+ b=0 Margin wx+ b=-1 Support Vector Machines: SVM >> V. N. Vapnik, "The nature of statistical learning theory", Springer-Verlag, 1995

36. Fit Summarize data data What is learning? • Data • Training set • Test point looking for such that • Learning is balancing • Hypothesis set (Neural networks, Kernels) • Fitting criterion (least square, absolute value) • Compression criterion (penalization, Margin) • Balancing mechanism (cross validation,generalization) SVM Learning is summarizing >> S. Canu, A. Rakotomamonjy, Ozone peak and pollution forecasting using Support Vectors, IFAC workshop, Yokohama, 2001.

37. Summarize Inputadaptive scaling Data Representation Information retrieval Context evolution >> • Enumerate all combination …and score • Preprocessing • Information theory • Statistical test • Wrapper • Use a relevance index • Learn and select together Example of relevance index for a toy problem with 2 relevant features and 50 irrelevants Global formulation >>Y. Gandvalet and S. Canu, "Adaptive Scaling for Feature Selection in SVMs", accepted for publication at NIPS 2002

38. Summarize patterns Data Representation Information retrieval Context evolution >> • Dimension reduction by • >>multi-resolution analysis • (just like in your eyes…) • Learn at the relevant scale • >> multi scale representation • Efficient implementation • ridgelets, curvelets • wavelets’ kernel "Kernelize" wavelets >>A. Rakotomamonjy and S. Canu, "Frame, Reproducing Kernel, Regularization and Learning", accepted in JMLR 2002

39. Learning machines challenges Data Representation Information retrieval Context evolution >> • Hypothesis set • Multi scale data representation: wavelets • Use context: mixture of experts • Fitting criterion • Sparse distance criterion • Select relevant input (adaptive scaling) • Relevant distance: adapt the kernel • Compression criterion • Information issues • Global optimization • Balancing mechanism • Efficient direct algorithm (one shot learning) Towards Context based learning

40. Context assessment Data Representation Information retrieval Context evolution >> • Deal with uncertainty • plausibility / credibility • unknown states / ability to evolve • data fusion: evidence theory • Take into account prior knowledge: transitions • temporal representation • uncertain transitions • learn probabilities or possibilities • Learn the model • don't start from scratch • create and delete contexts • Adapt context determination to user • from a global imprecise context to specific context How to implement context?

41. Context implementation • Context = state • List of variables • Petri's nets • State = stochastic • Markov model • Bayesian networks • Identify = decision theory (data fusion) • Information retrieval • Learn context • Knowledge discovery • Create / delete • Context hierarchy (time granularity) Context is a language How to retrieve the context? Henry Lieberman: http://web.media.mit.edu/~lieber/

42. New idea to deal with context Data Representation Information retrieval Context evolution >> • Current context: working memory • Prior knowledge: transition law • Available information: evidence • Data fusion • Learn context • Transition law • Context retrieval from data • Context is a language • Speech recognition • Markovian model • Evidence • Language + previous state • Locator's adaptation Adapt speech recognition ideas to context http://htk.eng.cam.ac.uk/

43. Context: Research chalenges Data Representation Information retrieval Context evolution >> • Inputs • Deal with uncertainty (and missing data) • Representation • Data fusion (multimedia fusion) • Context • Define a language • Represent previous state • Learn transition • Feed Back to inputs • Adapt transition to the user • Loop the user: reinforcement • Control mechanism (stability/plasticity dilemma) Challenging research issues http://cslu.cse.ogi.edu/tutordemos/nnet_training/tutorial.html

44. Break through Theoretical models are essentials (Mark Weiser, Computer Science Challenges for the Next Ten Years) • What is information? • Computer science • Coding • Signal • Mathematics • Statistics & computer science • Pattern recognition • Functional analysis • ?????? …remember Albert and relativity >> L. Devroye, L. Györfi and G. Lugosi, "A Probabilistic Theory of Pattern Recognition", Springer-Verlag 1996.

45. My long bet Before 2050 We will faced a scientific revolution regarding information definition Comparable with the one induced in physics by the relativity theory $ 500 To greenpeace Long bet fundation at San Francisco http://www.longbets.org/

46. Research challenges Bayesian networks • create context • how to define prior contexts: user’s needs • how to represent contexts: stochastic automaton • learn from data: modify, create and destroy context • decide context • validate data software sensors • select relevant inputs representation + distance • select relevant patterns wavelets • select relevant situations SVM and kernel • make decision using data fusion Dempster-Shafer + EM • loop with the user • reinforcement learning • user’s needs Integrate: create relevant learning architecture

47. Questions? • Asia • Scurry™, Wearable & Virtual Keyboard - Samsung, • K. Doya for reinforcement • America • Context Aware Computing group - Media lab MIT • CMU, Stanford • Georgia tech: Future Computing Environments • Smart Matter Integrated Systems (Xerox PARC) • Montreal – learning lab • Australia • ANU for learning • University of South Australia -  wearablecomputerlab • Europe • Telecooperation Office (TecO) at the University of Karlsruhe • The disappearing computer, a EU-funded proactive initiative • The Smart-Its project • Equator project focuses on the integration of physical and digital interaction • Perceptual Computing in general and Computer Vision in ETH Zurich • IDIAP for machine learning and speech recognition • PSI, France for learning Some context aware references

49. From macroscopic… >> Data Representation Information retrieval Context evolution

50. …to Microscopic data >> Data Representation Information retrieval Context evolution MCell Simulation of miniature endplate current generation at the neuromuscular junction. Image rendered with Pixar Photorealistic RenderMan. http://www.mcell.cnl.salk.edu/