Managing Context Information in Mobile Devices

1. Managing Context Information in Mobile Devices 2011. 3. 28. Panu Korpipää, Jani Mäntyjärvi,Juha Kela, Heikki Keränen, and Esko-Juhani Malm VTT Technical Research Centre of Finland

2. Outline • A context management framework • Information sources • Ontology for sensor-based context information • Inference mechanism concept • Bayesian reasoning for higher level contexts • Application programming interface • Adding context • Requests and responses • Subscriptions and indications • Example application • Discussion

3. A context management framework • Motivation • Mobile device users’ desire • Access and manipulation of the users’ information • Services specific to their location, time, and environment • Gathering context information from various sources • Sensors, networks, device status, user profiles, and other sources • Enhancing mobile applications’ usability • Letting the applications adapt to the users’ conditions • Problems to achieve true context awareness • Uncertain, rapidly changing, partially true data from multiple heterogeneous sources • Extracting relevant context information by fusing data from several sensors • Many factors such as noise, faulty connections, drift, miscalibration, wear and tear, humidity to degrade data acquisition

4. A context management framework • Problems to achieve true context awareness • Overlapping extracted contexts, changing contexts with time, only partially reliable approximations • Mobile devices’ dynamic environment • Learning from multidimensional data • Impossible easy generalization beyond training data • Occurrence of problems using even sufficiently reliable contexts • Users with different ideas of “context” • Proposed approach to address the challenges • A uniform mobile terminal software framework • Providing systematic methods for acquiring and processing useful context information from a user’s surroundings • Giving the context to applications

5. A context management framework • Framework recognizing semantic contexts in real time • Dealing with uncertain, noisy, and rapidly changing information • Delivering contexts for the terminal applications in an event-based manner • Application programming interface (API) for using semantic context information • Using an expandable context ontology to define contexts • Framework properties • Blackboard-based approach underlying communication paradigm between framework entities • Focusing on mobile terminal capabilities rather than infrastructure • Framework for the Symbian platform to achieve true device mobility, high performance, and a broad user base

6. A context management framework • Four main functional entities of the context framework • Context manager • Resource server • Context recognition service • Application • Context manager • A central server for providing services to other entities • Running on the mobile device itself • Black board-based context manager • Central node storing context information from any source • Serving context to clients in three ways • Direct querying the manager to gain context data • Subscription to various context change notification services • Using higher-level (composite) contexts transparently

7. A context management framework • Resource server • Connecting to any context data source • Posting context information to the context manager’s blackboard • Delivering context to the clients according to their subscriptions • Delivered data’s abstraction level → high enough • Delivered data’s frequency → low enough (not overwhelming) • Performing low-level recognition before delivering the data • Low-level context recognition process flow in the resource server • Measurement phase • Reading the sensors and output raw data • The preprocessing phase builds measurement data arrays that contain a certain number of samples (quantization of time dimension) and calculates generic features for each time interval.

8. A context management framework • Resource server • Performing low-level recognition before delivering the data • Preprocessing phase • Building measurement data arrays that contain a certain number of samples (quantization of time dimension) • Calculation of generic features for each time interval • Feature extraction • Calculation of more specific context features • Quantization phase • Binding the feature values to the real-world context which has a meaning for a person or an application according to a predefined ontology • Using fuzzy sets or crisp limits for quantizing extracted features • Outputs context atoms (base context units) that applications can use or recognition services can refine

9. A context management framework • Resource server • Two methods for quantization • Set crisp limits in a true-false labeling with fuzzy membership functions denoted as μ(x) • Quantizing environment sound intensity • Three quantities (Silent, Moderate, and Loud) corresponding to the three membership functions • If one of these is true, the others are false.

10. A context management framework • Resource server • Two methods for quantization • Apply a fuzzy set for features, resulting in continuous valued fuzzy labeling • μL(x) = 0.7 / Silent + 0.3 / Moderate + 0 / Loud

11. A context management framework • Context recognition services • Sharing recognized higher-level contexts • Addition and removal of recognition services by developers or processes • Using a set at a certain time instant or a time series of context atoms as input • Returning single higher-level contexts for the context manager • Summary of framework • Resource server and recognition service • Converting an unstructured raw measurement data flow into a representation defined in the context ontology (human-interpretable context information) for the applications in an event-based manner • Framework providing a semantic interface that enables more systematic and rapid application development • Efficient reuse of context information compared to using raw measurement data

12. A context management framework • Blocks for change detection and security • Not focusing on the functionalities • Any change detection services to detect context change • Security module for checking the trustworthiness of incoming contexts

13. Information sources • Acquiring context information from many possible sources • Using a microphone, three accelerometers, two channels for light, and sensors for temperature, humidity, and touch • Sensor data provided the starting point • Device processes that represent mobile devices’ internal information, such as applications currently running • The Internet and other networked resources • Global Positioning System, among the most important sources of location information • Internal device processes, which provide explicit information on device application use and users’ scheduled tasks, preferences, and social networks • Time information, used to associate certain events with others and form event sequences that might relate to a current higher-level context or predict a future event • Providing continuous and ad hoc connectivity to local and remote data • Supporting various wireless networks, such as GSM and GPRS, and short-range ad hoc networks, such as Bluetooth

14. Ontology for sensor-based context information • A common structure for representing information • Managing the context information systematically • Design of an ontology for representing sensor-based context information • A schema for ontology • A client-usable, extendable vocabulary that presents the terms for describing context information • Using the Resource Description Framework (www.w3c.org) as the description syntax

15. Ontology for sensor-based context information • Description of each context using six properties • Containing at least Context type and Context value • Context type • Referring to the context category • Primary parameter of all subscriptions and queries • Context value • Referring to the semantic or absolute “value” of context type • Usually being used together with context type • Containing an absolute numerical value or feature describing context in some cases • Confidence • An optional property which describes the context’s uncertainty • A probability or a fuzzy membership of context depending on the source • Source • Description of the context’s (semantic) source

16. Ontology for sensor-based context information • Description of each context using six properties • Timestamp • Denoting the latest time the context occurred • Attributes • Specifying the context expression freely • Containing any additional details not included in the other properties • Higher-level context and context atoms links to a recognition service • Containing the Input context type set for recognizing the higher-level context • Three properties for describing recognition services: • Context type • Using the category of the higher-level context as a normal context atom to request the recognition for the type

17. Ontology for sensor-based context information • Three properties for describing recognition services: • Context recognition service • Receiving input defined in Input context type set • Performing the classification, and returning the higher-level context • Input context type set • Subscription of the input context type set to be notified of the changes • Inferring the higher level context upon change in input type set • Seven context values in higher-level contexts from a set of 47 audio-based context atoms

18. Ontology for sensor-based context information • Management of uncertainty of sensor-based information • Confidence property • Description of context uncertainty: a context derived from multiple sensory input only holds true with a certain probability • Partly true context • Boundary between two contexts is not clear and discrete

19. Inference mechanism concept • The conceptual blackboard-based system • Use of any inference framework for reasoning new contexts or actions • Context uncertainty described in the confidence property • Useful in such inference frameworks as probabilistic networks, clustering, or case based reasoning • Recognition services • Input : information from the common data space • Output : higher-level contexts from either a set of contexts at a certain moment or a context history

20. Bayesian reasoning for higher level contexts • Using a naïve Bayes classifier to recognize higher-level contexts • Naïve Bayes classifier for online context inference and sensor fusion • Robust method with missing, uncertain, and incomplete information • Possible to use context data described by the ontology, a vector of context atom confidence values • Treating fuzzy membership values as virtual evidence • Computationally efficient • Training and inference both have a linear complexity in input data size • No requirement of background information for modeling • Exception for choosing the relevant network inputs • Examples of contexts to be recognized by a naive Bayes classifier

21. Bayesian reasoning for higher level contexts

22. Bayesian reasoning for higher level contexts • Higher-level context Outdoors from the context atoms • Indoor/Outdoor Bayesian network • Classification of the current situation’s context atom confidence values (vector) • Conditional probability indicating each input’s probability given the output • Learning conditional probabilities from the training data • Using naïve Bayesian networks to classify 9 contexts of a mobile device user in their normal daily activities • Nine contexts measured from a continuous real-life scenario containing different activities such as driving a car, running, walking, using an elevator, listening to different kinds of music, and speaking • The measurement system hardware • A small sensor box attached to the shoulder strap of a backpack containing a laptop

23. Bayesian reasoning for higher level contexts • The classification results • Indication that naïve Bayes classifier can extract situations fairly well • Valid only in a restricted scenario • Experiments under controlled scenarios • Nine different contexts, and measured by four people • Car, Elevator, and Tapwater with nearly 100 percent accuracy • Supervised learning approach • which defines the target classes and their inputs before training

24. Application programming interface • Context manager API • Implemented on the Symbian platform

25. Adding context • Processing and collecting the context properties by resource servers • Method for adding context: ContextAdd( ContextObject )

26. Requests and responses • Request of context information from the context manager database • Exposition of context ontology to application developer • Request context information by the context ontology tree (sub branch) • Response containing all contexts under the sub branch • Denoting responses to requests with an arrow (→) • ContextRequest( Environment:Light:Intensity ) • →Bright • ContextRequest( Environment:Light ) • →{Bright, Natural, NotAvailable} • ContextSetRequest( {Environment:Humidity, • Device:Activity:Position} • →{Dry, AntennaUp} • ContextSetRecognitionRequest( • Environment:Sound:Type ) • →Car

27. Subscriptions and indications • Giving applications the required context information in an event-based manner • Subscription to context change notifications • Examples illustrate how applications use this service • ContextChangeSubscribe(Device:Activity:Placement) • →AtHand • ContextChangeSubscribe( • Environment:Temperature:Absolute ) • →21 • ContextStartSubscribe( Location:Facility, MovieTheatre) • →MovieTheatre • ContextChangeSubscribe( • Environment:Sound:Type ) • →Elevator

28. Example application • Using the context ontology to demonstrate the use of sensor-based context information in mobile terminal applications • Such as browsing and information presentation in the user interface • Environment:Light:Intensity, Environment:Sound:Intensity, and User:Activity:PeriodicMovement • Environment sound intensity • Control of the volume of operating tones • User activity and ambient light level • Adaptation of font size, screen brightness, and service content • Context-based information representations of a bus timetable service

29. Example application • Using fuzzy rules to help guide the application adaptation • Fuzzy rule definition for adapting font size • Using the logical operation AND • Intersection between two fuzzy sets • Environment:Light:Intensity and User:Activity:PeriodicMovement: • Using the logical operations OR and NOT • μFONTSIZE (x) = • min(μUSER:ACTIVITY:PERIODICMOVEMENT(x), • μENVIRONMENT:LIGHT:INTENSITY (x)).

30. Discussion • Framework and context ontology • Providing a semantic application programming interface for handling imprecise information from multiple sources • Many remaining obstacles to the rapid use of many contexts derived from sensor data • Need for more sensors and a large data collection to gain enough information • Several advantages of high-level context • Event based communication of high-abstraction-level data • Communicating low-level data up to a client continuously • Consumption of more processing power and the limited battery resources. • Optimal use of the context manager storage