The Context Engine - A Device to Augment Cognition

1. The Context Engine -A Device to Augment Cognition Perry McDowell The MOVES Institute

2. Augment Cognition • Add to your thinking, hopefully improving it • That’s been the whole point of computers • We want to make it completely natural and automatic • In some ways, similar to: • An admiral’s aide or efficient secretary • A senior NCO or subject matter expert • Want to do this with a wearable computer • Move nearer to ubiquitous computing

3. Literary Inspiration for the Military Context Engine • “Our suits give us better eyes, better ears… better intelligence… the beauty of it [is] you don’t have to drive it, fly it, conn it, operate it, you just wear it… This leaves you with your whole mind free to handle your weapons and notice what is going on around you… If you load a mud foot down with a lot of gadgets that he has to watch, someone a lot more simply equipped – say with a stone axe – will sneak up and bash his head in while he is trying to read a vernier.” - Robert A. Heinlein, Starship Troopers

4. Steve Mann’s Six Attributes of Wearable Computing • Unmonopolizing of the user's attention • It does not cut you off from the outside world. • You can attend to other matters while using the apparatus. • Assumes computing will be a secondary activity. • Provides enhanced sensory capabilities. • Unrestrictive to the user • You can do other things while using it

5. Steve Mann’s Six Attributes of Wearable Computing(cont) • Observable by the user • It can get your attention continuously if you desire • The output medium is constantly perceptible by the wearer. • Controllable by the user • Attentive to the environment • Environmentally aware, multimodal, multisensory. • Communicative to others • Can be used as a communications medium.

6. Steve Mann’s Six Attributes of Wearable Computing(cont) • Implied by the above six properties is that it must also be: • Constant: always on, running, and ready. • Personal: human and computer are inextricably intertwined. • Prosthetic: you can adapt to it so that it acts as a true extension of mind and body; after time you forget that you are wearing it. • Assertive: resists, if you wish, prohibition or requests by others for removal. • Private: others can't observe or control it unless you let them.

7. Example of Uses -Finding a Restaurant • You’re downtown and decide you want to eat out • The engine makes a recommendation based upon several factors: • Location • Cost • Favorite foods • Favorite restaurants • Tonight’s specials • Friends’ favorites • Reviews • Current wait

8. Finding a Restaurant • Let’s take it one step further: • You can tell it you want to take your wife out for your anniversary - handle it • It knows your favorite romantic restaurants, so it makes a reservation at one • It contacts one of the babysitters you use and arranges services for that night • Orders flowers delivered to your wife’s office • Reminds you to buy a present, or orders one from the web for you • Schedules it with your wife’s contact engine • Tells you that everything is ready to go

9. Examples of Uses - Watch Turnover • At watch relief, transfer all pertinent data to your relief • List of contacts and their status, current/ pending orders, equipment status, upcoming events, etc. • The system can’t forget anything, rush so it can hit the pit, misinterpret what it’s told… • The system would also be able to give you advice - “No, you don’t want to turn left…”

10. Example of Uses - Crossing the River • You are the leader of a platoon of infantry on a covert mission, currently moving to conduct a rendezvous across a river • The bridge you were expecting to use is destroyed • What are you going to do now?

11. Crossing the River (cont) • Determining your current options, weighing them, and deciding upon a course of action takes time • Context engine immediately determines options and the best course of action and makes a recommendation • It can explain its reasoning fairly quickly to you

12. Examples of Uses -Summary • In the dinner example, it acts mainly as an assistant - it knows your likes and searches data for the best match • In the river example, it acts as an expert system, giving you a better solution • In the watch example, it combines the two - it stores data for use later, but can have intelligence (either the previous watch officer or from the system) built in

13. Goals • Our goal for this project is to take Commercially Off the Shelf (COTS) hardware and software to create a proof of concept for the Context Engine.

14. Difficult Problems • Determining the “symbolic content” of the user’s environment • Once it is symbolized, interpreting the environment to give information and advice • Presenting the information to the user in an unobtrusive and helpful manner

15. Visual Display Aural Display Actions People Determine Course of Action Outer World Abstraction Output to User Goal 3 Goal 2 Goal 4 Goal 1 Goal n Location Objects Preferences User’s Commands Breakdown of the Context Engine’s Processes

16. Actions People Outer World Abstraction Location Objects Initial Process - Recognize the World To course of action

17. Symbolic Content • Having the computer determine where it is, who/what is present and what is happening • Computer visualization • Wireless recognition • Bluetooth • Aural clues • GPS

18. Current Systems Which Convert Data to Abstraction • Face Recognition • Several companies has fairly effective facial recognition systems. • Lau Technology’s daughter company, Viisage, scanned all the faces at the Super Bowl, looking for terrorists • We anticipate getting an SDK from one of these companies to begin our research

19. Current Systems Which Convert Data to Abstraction • Wearable belt for outdoor sports • Determines events which might indicate the wearer is incapacitated • High acceleration • Vital Signs • Alerts authorities to potential injury

20. Determine Course of Action Goal 1 Goal 4 Goal n Goal 2 Goal 3 User’s Commands Determining The Course of Action From World Recognition To Output to User

21. Determining the Course of Action • Once you know what is in the environment, what does that mean? • Network solution • Take the dynamic extensibility from NPSNET-V • Embedded XML information explains what an object is and how to interact with it • SIMS style smart environment • Objects broadcast what they can do

22. Determining the Course of Action • Get input from the user’s body • Affective computing takes body measurements to determine your mental/ physical state • The context engine will be able to indicate when medical assistance is required • Expedite triage also • System knows not to deliver bad news when you’re angry/ tired/ morose/ etc. • System can tell you’re beginning to panic and “slaps you in the face”

23. Affective Computing Gear • This affective wearable includes a ProComp sensing system (upper left corner) surrounded by four sensors, clockwise from top: respiration, galvanic skin response (GSR) (center, left), blood volume pressure (BVP), and electromyogram (EMG). This unit attaches to a PC104 standard based computer (lower right) which receives data from a Twiddler hand-held keyboard (lower left). and displays data with the Private Eye (far right, below respiration sensor.) (J. Healey)

24. Determining the Course of Action • User can input commands which give the computer tasks to determine • The user’s commands are evaluated in light of the current goals • Goals are determined either explicitly or implicitly • Explicitly means that the user has previously defined this as a goal • Implicitly means the computer has learned that this goal is important

25. Determining the Course of Action • Database sharing • Gives one user the knowledge and insight of another user • Can be as simple as identifying someone • Can be as complex as planning an exercise • Will raise emphasis on training people's machines closer to the level we currently spend on training people • May be possible to train both simultaneously

26. Visual Display Aural Display Output to User Preferences Outputting Information to the User From Course of Action

27. Presenting Information to the User Interface must be: • Light • Small • Unobtrusive • Intuitive • Robust • Easy to use • Cheap • Easier said than done

28. Presenting Information to the User • “Fitting” the system to specific users • Everyone wants different information • User must find the given information useful • Don’t want the annoying paperclip scenario

30. Conclusion • Context Engine must be easy and natural to use • The problem has three major areas: • Recognizing the world • Determining the course of action • Presenting information to the user

31. NPS/MOVES Work Construct a virtual world and build a more complete and robust engine • Several options as to the input to the world: • Joint SIGINT Avionics Family (JSAF) Data • Plan on using agents: • To determine context from the symbolic representation • To determine recommended course of action

32. Potential Thesis Work • You can work in any part of the project discussed in this brief • First part will be extremely difficult • Second part will be difficult, but you’ll be working on a smaller part of the problem • Third part will be very interesting, and is a great opportunity for someone interested in human factors work

33. Questions? • Contact Information: • Perry McDowell • Lecturer of Computer Science • plmcdowe@nps.navy.mil • http://www.movesinstitute.org/~mcdowell