End-to-End Application Considerations

1. End-to-End Application Considerations Daniel P. Siewiorek Carnegie Mellon University February 2013

2. Outline • Introduction • Individual • Wearable Computers • Augmented Reality • Infrastructure • Museums • Hospital • e-Display • Ad Hoc • Wildfire • Volcano • Concluding Lessons

3. Outline • Introduction • Individual • Wearable Computers • Augmented Reality • Infrastructure • Museums • Hospital • e-Display • Ad Hoc • Wildfire • Volcano • Concluding Lessons

6. Wearable Applications and Architecture • Procedures - upload at completion • Work Orders - incremental updates • Collaboration - real time interaction • Client-Server • Thin Client Legacy Systems • Interactive Electronic Technical Manuals (IETMs)

7. Time Rate of Change of Data Taxonomy • Procedures. Maintenance and plant operation applications are characterized by a large volume of information that varies slowly over time. • A typical request consists of approximately ten pages of text and schematic drawings. Changes to the centralized information base can occur on a weekly basis.

8. Savings Using Tactical Information Assistants in Marine Heavy Vehicle Maintenance SAVINGS FACTOR VuMan 3 Field Trials Current Practice Personnel 2:1 SAVINGS FACTOR Current Practice VuMan 3 Field Trials Inspection time 40% less

9. Time Rate of Change of Data Taxonomy (continued) • Work Orders. The trend is towards more customization in systems. • Manufacturing or maintenance personnel receive a job list that describes the tasks and includes text and schematic documentation. This information can change on a daily or even hourly basis.

10. Time Rate of Change of Data Taxonomy (continued) • Collaboration. An individual often requires assistance. In a “Help Desk” an experienced person is contacted for audio and visual assistance. The Help Desk can service many people simultaneously. • Information can change on a minute-by-minute and sometimes even a second-by-second basis.

11. C-130 Help Desk

12. Lessons Learned From Usage • Maximum Size, weight, energy consumption before change user behavior • No fixed relationship between input/output/display • User less patient, expect instant response • Intuitive to use, no user’s manual • Information overload, user may focus on computer rather than physical world • Potential to lose initiative, user does exactly what the computer tells them to do

13. Based on a LectureDan Siewiorek, Thad Starner, Asim SmailagicMorgan & Claypool • U - User • C - Corporal • Interface physically without discomfort or distraction • A - Attention • Divided between physical and virtual world • M - Manipulation • Controls quick to find, easy to operate • P - Power

14. Major Factors in Portable Electronic Systems and Their Relationship to Design Disciplines Power Corporal Attention Manipulation User

15. UCAMP User

16. Sensor-based models of human activity and situation • Example: • Using just the sensors in a laptop we have predictive model that determines “now is a bad time to interrupt”more accurately than human observers Better than human observers with just sensors already in laptop! Researchers Interns All Subjects Researchers Interns Researchers Managers Managers Managers 76.9% HumanObservers Full Sensor Sets No Camerasor Microphones Only aLaptop

17. Cognitive Load

18. fMRI Experiment Configuration

19. JUL 31, 2001 Car Calls May Leave Brain Short-Handed By SANDRA BLAKESLEE Scientists have bad news for people who think they can deftly drive a car while gabbing on a cell phone. The first study using magnetic resonance images of brain activity to compare what happens in people's heads when they do one complex task, as opposed to two tasks at a time, reveals a disquieting fact: the brain appears to have a finite amount of space for tasks requiring attention. The study, published in the Aug. 1, 2001 issue of the journal NeuroImage, was led by Dr. Marcel Just, a psychology professor and co- director of the Center for Cognitive Brain Imaging at Carnegie Mellon University in Pittsburgh...

20. Capacity constraint in association areas: Activation volume is less in dual task compared to single tasks, even for tasks without cortical overlap (auditory comprehension and mental rotation)

21. Assessing measures of cognitive load while performing tasks in divided attention • Cognitive load can be a measure for whether, when and how to present information while performing a number of different tasks • Need to manage human attention and understand when users can attend to new information • Necessary while driving, learning new skills • Particularly an issue with elders

22. Assessing measures of cognitive load • We are working to develop a reliable, real-time and objective measure of cognitive load for contexts of divided attention • Current techniques not appropriate • Post-hoc, so not real time • Not accurate • Subjective measures

23. Study of cognitive load • Six ECTs: visual perception and cognitive speed • Four sensors • Gaze tracking

24. Assessing Cognitive Load • 20 young adults • Compared sensors for determining cognitive load • Best feature: > 74% • Sensors/features: median heat flux, ECG • > 81% across all participants • Older adults • Same sensors useful, although heat flux also seems important • BodyMedia device seems to be sufficient to capture cognitive load with elementary tasks • Need to see how this transfers to real settings • Then, use information about cognitive load in virtual coaches

25. User Lessons • “The user is not I” • Built two dozen systems to identify repeated patterns (e.g. procedures, work orders, team collaboration) • Mobile Users are more impatient than desktop users • Must operate more like a flashlight than a computer rebooting • Institutional Review Board • “But madam, they are Marines and they do what they are told” • Be prepared to reduce functionality • “It’s a fine goal” • Instant Creditability • “I served with you on the Vinson’s first world cruise”

26. UCAMP Corporal

27. “If it looks good,it will fly well” • MoCCA and VuMan 3 received the prestigious Industrial Design Excellence Awards (IDEA) from award co-sponsors Business Week magazine and the Industrial Designers Society of America (IDSA). VuMan 3 MoCCA

28. NATO Soldier Using CMU TIA-P Language Translator in Bosnia

29. Corporal Lessons • Air Logistics Command, Sacramento CA • The vapor barrier on a hot August day • Fort Gorden Battle Command • “Does this bend?!” • Six units to Bosnia ….. none returned

30. UCAMP Attention

31. But, I’ve Always Done it This Way! On A KC-135 Aircraft was being pressurized at ground level. The outflow valves which are used to regulate the pressure of the aircraft were capped off during a 5 year overhaul and never opened back up. The post-investigation revealed: that a civilian depot technician who, "had always done it that way," was using a homemade gauge, and no procedure. The technician's gauge didn't even have a max "peg" for the needle and so it was no surprise he missed it when the needle went around the gauge the first time. As the technician continued to pressurize the aircraft, and as the needle was on its second trip around the gauge the aircraft went "boom" - the rear hatch was blown over 70 yards away, behind a blast fence!

32. KC-135 Pressure Test Results

33. Attention Lessons • The more information and more engaging the virtual world, the less interaction with the physical world • Use systems for reference (e.g. information dashboard) then focus on physical world

34. UCAMP Manipulation

35. Selection of “hot links” with CMU’s Wheel/Pointer

36. Manipulation Lessons • Strong user interaction mental model • Not only trained in the field in less than ten minutes but also able to train the next user • No users manual • Training without access to what the user is seeing • Without stationary reference, users easily become disoriented and confused with traditional input devices • “Which is the left button” • Control devices have to be flexible to mount on different parts of the body • Boeing – shoulder holster • Sailors on aircraft carrier – shoulders used to carry tie-down chains

37. UCAMP Power

38. Power Lessons • It’s the peak, not the average • Muffled pops • Warning labels are no substitute for good design • More to be gained on reducing demand than increasing source • Current batteries have half the energy density of dynamite • Current user interactions demand all resources to reduce response time • Land Warrior • Army standard operating system • Nine different battery types

39. Speech Enabled Augmented Reality for Maintenance [Goose] • SEAR – Speech Enabled Augmented Reality • 3 D Augmented Reality graphical view with location-sensitive 3 D speech-driven interface • 3 D component specific vocabulary triggered by proximity sensors • Display component specific commands when in vicinity • Key word activated • Simultaneous 3 D “parameterized” synthesized speech streams • For current values from physical plant • Coded visual markers for tracking, location • 4 X 4 matrix with dots for over 10,000 combinations

40. Room “augmented” with the CAD drawing of a floorplan Augmented Reality - Maintenance of Nuclear Power Plant Components

41. Speech Enabled Augmented Reality for Maintenance [Goose] • System Architecture and User Interface

42. Outline • Introduction • Individual • Wearable Computers • Augmented Reality • Infrastructure • Museums • Hospital • e-Display • Ad Hoc • Wildfire • Volcano • Concluding Lessons

43. Interactive Museums [Fleck] • San Francisco Exploratorium • Several hundred interactive exhibits • Frequently rotated off the floor • Prior work • PDA, Acoustic Guides, IR beacons for pointing

44. Interactive Museums - Services • Informer – detailed information on exhibit • Suggester – what to try • Remember – build record of experiences, selects what to record • Guider – suggests an order of exhibits* • Communicator – instant messaging, leave notes at exhibits* * Not implemented

45. Interactive Museums – Lessons Learned • Overall positive responses • Not enough hands to hold PDA and operate exhibit interfering with exploration • Undesirable demand on user attention • Lost in hyper reality – focus on device rather than exhibit • Wow factor – part of positive feedback • Beacons OK but sometimes picked up other beacons • Browser interface – people unfamiliar with stylus • Content design – did not know what was clickable • Forgetting to use Remember – application complexity

46. Pervasive Technologies in a Hospital [Hansen] • Hospital scheduling, coordination, and awareness system • Location tracking, context awareness, large interactive displays, mobile phones • Awareness Media • Status of work in Operating Rooms • Video stream for accessing current state • Progress bar • Chat area • Schedule • Location tracking of who is in Operating Rooms

47. Pervasive Technologies in a Hospital - Scenario • Acute patient • Find/schedule Operating Room • Find surgeon not in Operating Room • Send message • Notify patient ward • Scheduled patient notified of their surgery postponement

48. Pervasive Technologies in a Hospital – Hardware Lessons

49. Pervasive Technologies in a Hospital – Software Lessons

50. Pervasive Technologies in a Hospital – User Lessons