Towards improved user-product testing with cognitively enhanced scenarios

1. Towards improved user-product testing with cognitively enhanced scenarios Wilfred van der Vegte

2. Where I work Delft University of Technology (2011 figures) • Founded in 1842 • 17,250 students • 12 undergraduate programmes • 33 post-graduate programmes • 2,540 academic staff • 8 faculties (departments) Faculty ofIndustrial Design Engineering Founded in 1969 Largest university-based design* school worldwide(*product design, consumer durables) 1,980 students >4,700 alumni (MSc level) 3 departments,94 scientific staff

3. Where I work Section of Computer-Aided Design Engineering • ICT in product design • Historically: CAx • Currently: Cyber-physical systems, ubiquitous computing, smart products:how to apply technology in products & in product design • Headed by Prof. dr. mult. Imre Horváth • ~20 people, ~10 permanent scientificstaff

4. Towards improved user-product testing with cognitively enhanced scenarios Wilfred van der Vegte

5. Typical engineering simulations

6. Involving humans in product simulations Real humans (interactive simulation) Virtual humans (computer models) human model perception Does not require human subjects decision-making (brain) ? metabolism motor control(brain, CNS) action (muscles) human model computer simulation computer simulation product model product model

7. Simulating use with virtual humans virtual human perception do we have to simulate all of this? decision-making (brain) scenario bundle (by designer) metabolism motor control(brain, CNS) approximate action (muscles) No ...let’s simplify approximate already solvedby others

8. Designer’s conjecture of human actions /decision-making Graphical notation Possible paths through use processconnecting interactions Contains multiple options(courses of use process) Controls a physics simulation of the product Is specified as an automaton(←in this case a statechart) The scenario bundle

9. Example: snack dispenser scenario bundle programming of product simulation

10. Advantages • Use process of the product can be tested if only a computer model is available • No human subjects needed • Designer can do what-if studies • One scenario, multiple varieties of the product design • Varieties of how the product is programmed • Varieties in human behaviour, e.g., hesitation • Varieties of scenario: rearrange blocks & arrows • Building blocks of scenarios are reusable in other design projects

11. Key limitations • Simulation of physics computationally demanding and not (yet) very stable • Algorithms for realistic simulation of low-level human motor control exist, but have not been included • Lack of realism in human information processing (thinking, reasoning, ...): only what the designer preconceives is included But ... Are animations of physics and human motor behavior always needed? No! Often only the course of values of specific variables over time is important!(power consumption, temperature, speed, ...)

12. Demonstrative example: espresso machine with power-save function for water heating • Investigation of • boiler temperature • energy consumption • over a typical use process in which several coffee-making sessions take place, over several hours / days / weeks /... • with given settings for power saving (reduce power by x% after non-use over Δt = t1and by 100% after another Δt = t2)

13. Building a scenario bundle for using the espresso machine

14. Building a scenario bundle for using the espresso machine

15. Building a scenario bundle for using the espresso machine

16. Building a scenario bundle for using the espresso machine

17. Building a scenario bundle for using the espresso machine User

18. To compensate for the missing 3D physics simulation, we have now used a hybrid automaton (discrete+continuous) Models_and_specifications Boiler User User_cupboard Compute_input_time Pump 2 Espresso_machine Supplier In addition, the statechart has become a timed hybrid automaton (THA) to efficiently deal with timing (latency, delays, time-outs,...) Detect_power_change no_heating_or_cooling Supplier_processing Espresso_machine_logic OFF/discharge=0,P_pump=0,empty_sound=0 /T_water=T_amb store_previous_value [P_change>0]/t1=t Coffee_serving Boiler_thermostat Track_consumption_remotely T_last=T_amb /P_prev=P_heat [pump==0] [pump==1] heating_or_cooling P_change=0 Prepare_orders during:input_time=t-t1 [hasChanged(P_heat)==1] Espresso_machine_physics ON/P_pump=59 exit:input_time=0 Shipping Pump Reservoir compute_P_change 1 detect_capsule /P_change=P_heat-P_prev [capsule_inserted==1] Courier_service Boiler Cup_filling present absent [P_change<0]/t1=t T_last=T_water [capsule_inserted==0] /flow=4.29 /flow=5.86 [P_change>0]/t1=t User Compute_T_water [capsule_inserted==0] Run 2 build_up_pressure/ deliver /discharge=flow Exponential after(2.5,sec) during: T_water=T_amb+P_heat*k_1... run_dry/discharge=0, -(T_amb+P_heat*k_1-T_last)*ml.exp(-(k_3/m_water)*input_time) [reservoir_content<5] empty_sound=1 T_water = T_amb + P_heat*k1 - k2*exp[-(k3/m_water)*t]; k_2 = 1-(T_amb_P_heat*k_1-T_last) [P_heat==0&&... Constant_and_cold T_water-T_amb<0.5] [P_heat>0] /T_water=T_amb

19. User

21. Energy consumption ( kWh ) Reservoir content ( ml ) Serving User takes break These simulation outcomes can be generated up to 5000 faster than real time espressos Serving lungos User takes break User refills reservoir Boiler water temperature ( K ) Thermostat controls Power - save mode temperature Thermostat controls temperature Power - save mode User switches machine off Time ( s )

22. Revisiting the limitations • Simulation of physics has been simplified (no more 3D), and is fast and reliable • Low-level human motor control is disregarded, still the whole use process can be simulated (in this case) • Still lack of realism in human information processing (thinking, reasoning, ...): only what the designer preconceives is included

23. So ... how to increase realism in human information processing (thinking, reasoning, ...)? • Aspects of human information processing to be simulated • Logic of decision making:under which condition what action is taken?e.g. “IF cup is full THEN retrieve cup from machine”:straightforward execution of ‘normal’ use, assuming a particular history of preceding events. • But can a simulation predict a user acting according tothe production rule“IF cup is full THEN stick finger in it”? →unlikely!

24. So ... how to increase realism in human information processing (thinking, reasoning, ...)? • Aspects of human information processing to be simulated • Logic of decision making:under which condition what action is taken?e.g. “IF cup is full THEN retrieve cup from machine”:straightforward execution of ‘normal’ use (instructions), assuming a particular history of events, including required preceding actions. • We can howevertry to generatetypical aberrationsfrom‘regular use’:the so-callederror phenotypes • taxonomy accordingto Hollnagel→ error mode simple phenotype complex phenotype ( applies to one action ) ( applies to multiple interconnected actions ) repetition restart action in wrong place reversal jumping omission undershoot action at wrong time delay premature action action of wrong type replacement insertion sidetracking action not included in capture current plans intrusion branching overshoot

25. So ... how to increase realism in human information processing (thinking, reasoning, ...)? • Aspects of human information processing to be simulated • Processing time:How long does it take to accomplisha given action, taking into accountaspectssuch as memory retrieval,memory capacity, learning,multitasking, distraction, etc. • These aspects can be simulatedusing cognitive architecturessuch as ACT-R • A cognitive architecture is • a blueprint of the human mind • based on findings from brain science • filled with psychologically validatedtask models expressed as production rules intentional module ( not identified ) declarative module goal buffer ( temporal cortex / ( dorsolateral hippocampus ) prefrontal cortex ) retrieval buffer central production ( ventrolateral system prefrontal cortex ) ( basal ganglia ) visual motor buffer buffer ( parietal ( motor cortex ) cortex ) motor visual module module ( motor ( occipital cortex / cortex ) cerebel - lum ) external world

26. How to realize simulations with cognitively enhanced scenarios (CES) intentional module e l u motor motor d d l declarative module goal buffer r o o buffer module central m w l goal produc - l a a n n buffer tion r o central production i e t t retrieval buffer system visual visual n x system e e t buffer module n i motor visual buffer buffer declar - re - ative trieval visual motor module buffer module module external world

27. How to realize simulations with cognitively enhanced scenarios (CES) error mode simple phenotype complex phenotype applies to multiple ( applies to one action ) ( interconnected actions ) repetition restart Human-ErrorPhenotype Generator action in wrong place reversal jumping omission undershoot action at wrong time delay premature action action of wrong type replacement insertion sidetracking action not included in capture current plans intrusion branching e l u motor motor d overshoot d l r o o buffer module central m w l goal produc - l a a n n buffer tion r o i e t t system visual visual n x e e t buffer module n i declar - re - ative trieval module buffer Scenario bundle (THA) (human actions) THA of the ‘world’(product, environment) Cognitive Architecture (ACT-R)

28. Operation of ship locks –our first real-life application? usually combined with a movable bridge various types of boats and skippers operator works remotely, usingmultiple monitors gates, traffic lights, leveling, etc., are all man-controlled usually 2-3 chambers

29. Operation of ship locks –our first real-life application? The Dutch Government Agency of Public Works & Water Management is developing a new system of centralized control rooms for locks, movable bridges and other objects • Potential problems: • Multitasking, but also boredom, can cause cognitive overload, cognitive lock-up& other error-invoking mental phenomena • Errors can cause injury, collisions or even flooding • With CES simulations the system can be pre-tested • without human subjects • batchwise, systematically varying parameters (weather, traffic density, ...) • to reveal incidents theoretically happening once in 10/100/1,000 years • that cannot be discovered through interactive testing

30. Envisioned setup for first application error mode simple phenotype complex phenotype applies to multiple ( applies to one action ) ( interconnected actions ) repetition restart Human-ErrorPhenotype Generator action in wrong place reversal jumping omission undershoot action at wrong time delay premature action action of wrong type replacement insertion sidetracking action not included in capture current plans intrusion branching motor motor overshoot buffer module central goal produc - buffer tion system visual visual buffer module declar - re - ative trieval module buffer Scenario bundle (THA) (human actions) THA of the ‘world’(product, environment) Cognitive Architecture (ACT-R)

31. Conclusions Simulation with cognitively enhanced scenarios (CES) • can help evaluating human-system interactions where many variations can influence the outcomes • based on psychologically validated knowledge on the workings of the human brain • benefits from simplifications & shortcuts:simulation speed is determined by slowest element in simulation loop →avoid complex models & algorithms