GERONTECHNOLOGY Evaluation methods Master Class 2007

1. GERONTECHNOLOGYEvaluation methodsMasterClass2007 Prof. dr. D. G. Bouwhuis Cognitive Ergonomics Technische Universiteit Eindhoven TU/e

2. Aggregation levels of Human Behaviour • Physiology: size, weight, strength, level of development/maturation/aging continuity, homeostasis 2. Behavioural: actions, perception 3. Cognitive: thinking, reasoning, deciding, understanding, planning 4. Social: communication, transactions, care, cooperation 5. Professional: productivity, behavioural economy, execution, control 6. Cultural: education, social environment, physical environment

3. Aggregation levels of Human Behaviour • Human behaviour always proceeds on a number of these layers at the same time, • this requires a multidisciplinary approach: every personal situation or state takes place on a number of levels. • Every behaviour has a physiological basis, it mostly has cognitive components, it is always action-related, it may have social components, and is certainly culturally influenced.

4. types of behaviour Shneiderman’s classes of users(1992) • Novice users • Knowledgeable, intermittent users • Expert, frequent users this distinction is a useful one if one wants to explore new types of interfaces with relevant users

5. types of behaviour Older users Older users are much more critical concerning experimental situations than younger subjects. They may refuse to participate, or are not interested. Or indignant. there are four types of older users: • pioneers; routine users, eager teachers • beginning users; showing enthusiasm • aspiring users; worried about effort / learning • refusers; unwilling to participate at all. Probably the largest group

6. Why usage research? • Technology push does not always suit the actual or the desired usage of the technology • The user is not the designer • If a user cannot use a product “it does not work!” • Users do things you never expect

7. Problem Solving Human Behaviour theory of problem solving • a problem starts in the ‘initial state’ that has to be changed according to the goal • the problem solver chooses an action that will change the initial state • there follows an evaluation of the new state, and how far it is removed from the desired final state • further steps are made until the goal is attained, which is the ‘final state’.

8. Problem Solving Human Behaviour theory of problem solving • example: want to know the time initial state action step next state time unkown look at watch watch broken time unknown switch on TV time unknown set to channel 9 time unknown look up time bottom left>time known goal satisfied final state

9. Problem Solving Human Behaviour theory of problem solving • In every task the user is bringing the initial state of the product, or the environment that the product controls, to a desired state: the final state. • There is a ‘problem solving situation’ when the outcome of a state change is uncertain, or unknown. • Problem solving is made more difficult when the action steps are unknown, e.g. in unfamiliar products

10. Problem Solving Human Behaviour theory of problem solving Norman’s seven stages model (1986) is an instantiated model of problem solving for user-product interaction. 1. Establish the Goal 2. Forming the Intention Intention 3. Specifying the Action Sequence Selection 4. Executing the Action Execution 5. Perceiving the System State 6. Interpreting the System State 7. Evaluating the System State Evaluation

11. Problem Solving Human Behaviour theory of problem solving Norman’s seven stages model (1986) is an instantiated model of problem solving for user-product interaction. 1. Establish the Goal 2. Forming the Intention Intention 3. Specifying the Action Sequence Selection 4. Executing the Action Execution 5. Perceiving the System State 6. Interpreting the System State 7. Evaluating the System State Evaluation

12. Measurement of Human Behaviour • Speed, (latency, reaction times, processing time) • Accuracy (probability correct) • Subjective measures (ratings, attitudes) • Acceptance (choice, usage frequency) • Physiological measures (heart rate, Galvanic Skin Response…) • Behavioural measures (posture, pointing, facial expression..) • Neural imaging (fMRI, CT scan, EEG)

13. Measurement of Human Behaviour • Speed, (latency, reaction times, processing time) Except when task execution or user control takes excessively long, speed measures are less relevant in gerontechnology, as the result, and the benefit of the result is valued much higher than the time it takes to attain it.

14. Measurement of Human Behaviour • Accuracy (probability correct) easy to observe, can provide useful diagnostics

15. Measurement of Human Behaviour • Subjective measures (ratings, attitudes) • easy to measure, but can be misleading, respondents have tendency to socially desirable • reliability diminishes when question is general and not specific about the exactfunctionality or product

16. Measurement of Human Behaviour • Acceptance (choice, usage frequency) usage frequency is one of the most reliable measures of acceptance and usability

17. Measurement of Human Behaviour 5. Physiological measures (Heart Rate, Heart Rate Variability, Blood Pressure, Galvanic Skin Response…) popular with the engineer, but less useful as • valid and precise measurement is difficult, • requires much pre-processing and post-processing • and may create endless difficulties with medical ethics committees.

18. Measurement of Human Behaviour 6. Behavioural measures (posture, pointing, facial expression..) Mostly easy to observe, but not always reliable, complaints, statements of discomfort, refusals are mostly useful indicators of rejection.

19. Measurement of Human Behaviour 7. Neural imaging (fMRI, CT scan, EEG) • very expensive, or laborious. • requires extensive interpretation • not suitable for measurement in representative situations

20. Measurement of Usability Nielsen’s usability measures • Learnability • Efficiency • Memorability • Errors • Satisfaction Nielsen, J. (1993) Usability Engineering, Cambridge (MA) Academic Press Professional

21. Measurement of Usability ISO Draft International Standard 9241 The effectiveness, efficiency, and satisfaction with which specified users achieve specified goals in particular environments. • Effectiveness • Efficiency • Satisfaction standardized • Memorability in preparation • Learnability

22. Measurement of Usability ISO Draft International Standard 9241 • Effectiveness The accuracy and completeness with which specified users can achieve specified goals in particular environments.

23. Measurement of Usability ISO Draft International Standard 9241 • Efficiency The resources expended in relation to the accuracy and completeness of goals achieved.

24. Measurement of Usability ISO Draft International Standard 9241 • Satisfaction The comfort and acceptability of the work system to its users and other people affected by its use.

26. measurement of experience a gerontechnological perspective • effectiveness the intended, and the perceived effect of the technical provision should be clearly apparent such that it leads to comfort (8); technical effectiveness is not enough!

27. measurement of experience a gerontechnological perspective 2. efficiency the perceived effort in control should be as low as possible: • there should be a high level of self-efficacy • cost of access should be as low as possible

28. measurement of experience a gerontechnological perspective 3. satisfaction encompasses three issues: • general satisfaction with the installed technical provision • satisfaction about the control of the provision • attaining a satisfactory effect by the provision

29. measurement of experience a gerontechnological perspective 4. learnability control of the technical provision should be easy to learn, (possibly no learning at all) and there should be knowledge about the operation and effect of the provision (error reduction after repeated use, increase in speed of operation)

30. measurement of experience a gerontechnological perspective 5. memorability after a period of non-usage, control of the technical provision should be easily and successfully resumed. (within an acceptable time, without errors)

31. measurement of experience a gerontechnological perspective 6. fun if the technical provision presents a dynamic programme, e.g. TV, it should be sufficiently entertaining and contribute to well-being

32. measurement of experience a gerontechnological perspective 7. comfort should be seen as a psychological variable, not a technical one. It is more than satisfaction as it should increase well-being, rather than stabilizing it.

33. measurement of experience a gerontechnological perspective 7. comfort should be seen as a psychological variable, not a technical one. It is more than satisfaction as it should increase well-being, rather than stabilizing it.

34. New technology

35. VieDome Heeze: telemedicine DBV picture

36. Research rationale • Questions: • Which user aspects are critical in the acceptance of telemedicine applications? • How can user experiences of these telemedicine applications be measured?

37. research rationale • Field study • Qualitative measurement in the field: triangulation method: (1. questionnaire, 2. interview, 3. observation) • Controlled laboratory study • Measurement and optimization ease of us. Quantitative measurement in the lab: efficiency & effectiveness(time and errors)

38. Results: factors that hardly affect acceptance • Perceived usefulness: dependent on individual health properties Most participants state that telemedicine is useful • Social Ifluence: only affects the choice to participate in the study. Does not determine actual use. Influence of care providers, doctor, nurses, children

39. Results: factors that do affect acceptance • Installation & Support: • bad feelings if installation takes weeks • clear explanation by nurses • wish to look up later the instructions (memory problems) • hardly any feedback in the user interface • not clear whom to address with interface problems, technical problems or health problems (often diabetes nurse)

40. Results: factors that do affect acceptance • Attitude • negative attitude (application hardly used) is reinforced by problems at installation, operation and interface • positive attitude (application used) reduced by the same problems • Ease of use: • results in line with labexperiment

41. the redesign • adaptation of the screen interface for glucose registration Origineel Herontwerp

42. labstudy 2nd part • Averaged over all tasks users make fewer errors with the new interface • Repeated tasks are executed faster with the new interface

43. Technology Acceptance Model (TAM; Davis) perceived usefulness resourcefulness, mobility financial, fun individual properties: limitations by illness or age user acceptance attitude: distrust, resistance, openness actual use perceived ease of use support: installation, explanation, operation, feedback data, video consult social influence: care providers, friends, family