Model-Based Evaluation

1. Model-Based Evaluation INTRODUCTION What is Model-Based Evaluation? using a model to obtain predicted usability measures by calculation or simulation Why Use Model-Based Evaluation? Standard usability design process. user testing of prototypes a straightforward, well-documented methodology with a prove record of success Practical limitations of user testing too slow and expensive – tighten the iterative design loop Theoretical limitations of user testing lacks grounding in psychological theory only in the intuition of the designers

2. Model-Based Evaluation The model-based approach. to get some usability results before implementing a prototype or testing with human subjects iterative design is done first using the model, and then by UT What “interface engineering” should be. a deliberate attempt to develop and apply true engineering methods for UI design the rigorous and science-based techniques for how to evaluate UIs without user testing THREE CURRENT APPROACHES Task Network Models task performance in terms of a PERT-chart-like network of processes completion time as well as symbolic and numeric inputs/outputs

3. Model-Based Evaluation Cognitive Architecture Models a set of hypothetical interacting perceptual, cognitive, and motor components assumed to be present in the human a simulated human performing in a simulated task environment tend to be rather complex; limited experience in actual design setting GOMS Model Card, Moran, and Newell (1983) THEORETICAL BASIS FOR CHOOSING A MBE Psychological Constraints are Essential aroutine, production, or engineering activity not a scientific research the relevant psychological science must not only be developed first, but also must then be systematized and encapsulated in the modeling methodology itself constraints must provide the psychological validity of the model

4. Model-Based Evaluation A Brief History of Constraints in Modern Psychological Theory psychological theory in a form of info processing stages such as perception, memory, decision making, and action Human Operator Simulator (HOS) a set of micro-models for low-level perceptual, cognitive, and motor activities invoked by HORPROC Hick’s and Fitts’ law, formulae for visual recognition time, model of short-term memory retention, formulae for calculating various motor actions HOS was ahead of time because psychological theory was not well enough developed a task network version of HOS from Micro Analysis and Design, Inc., IPME, COGNET/IGEN Cognitive architecture – Anderson (1983) and Newell (1990) provides the coherent theoretical framework

5. Model-Based Evaluation Modeling Cognitive vs. Perceptual-Motor Aspects cognitive architecture lately incorporate perceptual and motor mechanisms – a misnomer modeling purely cognitive tasks is generally impractical modeling perceptual-motor activities is critical The Science Base Must be Visible it is critical that the psychological assumptions be accessible, justified, and intelligible any modeling system will have to be updated repeatedly as these theoretical and empiricalissues are thrashed out The Value of Generativity a single model can generate predicted human behavior for a whole class of scenarios especially powerful if runs as a simulation with (a) simulated device and (b) simulated human

6. Model-Based Evaluation The Role of Detail the usability lies in the details, not the generalities Cognitive Architecture are committed to detail. CAS dedicated to synthesizing and testing basic psychological theory naturally work at an extremely detailed level typically difficult to program and further currently unresolved psychological issues can become exposed to the modeler for resolution Task networks can be used before detailed design. TNM technique to assist in design stages before detailed design mission profile (what the system and its operators must accomplish in a typical mission)  functional analysis (functions that must be performed to accomplish the mission)  function allocation (which human operator or machine will perform each function)

7. Model-Based Evaluation TNM do not have any detailed mechanisms or constraints for representing human cognition and performance – lack of theoretical content in the modeling system itself the time distribution parameters are more problematic  subjective estimates of task time means and standard deviations and workload parameters PRACTICAL ISSUES Creating the Simulated Device a simulated human representing the user is interacting with a simulated device that represents the system under design can range from being a dummy device to a highly detailed simulation of the device interface and functionality the simulated device must be created before any prototype or final application requires constructing a simulated device from scratch

8. Model-Based Evaluation How to simplify the simulated device. an attractive simplification would be a programming language that is specialized for describing abstract device behavior in less than idealsituations a specialized language is unlikely to be an advantage – the time and effort necessary to learn a specialized language whose complexity is similar to general program language Identifying the Task Strategy A TA does not necessarily specify a task strategy. human performance in a task determined by (a) task requirements by TA, (b) CA, and (c) a specific strategy normal TA not necessarily identify the exact sequence of actions to perform and rarely specify the timing of actions Difficulties in identifying task strategy. task strategy chosen in the absence of data to fit a model to important to identify the best but is often not obvious, even to expert

9. Model-Based Evaluation A heuristic: Model what users should do. how do we find out what strategies users will follow in using the system under development?  the most effective approach is to assume the design will be used as intended perform at the highest possible level or at some adequate level? bracketing heuristic (Kieras and Meyer, 2000) – slowest-reasonable model and fastest-possible model Concerns over Model Validity Can you believe the model? Suppose a model implies critical design choices. Should you follow? A poor response is to build and test prototype just as if no modeling has been done because the purpose of MBE is to reduce the amount of prototyping and UT required to refine a design A better response is to understand how the model implies the design choices by answering the question.

10. Model-Based Evaluation “What aspects of the model are contributing to outcome?” critical aspects valid and properly represented – the model accepted Should you validate the model? the model can be validated after use by comparing the final user test results to the model predictions should special data to validate the model be collected priori to using it to guide the design?  No, data collection not during design Summary: Assessing model validity. instead of validating the model, assess its validity in terms of whether it meets the following basic requirements: the model strategy based on an analysis of what users should do? can or will users follow the same strategy as the model? are the assumptions about human abilities plausible?