What to Consider When Developing and Evaluating Games for Training

1. What to Consider When Developing and Evaluating Games for Training Teresita Marie Sotomayor, PhDScience and Technology ManagerUS Army Research Laboratory Simulation and Training Technology Center

2. Outline • This presentation will explore: • Generational differences and the characteristics of the new generation of trainees • Features of games that motivate and engage individuals • An overview of the elements that need to be considered when evaluating the effectiveness of games in a training environment

3. Generational Differences • When developing games for training it is important to consider nested generational groups within the learning population. • Generation X • Prefer fast-paced presentation of information • Frequent feedback • Sense of accomplishment • Visual – grown up in the 3Dworld • Single channelers Training Environment • Generation Y • grown up with the internet at their fingertips • World to them is virtual • Possibilities are endless • Multi-channelers • Generation Z • Born with complete technology • Highly creative and collaborative • Multi-taskers • React fast

4. Why? • When designing tools for training it is important to understand targeted audience: • Characteristics • Learning Preferences • Cognitive Styles

5. Sample Population Profile

6. Sample Population Profile

7. Sample Population Profile

8. Sample Population Profile 93% Gender and Generation Categorization

9. Sample Population Profile Descriptive Characteristics of Subjects without Video Game Experience 93%

10. Sample Population Profile First Person Video Game and Computer Ability Self Assessment

11. Sample Population Profile Distribution of Video Game Hours on a Typical Week

12. Why Games Engage? • Games are a form of fun • Games are a form of play • Games have rules • Games have goals • Games have outcome and feedback

13. Why Games Engage? • Games are adaptive • Games have win states • Games have conflict/competition/challenge/opposition • Games have problem solving • Games are interactive • Games are immersive

14. Why Games Engage? • Games are Interactive • Interactivity in Games reinforces: • Individual Control • Trial-and-Error • Constant Change

15. Instructional Games • Instructional Games: • Designed or modified to meet training objectives • Includes rules, constraints, and activities that closely replicate the constraints of the real world tasks being trained • Must be incorporated in the POI in a manner that ensures that trainees understand the training objectives and receive feedback on performance against the training objectives

16. Instructional Games • Instructional Games: • Should be considered as a training aid tool • If developed as a stand-alone training tool, the game must be designed such that all instructional capabilities provided by the trainer are incorporated • Feedback should be incorporated as a structured guided activity such that learning can be fostered and the experience can be meaningful to training

17. Instructional Games • Four main aspects of a “serious game”: • Context of the game and usage is essential to the effectiveness of the game in the training environment • 2. Learner Specification – user profile to understand characteristics, learning preferences and cognitive styles of the targeted group of learners

18. Instructional Games • Four main aspects of a “serious game”: • 3. Representation– how levels of immersion, fidelity, and interactivity are integrated such that it is effective and provides the required level of engagement • 4. Pedagogical model – learning theory and approaches behind the application to ensure that it is used effectively to support specified learning

19. When Designing a “Serious Game” • USER Centric Approach • - Gap Analysis and Requirement Definition • Structured SME Discussions • Observation of training to ID objectives and gaps in training • Training Curriculum Review • Task Analysis • - Refinement • Usability Studies • - Assessment • Training Effectiveness Evaluation

20. Usability Studies The usability of an interactive system can be defined, in part, in terms of how easily a user can access and use the intended functionality of the system to meet task objectives.

21. Usability Studies • Interactive systems should be designed such that they are intuitive, effective, and subjectively acceptable to users. • Performing a usability study early on can provide valuable information for system development.

22. Usability Studies Early identification of user issues translates into reduced system redesign costs, enhanced user satisfaction and decreased user training time and the need for technical support.

23. Usability Studies • User Centric Methodology utilized to: • - Gather user (instructors and potential users) feedback regarding the current state of the technology being developed • - Explore the feasibility of incorporating innovative training tools in the current program of instruction

24. Methodology • Observation of trainees and trainers interacting with the system • Feedback on their reactions captured through surveys and participation in structured focus group interviews.

25. Methodology • Questionnaires are effective instruments to collect: • Demographic data • Feedback regarding their experience with the Simulation • Reaction to training

26. Methodology • Usability questionnaires are effective instruments to gather subjective data regarding user’s reaction: • Benefit to Training, System Usability, Presence and Realism, and Motivation to Use • Training Objectives, Knowledge and Skills, Overall Reaction to Training

27. Methodology • Focus groups are effective instruments to gather collective subjective data that can’t be collected via survey • Feedback about the current state of the prototype • Scenario/Environment, Techniques, User Interface, Training Objectives and After Action Review (AAR)

28. Training Effectiveness Evaluation • The main purpose of a training program is to impart the acquisition of knowledge, skills, and competencies. Evaluating a particular training is a challenging task

29. Training Effectiveness Evaluation • Developed by Donald Kirkpatrick in 1952 and published in 1959 in the Journal of the American Society of Training Directors • Provides a methodology for evaluating training programs • It is widely used because is simple and practical and can be applied to different situations (is flexible)

30. Kirkpatrick's Model

31. Level 1 Reaction • Measure of how participants feel about the various aspects of a training program. • Why should we measure reaction? • People must like a training program to obtain the most benefit. • To ensure participants are motivated and interested in learning.

32. Level 1 Reaction • Guidelines: - Determine what needs to be found out about the training - Develop Instruments: • Questionnaires, focus groups, researcher observations on participants reaction to the simulation - Need to ensure that reactions can be tabulated and quantified - Obtain honest reactions by making the sheets anonymous

33. Level 2 Learning • This is a measure of the knowledge acquired, skills improved, or attitude changed due to training. In our context we are aiming to improve knowledge of concepts, principles, or techniques. • Why? • Measure of reaction of trainees is not an indication that learning has taken place. - Will indicate what principles, facts, and techniques were understood and absorbed by trainees.

34. Level 2 Learning • Guidelines: - Use a Control Group - Develop instruments to evaluate knowledge before and after training: • Pre/post test, assessments, ratings of performance - Use 100 percent scale - Evaluate results statistically so that learning can be proven - Use the results of the evaluation to take appropriate action

35. Level 3 Behavior • This is a measure of the extent to which participants change their on-the-job behavior because of training. It is also referred as transfer of training. • Why? - There is a big difference between knowing principles and techniques and properly using them on the job.

36. Level 3 Behavior • Guidelines: - Use a Control Group - Allow enough time for change in behavior to take place - Develop quantitative instruments: • Scenario based simulations, self efficacy instruments, on-the-job observations - Survey or interview one or more of the following: trainees, instructors, superiors, etc - Use the results of the evaluation to take appropriate action

37. Level 4 Results • Measure of the long term results of the training program in terms of reduced costs, higher quality, higher productivity, lower rates of employees turnover or absenteeism, etc. • Why? • It can be used to support acquisition of a specific training device or to include methodology in POI. • Warning: Evaluations become more difficult, complicated and expensive. How much of the improvements shown over time are due to training and not other variables.

38. Level 4 Results • Guidelines: - Use a Control Group - Allow enough time for results to be achieved - May be obtained from unit performance reports, quality inspections and interviews with Senior people - Use the results of the evaluation to take appropriate action

39. ?? QUESTIONS ?? Teresita.sotomayor@us.army.mil