What Can Massively Multiplayer Online Games (MMOGs) Train?

1. What Can Massively Multiplayer Online Games (MMOGs) Train? Harry O’Neil and Richard Wainess University of Southern California/CRESST DoD Training Transformation Technologies Developing, Distributing, and Assessing Joint KnowledgeSeptember 4, 2003Alexandria, VA DOD NTSA TrainTransTech v.6 9/04/03

2. CRESST Model of Learning Content Understanding Collaboration Learning Problem Solving Communication Self-Regulation

3. Typical Measures Non-existent Unvalidated Subject matter Format as templates Start anew Cross sectional Desired Measures Validated Cognition/transfer Researched models Reusable objects Embedded simulations Trajectories A Focus on Learning RECIPE FOR DISASTER: Collect everything and figure it out later

4. Collaborative Problem Solving in MMOGs GroupTeamwork Process Content Understanding Problem-Solving Strategies Self-Regulation 1. Adaptability 2. Coordination 3. Decision Making 4. Interpersonal 5. Leadership 6. Communication Facts Concepts Procedures Principles 1. Planning 2. Self-Checking 3. Effort 4. Self-Efficacy Domain Specific

5. Taxonomy of Teamwork Processes • Adaptability • Recognizing problems and responding appropriately • Coordination • Organizing team activities to complete a task on time • Decision Making • Using available information • Interpersonal • Interacting cooperatively with other team members • Leadership • Providing direction for the team • Communication • Clear and accurate exchange of information

6. Social Capital Trust Teamwork Skills MMOGOrganization Networks Efficacy Transparency Effort

7. Do Games Train? • The research indicates that computer games were potentially useful for instructional purposes and were hypothesized to provide multiple benefits • Promotion of motivation • Improvement of skills • Facilitation of metacognition • Limited empirical research conducted on games topic (17 studies, 1990–2003) • PsycINFO, Education Abs, SocSciAbs, • Adults, empirical (qualitative/quantitative) • Only one relevant abstract on massive multiplayer games

8. Different Mental Models 1Content understanding, problem-solving, self-regulation, communication, team skills. 2Action, role planning, adventure, strategy games, goal games, team sports, individual sports (Laird & VanLent, 2001). 3Implicit vs. Explicit: During or after (AAR).

9. Integrate into Plan of Instruction Instructor training Doctrinal accuracy Limited role for fantasy Integrate into recreation/family time Increased requirement for feedback, automated After-Action Reviews Student “training” need (e.g., self-regulation) Flexible Doctrine/Strategy/Tactics Fantasy role On-Duty vs. Off-Duty Trade-Offs If on-duty in military school If off-duty

10. Motivation provided by instructor Game can be less fun Good application Initial acquisition training Available time in hours/days/weeks Motivation provided by student/game Game needs to be fun Good application Refresher training; prevent skill decay Available time in minutes/hours On-Duty vs. Off-Duty Trade-Offs(cont.) If on-duty in military school If off-duty

11. Can MMOGs Train? • What to train? • Collaborative problem solving • Is there evidence? • There is little empirical work on effectiveness of games for adult training • Opportunities for an R&D agenda

12. Back-up

13. The Specification of What We Are Teaching Is Essential • From goal/objective, the instructional strategies follow (e.g., nature of feedback, timing of feedback, etc.) and assessment issues (e.g., how do we know that soldiers have more of X after playing the game) • Different Xs (e.g., leadership, battle command, situational awareness, decision making, tactical problem solving) require different measures, After-Action Reviews, take home packages, instructor training, homework assignments, etc.

14. Game: Abstract Search Process • Article criteria: video/computer game, training, adults • Databases: PsycINFO, Education Abs, SocSciAbs • Date limits: 1990-2003 • 17 relevant empirical abstracts • Articles being retrieved—there will be fewer relevant articles

15. Game: Abstract Search 1Used in training and adults. 2Either qualitative or quantitative information.

16. Multiplayer Search Process • Article criteria: multiplayer or massively multiplayer video/computer game, training, adults • Databases: PsycINFO, Education Abs, SocSciAbs • Date limits: 1990-2003 • 1 relevant empirical abstract • Article being retrieved

17. Multiplayer Abstract Search Training potential of multiplayer and massively multiplayer video games for adults 1Use of training and adults. 2Either qualitative or quantitative information.

18. Measurement Opportunities

19. Measurement Opportunities (cont.)

20. Measurement Opportunities (cont.)

21. Measurement Opportunities (cont.)

22. Formative Evaluation Activity • 1. Check the game design against its specifications • e.g., setting, tactics • Attend critical designer reviews • 2. Check the validity of instructional strategies embedded in the system against research literature • Literature reviews1,2,3 • Implementation of practice and feedback (e.g., After-Action Review) • 3. Conduct feasibility review with the instructors • Are right tasks being trained? • Review to be conducted with SMEs • 4. Conduct feasibility test with soldiers • One-on-one testing • Small-group testing

23. Formative Evaluation Activity(cont.) • 5. Does more game-based training lead to better game performance (e.g., loss ratios: losses/kills)? • – Need to track a soldier’s performance across multiple games • 6. Assess instructional effectiveness • Cognitive • e.g., does it improve domain knowledge, problem-solving skills, self-regulation, tactical skills, situational awareness? • Affective • e.g., does it change self-efficacy? • 7. Do experts and novices differ? • 8. Assess unanticipated outcomes • 9. Implement revisions