Jenn McNamara - Using Games for Assessment: Why? How? And Examples

1. Using Games for Assessment: Why? How? And Examples Jenn McNamara BreakAway Games Serious Play 2016 Chapel Hill, NC

2. Before we start we must remember… Games aren’t real – – images from “If games were real” cracked.com Before we start we must remember… Games aren’t real images from “If games were real” cracked.com

3. Topics Topics ■ A little bit about assessment ■ A little bit about games’ qualities to support assessment ■ Examples demonstrating their intersection Assessment Games

4. Assessments Serve To… Assessments Serve To… ■ Measure outcomes or knowledge levels  Attainment of specific levels of proficiency ■ Provide insight into the learning / thought processes ■ Provide immediate feedback  Specific gaps  Potential areas or direction for further instruction ■ Provide a record of behavior ■ Certify proficiency

5. Types of Assessment Types of Assessment ■ Formative Assessment employed by teachers during the learning process in order to modify teaching and learning activities to improve student attainment. Typically qualitative feedback that focuses on the details of content and performance. ■ Summative Assessment for purposes of external accountability. ■ Standardized Assessment scoring procedures, and interpretations are consistent and are administered and scored in a predetermined, standard manner. Formative Assessment: formal and informal assessment procedures Summative Assessment: seeks to monitor educational outcomes, often Standardized Assessment: questions, conditions for administering, Wikipedia

6. Engagement Engagement Fundamental components of the gameplay experience: Analysing immersion Ermi & Mayra

7. Support for Games for Learning Support for Games for Learning ■ Well designed games can enhance motivation and learning  a sense of accomplishment: Bandura; Gist et al.  informative feedback: Bransford et al.; Salas & Cannon Bowers  challenge /competition: Epstein & Harackiewicz; Reeve & Deci ■ Provide successive tasks to help players make progress towards concrete, specific and timely goals  goal setting: Locke et al.; Locke & Latham; Schunk & Ertmer ■ Provide repeated interactive experiences in task based environment  expertise: Glaser; Chi et al.; Bransford et al  anchored instruction: Bransford et al.; CGTV  active learning: Rothman; Chi; Mayer; Vogel et al. Entertainment Software Association 2012; Hussain et al 2009

8. Games and Formative vs. Summative vs. Standardized Assessment Games and Formative vs. Summative vs. Standardized Assessment ■ Games play nicely for formative assessment and learning  Fundamental teaching element in support of formative assessment they provide constant, clear feedback that allows for correction and learning ■ Relationship between standardized or summative assessment and games is potentially problematic ■ An ideal standardized or summative assessment purposely avoids teaching in the interest of maintaining validity – consider simulation vs. game or redefine assessment

9. USC Standard Patient Studio USC Standard Patient Studio In the US, adequate diagnostic interview & physical exam are the #1 and #2 sources of indemnity payout. Desire standardized, repeatable, virtual, emotionally expressive, standard patient encounters for medical diagnostic skills and interview training without expenses of live actors and recorded video evaluation. USC Standard Patient Studio -Simulated patient interactions with performance measurement / formative guidance. -Allows spoken or typed interviews, branching encounters, diagnostic tests and assessment. - Reduced authoring from 6 months to < 1 day with no coding. VALIDATION - >68% case performance gains after 20 minutes of use - 92% recognition & 96% accuracy medical interview natural language - automatic medical performance assessment accurate within 5% RECOGNITION 2015 Gold Serious Play Award 2015 IMSH SGVE Challenge Winner PIAA Data Sharing Project DSP2009-2013; 2014 Edition. Funded by the Defense Medical Research Development Program. 9

10. Standardized Patient Studio Feedback Standardized Patient Studio Feedback 10

11. Standardized Patient Studio Feedback Standardized Patient Studio Feedback 11

12. TraumaSim TraumaSim / RAMPART / RAMPART Created a simple 2D environment: - present multiple patients - manage common distractions with complimentary goals: - study physician decision making - evaluate construct validity of using a serious game for this purpose University of Pittsburgh physician / researcher studying trauma referral behaviors of ED physicians couldn’t simulate real world performance in the laboratory using traditional written cases. Pitt researchers validated that using TraumaSim: 1. physicians made decisions consistent with actual practice 2. that cognitive load could be manipulated 3. load increased use of heuristics consistent with cognitive theory RECOGNITION 2015 Serious Play Bronze Award Mohan, D., et al. Assessing the Validity of Using Serious Game Technology to Analyze Physician Decision Making (2014) http://dx.doi.org/10.1371/journal.po ne.0105445 12

13. TraumaSim TraumaSim / RAMPART / RAMPART 13

14. TraumaSim TraumaSim / RAMPART / RAMPART 14

15. The NBCOT Navigator™ The NBCOT Navigator™ INSTITUTE OF MEDICINE REPORT Concept for new tools derived from IOM Report: Health Professions Education: A Bridge to Quality PRACTICE ANALYSIS STUDY input from 2,500 certificants asking for interactive experiences VALIDATED 6 DOMAINS •Client-Centered Care •Work in Interprofessional Teams •Utilize Informatics •Employ Evidence-Based Practice •Apply Quality Improvement •Professional Responsibility VIRTUAL CONTINUING COMPETENCY ASSESSMENT PLATFORM Created with BreakAway Games to offer accessible, engaging and dynamic tools to provide: • Feedback on current skills • Career enhancement / growth • Evidence-based resources PILOT TESTED AND LAUNCHED Total # of users: 512 & 13,227 Total # of tools : 6,561 & 31,141 (9 week pilot & first 38 weeks live) RECOGNITION Judge’s Choice Most Innovative Product Demo -Association of Test Publishers Conference 2016 Silver Serious Play Award 2016 15

16. NBCOT Navigator™: A NBCOT Navigator™: A Portal + Portal + 4 4 Types of Assessments Types of Assessments PORTAL AND CERTIFICANT DASHBOARD ANIMATED CASE SIMULATIONS SELF REFLECTIVE ASSESSMENT AND GAMES MINI PRACTICE QUIZES 16

17. Contact Information Contact Information Jenn McNamara Vice President Serious Games and Strategic Partnerships BreakAway Games 610 420-8812 jmcnamara@breakawayltd.com