E19.2250 Research on Simulations and Games for Education

1. E19.2250 Research on Simulations and Games for Education • Jan L. Plass, DMDL

2. Overview • Playtesting • Goals • Design • Methods • Setup

3. Game Genres • Goals for Games for Learning • Literacy • Focus on social learning, identity • Exploration and knowledge creation • Legitimate Participation in community of practice • Apprenticeship (reciprocal) • Learning through design & creating new artifacts

4. Game Research • Playtesting–Research Goals • Guide Design of New Games • Playtest, Usability of Game Prototypes • Efficacy, Evaluation of Completed Games • Advancement of Theory

5. Games Research • Playtesting • Obtain user opinions about a game • Questions concern levels of difficulty, fun, engagement • Follows in-house QA beta-testing • Typically involves 100...103 participants

6. Games Research • Usability Research • Usability Research: evaluate perceived and actual effectiveness of the design • Provide guidance for specific design decisions using a variety of methods involving the target audience • Typically involves 101...102 participants

7. Games Research • Efficacy Research • Evaluate the ability of the game to have the intended outcomes–cognitive, affective, or skills outcomes • Typically involves >103 participants

8. Class Activity • What are the function and process of an Expert evaluation/ Heuristic review? • What research questions are addressed? • What data is collected? • What happens with the results?

9. Playtest • Research Design • Focussed Research Questions • Sampling of Participants • Player Group Sizes and Composition • Location of Testing (Lab, field) • Standardization • Reference Data • Heuristics (experienced-based methods)

10. Group Activity • Play the Angles Game. Devise 3 research questions for playtesting of this game(15 min., Groups of 3–4) • What research questions are most important? • What function would the answer serve? • How could the question be addressed?

11. Playtest • Research Methods • Questionnaires • Interviews • Think-Aloud Protocols • Video Observations • User Logs (Events, User actions)

12. Playtest • Questionnaires • Learner/Player Profile • Learner Variables • Learner Responses • Consider • Types of questions • Question content • Question wording • Response format • Question placement and sequencing

13. Playtest • Questionnaires • Intrinsic/Extrinsic Motivation Scale

14. Playtest • Questionnaires • Self Esteem Scale

15. Playtest • Child Questionnaires

16. Playtest • Parent Questionnaires

17. Playtest • Interviews • Structured, Semi-Structured, or Open-Ended • Semi-Structured • How much did you like this game? • Would you play this game again in the future? • What did you like most about this game? • What did you like the least? • Did you make enough progress?

18. Playtest • Think-Aloud Protocols • Learner utterances during play • (more on that next week)

19. Playtest • Video Observations • Video Recording of Player Behavior and Game Play • Allows for later analysis of a variety of issues

20. Playtest • User Logs • Instrumented simulation or game writes game events and user actions to a log file or database • Need to specify what should be recorded • Highest Level: Every detail, allowing play back of game • Lowest Level: Only key events are recorded

21. Playtest • User Logs • id--Uniquely identifies each row of the table • game-A string identifiying the name of the game (with it's version number) • treatment-"Conceptual" or "Arithmetic" • userid-the user's id.  This is not required to be unique so it's up to us to make sure we don't reuse these, timestamps can help disambiguate • username-The user's name that they typed in themselves • timestamp-The date/time the rows was inserted into the table, rounded to the nearest whole second • location--The chapter&level or screenname where the event took place • who-The name of the entity who issued the event (for example, "ANGLE EDF", or "RIGHT ANGLE BUTTON") • what-The content of the message (for example, "SELECTED", or "Correct") • extra- Additional values pertinent to the message, but is often unused

22. Playtest • User Logs • CyberBuch Böll Survey Monday, November 18, 1996 • 12:15:38 PM Start using stack • 12:15:38 PM Environment: Annotations: TXTE PIC MOV Comprehension: Off Vocabulary: Off Forced lookup: On • 12:15:38 PM User Name Clarke XXXX • 12:15:40 PM Open Card 1 • 12:18:44 PM ärmlich TextEngl • 12:18:54 PM ärmlich Picture • 12:19:04 PM döst Picture • 12:19:16 PM döst TextEngl • 12:19:49 PM See Picture • 12:19:56 PM See TextEngl • 12:20:11 PM Wellenkämmen TextEngl • 12:20:20 PM Wellenkämmen Movie

23. Playtest • User Logs

24. Playtest • User Logs

25. Playtest • User Logs

26. Playtest • Research Setup • Lab or Field setting • Equipment to use for study • Role of Experimenter • Internet access

27. Playtest • Technical Setup • Data Capture from XBox, PS3

28. Playtest • Technical Setup • Data Capture from PC

29. Playtest • Playtest Research Plan • Research Questions • Research Design (player groups & composition, location) • Participant profile criteria • Sampling Plan • Method (Survey Instruments, Procedure, Technical Setup) • Analysis of Data

30. Group Activity • Develop your own research plan for playtesting of the Angles Game! (15 min., Groups of 3–4) • Research Questions • Research Design (player groups & composition, location) • Participant profile criteria • Sampling Plan • Method (Survey Instruments, Procedure, Technical Setup) • Analysis of Data

31. Statistics • Descriptive Statistics • Frequency Distribution • Measures of Central Tendencies • Variability • Correlations • Mean Comparison • Statistical Power and Effect Size

32. Statistics • Frequency Distribution

33. Statistics • Measures of Central Tendency • Mean • Mode • Median

34. Statistics • Variability • Range, Variance, Standard Deviation

35. Statistics • Variability • Range, Variance, Standard Deviation

36. Statistics • Means Comparison • T-test

37. Statistics • Statistical Power and Effect Size

38. Out of Class Activity • Play a game! • Play a computer/console game at home • Study a game! • What research questions do you find interesting exploring in the game you played?

39. Extra Slides • Backup Slides to follow

40. Cognitive Design Factors • Evidence-centered Design • Define educational goals of game • Define outcomes (e.g., desired knowledge of learners) • Define and identify acceptable evidence for learning • Design activities to learn and provide evidence of learning

41. Game Research • Research Frameworks

42. Research Framework

43. Research Framework • Alternative Knowledge Claims • Postpositivist knowledge • Socially constructed knowledge • Advocacy/Participatory Knowledge • Pragmatic Knowledge

44. Research Framework • Strategies of Inquiry • Qualitative • Ethnographies • Grounded Theory • Case Studies • Phenomenological Research • Narrative Research • Quantitative • Experiments • Quasi-Experiments • Correlational • Mixed Methods

45. Research Framework • Methods • Surveys & Questionnaires • Field observations • Open-ended interviews • Knowledge Tests • Psychometric Scales • Log files • Biometrics • EEG, EKG, EMG, NILS, GSR

46. Generate Learning Ideas Generate Gaming Ideas Analyze Requirements Formalize Ideas Evaluate Results Test Ideas Overview • Learning Game Design Model (Plass, 2010) Brainstorming No problems Physical Prototype Digital Prototype Problems with Design Design Documentation Refine and Polish Quality Assurance Educational Effectiveness Production