Assessing the Benefits of Online Scenario Simulation Tools in Security-Related Studies at UMUC

1. Assessing the Benefits of Online Scenario Simulation Tools in Security-Related Studies at UMUC Irmak Renda-Tanali, DSc. Collegiate Associate Professor Director, Homeland Security & Emergency Management Graduate School of Management & Technology University of Maryland University College Fourteenth Annual Emergency Management Higher Education Conference Federal Emergency Management Agency, U.S. Department of Homeland Security Thursday June 9, 2011

2. Outline • Background • Research Objectives • Methodology • Research Design • Tools • Results • Survey Results • Actual Performance Results • Conclusions • Recommendations Irmak Renda-Tanali

3. Background • Why such study? • Emergency Management and related fields are best learned through scenario studies • Textbooks and lecture-based learning do not effectively teach: • Consensus gathering • Collective decision making • Decision making under uncertainty • Testing and validating assumptions • How to manage major crises and emergencies • Textbooks and lectures do not capture the dynamic, time-sensitive, context-dependent, multi-disciplinary nature of the emergency/crisis scenarios. Irmak Renda-Tanali

4. Background • Why such study (cont’d)? • Course evaluations and teaching experience show: • Students want dynamic, hands-on exercises that simulate real world environments, real world thinking. • Textbook reading and conference participations need further stimulating tools to enhance student understanding of topics. • Students want to be an active part of decision making process as crisis events unfold. • They want to learn what are the good AND bad decisions and why, by actively seeing consequences of their decisions in reaction to events that unfold. • Case studies looking backward are not so interesting as the events and consequences are already known. • Students learn better by hands-on practice than just by reading. Irmak Renda-Tanali

5. Background • Examples of current practice: • Multi-media technology • Video, audio, web pages, web 2.0 tools: blogs, wikis, social networking sites, podcasts, vodcasts etc. • Play2Train, a virtual world platform, uses Second Life software with avatars (intelligent agents) mimicking human behavior. • Specific examples related to security studies: • A web-based scenario simulation tool used for public health undergraduate students (Spinello & Fishbach, 2004) • San Luis Rey town scenario simulation exercise for HS students at UMUC graduate studies (Boubsil & Gayol, 2006) • Play2Train, used to replicate tabletop exercises, e.g. Centers for Disease Control (CDC), Emory University, University of Illinois at Chicago’s Center for the Advancement of Distance Education (CADE), Seton Hall University, and Idaho State University (Hewitt et al, 2009) Irmak Renda-Tanali

6. Background • Examples (cont’d) • Specific examples related to security studies (cont’d) • Second Life and Play2Train in Master of Healthcare Administration online programs, used to enhance student discovery, critical thinking, and analytical skills. Irmak Renda-Tanali

7. Research Objectives • To capture the effectiveness of online scenario simulation tools used at UMUC, on student learning; • To identify the kinds of strengths and weaknesses of those tools; • To compare those tools to traditional methods of teaching; • Ultimately to enhance student learning in Emergency Management and related fields of study through the findings of the study. Irmak Renda-Tanali

8. Study Tools • Subjects: • 5 online sections of CCJS 320 – 145 students in total • 3 online sections of CCJS 420 – 78 students • 2 online sections of CCJS 425 – 45 students • 2 online sections of BSBD 641 – 51 students  total 319 students • Instructors • Developer-instructors Taught in Fall 2010 semester • Data Sources • Student survey for all sections of the above courses • Phone calls to faculty developed and/or use simulation • Course statistics measuring student participation and interaction • Gradebook • The generic end-of-semester evaluations Irmak Renda-Tanali

9. Areas Surveyed • Learning: • Understanding the material • Processing, testing, validating assumptions • Decision making skills • Getting close to real-life experience • Connecting events and objects (or actions) • Visualizing, conceptualizing, comprehending • Consensus gathering • Effective crisis communication • Effective use of resources • Effective emergency response to large scale crises • Student-to-student interaction • Student-to-faculty interaction Irmak Renda-Tanali

10. Areas Surveyed • Technology use • Workload • Student Satisfaction • Demographics Irmak Renda-Tanali

11. Assumptions • Learning: • Online scenario simulation tools help understand the material better. • They help better process, test, validate assumptions • Improve decision making skills. • Help get close to real-life experience. • Help connect events and objects (or actions). • Help with visualizing, conceptualizing, and comprehending. • Increased student-to-student interaction contributes to learning (Community of Inquiry Theory by Garrison et. al, 2000) • Increased student-to-faculty interaction contributes to learning (CoI). • Simulation tool  increased interaction learning Irmak Renda-Tanali

12. Assumptions • Technology use: • If there are no problems with the technology, student satisfaction is more  better learning • Workload: • No assumptions about the time spent versus contribution to learning • Student Satisfaction: • Students who responded favorably performed favorably – or did they? • Demographics: • No assumptions – a correlation study may uncover unknown facts Irmak Renda-Tanali

13. Setting CCJS 320 • Project Name: “Crime Scene Simulation” • Actions: Students virtually move around a crime scene, examine items, process evidence, place them in an evidence box, interview the witness, and record details in a journal. After all the evidence is collected, students enter into a virtual Crime Lab and request the appropriate tests for each evidence item such as medico-legal autopsy, fingerprint analysis, forensic biology, toxicology, firearm examination, etc. • Deliverable: After the tests are performed a detailed Crime Scene Report is generated. Students are graded based on the correctness of the evidence collected and the tests generated. The crime lab yields no results if incorrect tests are requested. Irmak Renda-Tanali

14. Setting CCJS 320 (cont’d) Irmak Renda-Tanali

15. Setting CCJS 320 (cont’d) Irmak Renda-Tanali

16. Setting CCJS 320 (cont’d) Irmak Renda-Tanali

17. Setting CCJS 320 (cont’d) Irmak Renda-Tanali

18. Results: Crime Scene Simulation • Response Rates (CCJS 320): Irmak Renda-Tanali

19. Results: Crime Scene Simulation • Understanding the material Correlation to itself Large (strong) correlation Large correlation (Pearson Coefficient) Medium correlation Low correlation Irmak Renda-Tanali

20. Pearson’s Correlation Coefficient Pearson's correlation coefficient between two variables is defined as the Covariance of the two variables divided by the product of their standard deviations. Simple to do using Excel spreadsheet with the embedded function. Irmak Renda-Tanali

21. Results Summary: Crime Scene Simulation

22. Results Summary: Crime Scene Simulation

23. Results Summary: Crime Scene Simulation

24. Results Summary: Crime Scene Simulation

25. Setting CCJS 420 • CCJS 420 Medical and Legal Investigations of Death Cold Case Simulation • Actions: Students begin by examining the reports and evidence from the original investigation. Once they are familiar with the homicide case and the people involved, they proceed to interview persons of interest. As the simulation progresses and student encounter new evidence, they will be able to submit that evidence for various types of forensic testing. • As students proceed with their investigation, they gather evidence, test results, reports, and witness statements. These items are placed into their binder as they are created or discovered. Deliverable: Arrest Warrant Report Objectives: Gather enough evidence, secure an indictment from the district attorney, and solve the case. Irmak Renda-Tanali

26. Setting CCJS 420 Irmak Renda-Tanali

27. Results: Cold Case File Simulation • Response Rates (CCJS 420): Irmak Renda-Tanali

28. Results Summary: Cold Case File Simulation

29. Results Summary: Cold Case File Simulation

30. Results Summary: Cold Case File Simulation

31. Results Summary: Cold Case File Simulation

32. Setting BSBD 641 Irmak Renda-Tanali

33. Results: Biological Attack Simulation • Response Rates (BSBD 641): Irmak Renda-Tanali

34. Results Summary: Biological Attack Simulation

35. Results Summary: Cold Case File Simulation

36. Results Summary: Cold Case File Simulation

37. Results Summary: Cold Case File Simulation

38. Results Summary

39. Results Summary

40. Results Summary

41. Results Summary

42. Results Summary

43. Conclusions • Online scenario simulations add value to understanding. • They help learning through discovery. • Provide near real-life, hands-on experience. • Help visualize, conceptualize and make connections between events and tools/objects. • There is a correlation between student satisfaction and grades. Further studies with control group settings can uncover this or more facts. • May or may not increase interaction among students. Further studies should look into this.

44. Conclusions • There is evidence of increased student-to-faculty interaction. • Technology glitches diminish value and satisfaction. (e.g. dead ends, loops) • Information glitches diminish value and satisfaction. • Age plays an important role in designing the content for the student audience. • A negative correlation between time spent and learning the material was uncovered. Was the time spent on technical problems? Further studies should look into optimal use of time. Too much time spent may bring diminishing returns.

45. Conclusions • The more sophisticated (game like) features make the tool more satisfactory. • Students would like a sense of achievement at the end (e.g. solving piece of a puzzle, putting pieces together) • Caution: Make them game-like but emphasize pedagogy. • If designed as a group work with role playing ability  may increase communication skills, help build core competencies in crisis management such as consensus gathering, effective resource allocation, understanding response structures in large scale crises and disasters.

46. Conclusions • Recommend a move from low budget productions to sophisticated online simulation tools if budget permits. • Expanded studies may show a link between the dollar amount spent for development and the student success outcome (Benefit-Cost Analysis). • The tools should allow room for changing/ alternating between multiple scenarios to prevent boredom among professors and student sharing of results. • Need buy-in from instructors. Engage them in designing the tools.

47. Conclusions • There are successful applications with proven results about other similar simulations employed by other higher edinstutions. UMUC can emulate on that. • The new generation as “Digital natives” learn better with game-like simulations.

48. Acknowledgments This research was funded by the UMUC Faculty Research Grant Program. Those who contributed to the research include Husein Abdul-Hamid, Susan Blankenship, Howard Krivan, and Howard Dobres.