7 November 2006

1. Scenario-Based Training: Lessons from Development and Implementation 7 November 2006 Presentation to: Storytelling as an Instructional Method Conference Mesa, Arizona Presentation by: Alan Spiker Anacapa Sciences, Inc. Santa Barbara, CA

2. Topics • What’s a Scenario and why is it needed? • Challenges Facing a Small Business Implementing SBT • Evolution of Scenarios as an Element of Training • Useful ISD Principles of Scenario Based Training • Measuring SBT Effectiveness • Lessons Learned: Navy, Commercial Aviation, Army

3. Topics (details) • What’s a Scenario and why is it needed for training? • - look to the Thesaurus • - many good reasons and some not-so good reasons for scenarios • Challenges Facing a Small Business in Implementing SBT • - cajole not coerce, evince not convince, insinuate not dictate • - implementing various types of SBIR-funded technologies • Evolution of Scenarios as an Element of Training • - Geospecific DB --- > Framework ---- > Wizard ----- > Serious Games? • Useful ISD Principles of Scenario Based Training • - eight that REALLY matter • Measuring SBT Effectiveness • - Kirkpatrick’s 4-Level formal method • - informal method in the real world • Lessons from Implementing SBIRs w/ Navy, Commercial Aviation, Army • - Sometimes a great notion • - Our scorecard: wins, losses, draws

4. What’s a Scenario? • An outline of the plot of a dramatic work, giving particulars to scenes, characters, situations, etc. • The outline or manuscript of a motion picture or television program, giving the action in the order in which it takes place, description of scenes and characters • An imagined or projected sequence of events, esp. any of several detailed plans or possibilities • Setting for a work of art or literature • Postulated sequence of possible events • An outline or synopsis of a play • Syn: scheme, plan, concept, sketch • Sources: Dictionary.com WordNet

5. What’s a Scenario really? • An outline of the plot of a dramatic work, giving particulars to scenes, characters, situations, etc. • The outline or manuscript of a motion picture or television program, giving the action in the order in which it takes place, description of scenes and characters • An imagined or projected sequence of events, esp. any of several detailed plans or possibilities • Setting for a work of art or literature • Postulated sequence of possible events • An outline or synopsis of a play • Syn: scheme, plan, concept, sketch • Sources: Dictionary.com WordNet

6. Scenarios are good because they: • Can be a script or a blueprint of what will happen • Define roles of non-trainees • Constrain allowable actions, decision points • Help establish cognitive realism and trainee buy-in • Lay out sequence of action to unfold • They define which events occur, and when • Give the storyline of actions • Serve as communication conduit for training team • And thus ……

7. Scenarios: • Prevent chaos in the training environment • Provide an antidote to unscripted free play in the sims • Make training more interesting to the trainee • Make training more believable – more face valid • Make training more personal, customizable • Are more consistent with our Experiential Learning Model approach to training adults • Can’t substitute for measurable learning objectives • Don’t ensure criterion-based training • Don’t guarantee positive transfer of training BUT Scenarios --

8. Our Challenge as a Small Business • Anacapa Sciences – small business est. 1969 • Implement SBT as part of SBIR products • Solicit claimants through insinuation, use of data-driven approach, value-added consulting on the side • - so no organizational clout but that’s hardly unique to us • Since 2000, attempted SBT in several SBIRs: • SamePage: team training in Shared Understanding for ARI (Ph. II) • SimDATT: scenario-generation tool for NavAir (Ph. II+) • (CT)2: web-based critical thinking training modules for ARI (Ph. III) • TARGET: serious game based trainer (GBT) for ONR (Ph. I) • Stalwart: Wiki-based team trainer for JFCOM (Ph. I)

9. Common Approach SBT is still training - Typical Steps in SBT Projects • Identifying training need – why SBT needed in first place • Finding suitable SBT sponsor – how SBT will help him/her • Defining training baseline – what SBT is replacing • Finding training objectives – usually a real scavenger hunt • Selecting form of SBT – one size does not fit all • Managing expectations – what SBT can and can’t do • Measuring SBT effectiveness – how to know if it’s working • Implementing SBT – what’s the exit strategy • Assessing feasibility– who’s planning the birthday party

10. Evolution of SBT Small Business Perspective • Scenarios WERE: Big, physical, fixed, geo-specific, 1-1 physical fidelity with criterion environment, created by SMEs, little measurement • Scenarios ARE NOW: Smaller, cognitive, dynamic, regional, 1-1 cognitive fidelity with criterion environment, created by project team, stopping points for measurement • Scenarios WILL BECOME: Micro, meta-cognitive, adaptive, generic, 1-1 higher order fidelity with criterion environment, created by anyone, embedded measurement

11. Evolution of SBT Mission Scenario trainee’s role

12. Evolution of SBT Instructor Script instructor’s actions

13. Evolution of SBT Characteristic Features Now • Synopsis – Storyline method of organization • Trainee role specified through multiple means • Timeline – used as scaffold on which to specify sequence of action • Events – are specified, selected, and inserted as basic unit of action

14. Evolution of SBT Synopses - Storylines Select scenario from a library of possible synopses

15. Evolution of SBT Trainee Roles Conveyed through instructions, tables, special OPORDs, maps, immersion exercises, vignettes

16. Evolution of SBT Timelines – Action Flow

17. Evolution of SBT Event Selection and Specification

18. ISD Principles of SBTOverview • Navair funded add-on to SimDATT project • Identified 69 principles from ISD realm that could be applied to guide SBT development • User version incorporated into SimDATT to aid designer

19. ISD Principles of SBTEight Really Useful Ones

20. ISD Principles of SBTFive-Part Scenario Structure

21. ISD Principles of SBTLink Objectives to Student Goals

22. ISD Principles of SBTLink Content to Objectives

23. ISD Principles of SBTInclude Meta-Cognitive Skills

24. ISD Principles of SBTVary Setup Conditions

25. (1) (2) (3) (5) (4) ISD Principles of SBTUse Telescoping Design to Create Content

26. 2 scenarios most like your present scenario in their emphasis on CRM ISD Principles of SBTUse Analogous Cases as a Scaffold

27. ISD Principles of SBTStress Critical Thinking in the Debrief

28. Measuring SBT EffectivenessThe Formal Way Kirkpatrick’s 4-Level Approach to Evaluation

29. Measuring SBT EffectivenessReal World Metrics - Checklist • Does system have its own room? • Does system have a real-person as champion? • Does using community pony up some of its own money to keep system going? • Are they establishing training objectives for each scenario? • Do they call us (for tech support) to keep system going? • Do they return our phone calls and e-mails? • Do they invite us to their parties?

30. Lessons LearnedAnacapa’s Scorecard • Why did SimDATT version for Navy fail? • Why did SimDATT version for ASU-Mesa succeed? • Why did SamePage come to a draw with the Army? • Why might (CT)2 win/lose/draw with Army? - too close to call now Our Record is 1-1-1-1

31. Why SimDATT Failed with Navy • Users liked the aviator interface and content • But didn’t need scenarios for their part-task training • Either FRS or refresher • So all the marks appear high in Level I-II evaluation • But our real-world metrics are very poor

32. Why SimDATT Succeeded at ASU-Mesa • Same basic system as for the Navy • ASU-Mesa embraced scenarios for their students • Make even initial training LOFT-like • Have intimate relationship with the “gaining unit” • Supporting a four-level Kirkpatrick evaluation

33. Why SamePage Stalled with Army • Users liked the concept • Users liked the immersion experience • Users liked the challenge, great for leader development • But need domain changes to cover multiple MOSs • Needed a wizard, not a framework

34. What We Needed Was Something Like this

35. And This

36. Why (CT)2 Still in Limbo with Army Have conducted multiple evaluations, all very positive Contains multitude of cognitive vignettes But where to put into curriculum? What has to drop out to put (CT)2 in? Yet our inputs reached other parts of CGSC curriculum

37. Trends in SBTWrap-up • SBT becoming more adaptive • SBT could become more meta-cognitively oriented • SBT will become more SCORMED • SBT scenarios will be updated often, perhaps daily • SBT will become more personalized (e.g., wikis, avatars) • SBT will become “jazzed up” to support a market-based approach to training (how many hits did you get today Mr. Training Manager?) • SBT will become SBD, scenario based development

38. Meta-Cognitive Skills in ScenariosEffects Based Thinking Tool

39. Meta-Cognitive Skills in ScenariosUse of Wikis • Key Features • Search • Edit • Save entries • Color code edit versions • url is: • http://www.anacapatechnical.com/projects/jtf