Automated Assessment for Adaptive Learning of Complex Tasks

1. Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT

2. Characteristics of Cognitively Complex Tasks • Often require multiple, non-trivial steps to complete • Performance can be highly variable • Often include interdependent task features • Often well suited to representation in games & simulations • Multiple scenarios • Low cost • Low risk • But… assessment can be difficult Examples: Tactical Planning Simulation Damage Control Simulation

3. Anti-Submarine Warfare Anti-submarine warfare (ASW) is a branch of naval warfare that uses surface warships, aircraft, or other submarines to find, track and deter, damage, or destroy enemy submarines. • Carried out by teams of people within and across ships • Requires synthesis of large quantities of information in short periods of time • Heavy emphasis on situation awareness and decision making skills (especially among TAO’s)

4. Assessment Methodology Overview

5. ASW OntologyConstruction Experts Rules / Doctrine DOMAIN Facts / Observations Procedures / Processes

6. Ontology Excerpt

7. Bayesian Network Development Ontology Nodes Latent Variables SonarPlanning Directionality of Links PassiveSonar Planning Active Sonar Planning Strength of Relationships Observable Actions UseShipPassiveSonar DropPassiveSonobuoys Observables / SME Input UseDippingSonar DropActiveSonobuoys UseShipActiveSonar Characterizing the nature of relationships is essential

8. Bayesian Network for TDA Avoidance ASW Skilled MEUProtection TDA Avoidance Torpedo CounterMeasures Sub Hunting TorpedoEscape TorpedoDeception TDA AvoidManeuvers SonarPlanning Plan for Opposition TorpedoEvasion TorpedoCounter NixieDropped MICDropped Opposition Estimation Intell on Opposition AlterCourseToAvoidFast AlterCourseToAvoidSlow PassiveSonar Use Active Sonar Skill Datum Estimation DontEverEnterTDA ShowMissionInfo ShowBriefing LimLines Estimation UseShipPassiveSonar DropPassiveSonobuoys 0.5 EstDatumSubSpeed EstDatumTorpedoRange DropActiveSonobuoys UseShipActiveSonar EstLLSubSpeed EstLLTorpedoRange UseDippingSonar

9. Bayesian Network for TDA Avoidance ASW Skilled MEUProtection TDA Avoidance Torpedo CounterMeasures Sub Hunting TorpedoEscape TorpedoDeception TDA AvoidManeuvers SonarPlanning Plan for Opposition TorpedoEvasion TorpedoCounter NixieDropped MICDropped Opposition Estimation Intell on Opposition AlterCourseToAvoidFast AlterCourseToAvoidSlow PassiveSonar Use Active Sonar Skill Datum Estimation DontEverEnterTDA ShowMissionInfo ShowBriefing LimLines Estimation UseShipPassiveSonar DropPassiveSonobuoys 0.5 EstDatumSubSpeed EstDatumTorpedoRange DropActiveSonobuoys UseShipActiveSonar EstLLSubSpeed EstLLTorpedoRange UseDippingSonar

10. Conditions for TDA Avoidance to be considered by Bayesian network

11. The Advantage of Bayesian Networks • Probability of mastery of the latent variables is inferred from the scored observable actions Latent Variables Example: Determining probabilities SonarPlanning P(ASP): Probability of mastery of concept Active Sonar Planning P(UDS | ASP): Conditional probability. Probability of mastery of concept UseDippingSonar, given information about mastery of concept Active Sonar Planning PassiveSonar Planning Active Sonar Planning P(UDS):Probability of mastery of concept UseDippingSonar P(ASP) x P(UDS | ASP) Observable Actions P(ASP | UDS) = P(UDS) UseShipPassiveSonar DropPassiveSonobuoys UseDippingSonar DropActiveSonobuoys UseShipActiveSonar

12. Real-Time Formative Assessment Observable Actions & Events CAA(with Bayesian Network) • FORMATIVE ASSESSMENT • Provide performance feedback • Provide practice / resources • Add / change tasks • Add / change affordances Simulation(Sandbox) • ADAPTIVE OPTIONS

14. 1st PROBLEM: Chinese Kilos Threat Problem Check the Briefing—Good! 00:00:00 .k.Announce. ShowBriefing 0.95 The fact that we’re up against Kilos implies a silent sub speed of 6 Kt.and a torpedo range of 12,000 yds.

15. 1st PROBLEM: Chinese Kilos Threat Problem Failed to Correct Estimates—Bad! 07:26:32 .k.Announce. EstSubSpeed 0.207:26:32 .k.Announce. EstTorpedoRange 0.2 Scored at end, because these values were never changed. The estimated sub speed was left at 3 Kt. So the limit lines were not accurate, relative to the briefing.

16. 1st PROBLEM: Chinese Kilos Threat Problem Failed to Correct Estimates—Bad! 07:26:32 .k.Announce. DontEverEnterTDA 0.1 Uh-oh! The MEU entered the TDZ.

17. Problem 1

18. Problem 1

19. Problem 1

20. Problem 1

21. Problem 1

22. 2nd PROBLEM: Threat West of Gibraltar Check the Briefing—Good! 00:00:00 .k.Announce. ShowBriefing 0.95 This time the mission brief gives specific estimates of hostile sub speed and torpedo range.

23. 2nd PROBLEM: Threat West of Gibraltar Datum Estimates The datum estimates forsub speed and torpedorange don’t match the briefing.

24. 2nd PROBLEM: Threat West of Gibraltar Datum Estimates Corrected—Good! 01:14:03 .k.Announce. EstSubSpeed 0.9501:14:03 .k.Announce. EstTorpedoRange 0.95 Scored at end. Set to the correct values.

25. 2nd PROBLEM: Threat West of Gibraltar Check Limit Lines Perhaps not a wide enough berth…

26. 2nd PROBLEM: Threat West of Gibraltar Plot a New Course A safer course?

27. 2nd PROBLEM: Threat West of Gibraltar Check Limit Lines Again Looks safer

28. 2nd PROBLEM: Threat West of Gibraltar Successful Avoidance 01:14:03 .k.Announce. DontEverEnterTDA 0.95 Success!

29. Problem 2

30. Problem 2

31. Problem 2

32. Problem 2

33. TDA Avoidance Probability of Mastery Entered TDA time Plan for Opposition Sub speed: Wrong estimate Probability of Mastery time

34. TDA avoidance Bayesian network sub-net ASW Skilled TDA Avoidance Torpedo CounterM. Sub Hunting MEUProtection Plan for Opposition TDA AvoidManeuvers SonarPlanning Opposition Estimation Intelligence on Opposition ManeuverSlow ManeuverFast B16: score 0.95 D16: score 0.7 F16: score 0.05 B20: score 0.95 D20: score 0.6 F20: score 0.1 GetMissionInfo GetBriefing EstSubSpeed EstTorpedoRange B5: score 0.9 D5: score 0.7 B12: score 0.95 D12: score 0.8 F12: score 0.2 B6: score 0.95 D6: score 0.7 B11: score 0.95 D11: score 0.8 F11: score 0.2 DontEnterTDAIfPos B24: score 0.95 C24: score 0.9 F24: score 0.1 Sense Loudly Sense Quietly ShipsActivSonar ActiveSonobuoys DippingSonar ShipsPassivSonar PassiveSonobuoys B40: score 0.95 C40: score 0.4 E40: score 0.3 C44: score 0.9 D44: score 0.6 F44: score 0.05 B36: score 0.9 E36: score 0.3 F36: score 0.2 B28: score 0.95 D28: score 0.6 B32: score 0.95 E32: score 0.3

35. Initial Evaluation Feedback Instructional Feedback High Level Assessments CAA Debugging Output

36. First Successful Problem

37. Into the Next Problem

38. Adaptive Problem Selection A new problem is advised.

39. Loading the Problem The advised problem is selected

40. Initial Actions in the New Problem

41. Preparing to Find the Sub Launching a helicopter

42. Detecting the Sub Sonobuoys detect sub

43. Recording of Performances & Assessments CAA Report Generation (CAA Monitor) Assessment History (Per Student) Log Files (Actions and Events) These performance recordscan be replayed. Simulation(Sandbox) Report

44. Questions / Comments? CCT