Cognitive Measurements to Design Effective Learning Environments

1. Cognitive Measurements to Design Effective Learning Environments Fred Paas1 & Slava Kalyuga2 1 Open University of the Netherlands 2 New York University, USA 3 I C L E P S WORKSHOP 2005 August 30

2. Overview COGNITIVE MEASUREMENT: GROUP-BASED INSTRUCTION Measurement of cognitive load Interpretation of performance, mental effort, and combined scores COGNITIVE MEASUREMENT: PERSONALIZED INSTRUCTION Diagnostic assessment of organized knowledge structures Applying combined efficiency measures in adaptive training

3. Basic Assumption If individuals are to learn effectively in a learning environment, the architecture of their cognitive system, the learning environment, and interactions between both must be understood, accommodated, and aligned Cognitive Measurements

4. Sensory Memory Working Memory Long Term Memory Cognitive architecture • Perceive incoming information • Attend to information • Limited capacity • Limited duration • Separate processors for visual and • auditory information   • Permanently store all knowledge • and skills in a hierarchical network • (schemas) • Unlimited capacity

5. Schema: • categorizes elements of information • according to the manner in which • they will be used • consists of a multi-dimensional web • of interconnected nodes of • information • can be treated by WM as a single • entity, and if the learning process • has occurred over a long period of • time, it may incorporate a huge • amount of information • can be processed consciously or • automatically

6. The Concept Cognitive Load represents the load that performing a particular task imposes on our cognitive system Capacitymax. Determined by the number of information elements and their interactivity Germane Extraneous Determined by the manner in which the information is presented to learners Intrinsic Capacitymin. Intrinsic Extraneous • not relevant for learning Germane • relevant for learning

7. Decreasing Extraneous Load Minimizing Extraneous Load • Goal-free effect • Worked-example effect • Completion effect • Split-attention effect • Modality effect • Redundancy effect Managing Intrinsic Load Increasing Germane Load Maximizing Germane Load • Sequencing effect • Fading support effect • Variability effect • Self-explanation effect • Imagination effect • Interactivity effect Cognitive Load Theory Instructional Techniques

8. Objective measures • Task and performance • Secondary task • Psychophysiological • Subjective measures • Rating scales Measurement of Cognitive Load

9. Start Given: A car that starts from rest and accelerates uniformly at 2 meters/s2 in a straight line has an average velocity of 17 meters/s. Goal: How far has it traveled? Operators: s = v * t , v = .5V and V = a * t (V=final velocity, v=average velocity, a=accelaration, t=time, s=distance) Problem statement and equations in working memory Equations with goal as only unknown Equation with subgoal as only unknown Equation with subgoal(s) and unknown(s) Problem solved Solve equations and add new known to working memory yes yes no no no Equation with goal and unknown(s) no yes no Subgoal(s) added to working memory yes no Goal-Specific Problem Solving

10. Start Given: A car that starts from rest and accelerates uniformly at 2 meters/s2 in a straight line has an average velocity of 17 meters/s. Calculate the value of as many variables as you can. Problem statement and equations in working memory Equation with only one unknown Stop search no yes Solve equations and add new known to working memory Goal-Free Problem Solving

11. Slow RT Rapid RT Resources to secondary task Resources to secondary task Cognitive resources to simple primary task Cognitive resources to complex primary task Fixed cognitive capacity Fixed cognitive capacity Measurement of Cognitive Load • Objective measures • Task and performance • Secondary task • Psychophysiological • Subjective measures • Rating scales

12. Measurement of Cognitive Load • Objective measures • Task and performance • Secondary task • Psychophysiological • Subjective measures • Rating scales

13. Heart rate variability Eye movements Pupil dilation

14. Measurement of Cognitive Load neither low nor high mental effort very, very low mental effort very, very high mental effort In solving or studying the preceding problem I invested: • Objective measures • Task and performance • Secondary task • Psychophysiological • Subjective measures • Rating scales

15. Subjective measures: Rating scales (NASA-TLX)

16. How to interprete performance and mental effort scores? Instructional Condition Performance (1-10) Mental Effort (1-10) A B C D 2 2 8 8 9 2 3 9

17. high efficiency low efficiency Efficiency of Instructional Conditions Performance M = P Efficiency = 0 Mental Effort

18. high motivation low motivation Motivation in Instructional Conditions Performance Mental Effort Motivation = 0

19. Instructional Efficiency/Motivation Performance high motivation high efficiency Mental Effort low motivation low efficiency

20. Challenges Measurement of Cognitive Load: Intrinsic items 1- how easy or difficult do you consider probability theory at this moment? Extraneous items 2- how easy or difficult is it for you to work with learning environment? 3- how easy or difficult is it for you to distinguish important and unimportant information in the learning environment? 4- how easy or difficult is it for you to collect all the information that you need in the learning environment? Germane items 5- how easy or difficult was it to understand the solution in the last animation? • Find usable objective technique • Distinguish between different types of load • Use measures to personalize instruction

21. COGNITIVE MEASUREMENT: PERSONALIZED INSTRUCTION

22. Expertise reversal effect:Cognitive load effects depend on levels of learner expertise:instructional designs or procedures that are effective for novices may be ineffective for more proficient (expert) learners.Instructional implications:- instructional techniques need to change with alterations in expertise;- it is critical to have simple rapid measures of learner proficiency (performance and mental effort).

23. Real time (rapid online) diagnostic assessment of organized knowledge structures.Why organized knowledge structures (schemas)?Cognitive studies of expertise: organized knowledge base in LTM is central to cognitive processing(De Groot, 1946/1965, Chase & Simon, 1973); they affects the way we process information in WM and solve problems (Novices vs Experts).

24. Solve for x: 5x = - 4

25. Solve for x: 5x = - 4 x = - 4/5

26. Solve for x: 5x = - 4 5x/5 = - 4/5

28. Diagnostic cognitive assessment should be • sensitive to different cognitive attributes • sensitive to different levels of proficiency • practically usable Typical time scale of cognitive processes: up to several seconds.

29. Rapid diagnostic approach: general: What is the highest level of organised knowledge structures (if any) a person is capable of retrieving and applying to the briefly presented material? first-step method: Presenting learners with a task for a limited time and asking them to indicate their first step towards solution.

31. Rapid verification diagnostic technique: Presenting learners with a series of possible task solutions for a limited time and asking them to rapidly verify the suggested solution steps. Physics (kinematics) vector addition motion problems A ship is traveling at 7 m/s. A dog runs across the deck at the same speed in a direction of 60° relative to the direction of motion of the ship. What is the velocity of the dog relative to the sea?

33. Combined efficiency measures E = P/R

35. Using rapid diagnostic techniques in adaptive training for dynamic learning task selection (tailoring levels of task complexity and learner support).

36. Selection algorithm governing the selection of learning tasks with different levels of difficulty (stages 1-4) and support (worked examples, completion tasks, and conventional tasks/problem solving exercises). Adapted from Kalyuga and Sweller (2004)