Productive Failure Manu Kapur Assistant Professor of Learning Sciences & Technology

1. Productive Failure Manu Kapur Assistant Professor of Learning Sciences & Technology National Institute of Education, Singapore ICET, Nov 22, 2007

2. Agenda • Set up the argument for productive failure • Study 1 – online setting (in Indian schools) • Study 2 – F2F setting (in a S’pore school) • Draw common patterns across the two studies • Draw implications using a complexity theory perspective

3. Argument for Productive Failure The situative, socio-constructivist perspective: learners need to be participate/collaborate in authentic, ill-structured problem-solving activities for meaningful learning to take place • Learners need to be scaffolded in their process of engaging in ill-structured tasks or else they may fail • But does this mean there is little efficacy embedded in un-scaffolded, ill-structured problem-solving processes?

4. Argument for Productive Failure 1. A Logical fallacy A implies B does not mean not-A implies not-B 2. Validity and reliability of measures 3. Several research programs point to the role of failure in learning and problem solving (VanLehn, 2003; McNamara, 2001; Schwartz & Martin, 2004, etc.) 4. The role of persistence vis-à-vis performance success

5. Study 1 Demonstrating an Existence Proof for Productive Failure

6. Purpose To examine whether or not there is a hidden efficacy in un-scaffolded, ill-structured problem-solving processes Context: Synchronous CSCL problem solving in Physics with N = 309, 11th grade science students across 7 high schools in India

7. 53 WSP Groups WSP Individual ISP Individual Pre-Test R (Triads) 50 ISP Groups WSP Individual ISP Individual Contrasting ISP with WSP The Study in Brief • Ill-structured groups showed: • Struggle with defining & solving the problem (MANOVA) • Complex and chaotic patterns of interaction (LSA) • Low convergence in their discussions (computational) • Poor group performance (ANCOVA), • BUT, better individual performance on both well- and ill-structured problems (Hierarchical Linear Modeling)

8. So? • It seems that there is efficacy embedded in un-scaffolded, ill-structured problem-solving processes • This efficacy can be extracted using a contrasting-case mechanism – a delay of structure • This efficacy seemed to be embedded in the chaotic, divergent, all-over-the-place interactional dynamics in the ill-structured groups

9. Implications • Question the default pedagogical rush to scaffold ill-structured problem solving • The ontology of learning & problem solving • Simple to complex – incremental, or • Complex to simple - emergent

10. Study 2 Exploring Productive Failure in a Singapore Classroom

11. Purpose • To test the productive failure hypothesis in a Singapore classroom, i.e., examine whether or not there is a hidden efficacy in un-scaffolded, ill-structured problem-solving and how it compares with traditional lecture & practice instruction • Context: • Clementi Town Sec School: A mainstream school • N = 76; Two classes of Sec 1 express-stream math students taught by the same teacher • Two curricula units, each lasting 7 lessons (about 2 weeks each); • Estimation & Approximation; Rate & Speed

12. PRODUCTIVE FAILURE (PF) CYCLE LECTURE-PRACTICE (LP) CYCLE Pre-test Pre-test Lecture, practice and feedback + HW PF Group Problem 1 Lecture, practice and feedback + HW PF Group Problem 1 cont’d Lecture, practice and feedback + HW PF Problem 1 Individual extensions Lecture, practice and feedback + HW PF Group Problem 2 Lecture, practice and feedback + HW PF Group Problem 2 cont’d Lecture, practice and feedback + HW PF Problem 2 Individual extensions Lecture, practice and feedback + HW Consolidation Lecture Post-test Post-test The Design in Brief (N = 76 Sec1 math students from CTSS, Singapore)

13. Example of an Ill-Structured Problem Gist of the Biking Problem (Speed Unit) Two friends, Jasmine and Hady, had to get to an exhibition by a certain time. They could walk or ride a bike or both. The constraint was that they had to reach the exhibition at the same time despite having different walking and biking speeds. Furthermore, a little while into their journey, one of the bikes breaks down, requiring re-strategizing for the rest of the journey.

14. Results • Process Analysis: • Problem Representations • Group & Individual Solution Scores • Self-report Confidence in their Solutions • Self-report Lesson Engagement • Rich interactional data remains to be analyzed • Outcome Analysis: • Pre-Post-test scores on rate and speed items: well-structured and ill-structured problem items

15. Process Analysis– Group Problem Representations

16. Process Analysis – Group Solution Scores

17. Process Analysis – Individual Solution Scores

18. Process Analysis – Confidence & Engagement Engagement Confidence

19. Outcome Analysis • Sample Well-structured Items • The flight distance between Singapore and Japan is 5316 km. A plane takes 6 hours and 15 min to fly from Singapore to Japan. What is the average speed of the plane? • David travels at an average speed of 4km/hr for 1 hour. He then cycles 6km at an average speed of 12 km/hr. Calculate his average speed for the entire journey in km/hr.

20. Outcome Analysis Ill-structured item Hummingbirds are small birds that are known for their ability to hover in mid-air by rapidly flapping their wings. Each year they migrate approximately 8583 km from Canada to Chile. The Giant Hummingbird is the largest member of the hummingbird family, weighing 18-20 gm. It measures 23cm long and it flaps its wings between 8-10 times per second. For every 18 hours of flying it requires 6 hours of rest. The Broad Tailed Hummingbirdbeats its wings 18 times per second. It is approx 10-11 cm and weighs approx 3.4 gm. For every 12 hours of flying it requires 12 hours of rest. If both birds can travel 1 km for every 550 wing flaps. If they leave Canada at approximately the same time, which hummingbird will get to Chile first?

21. Outcome Analysis: Overall Gains Controlling for the effect of prior knowledge as measured by the pre-test 10%, p = .002, ES = .75

22. Outcome Analysis 6%, p = .02, ES = .42 23%, p = .004, ES = .98

23. Going even further… • We also wanted to know how the PF cycle prepares students to learn and apply new concepts on their own • Extension Concept – Relative Speed • Half the students in each condition (PF and LP) took a scaffolded item on relative speed, the other halves took an un-scaffolded version • Then all students took an unscaffolded, conceptually difficult problem on relative speed.

24. Going even further… • Item 1: You and your friend start running at the same time from the same position but in opposite directions on a 400m running track. You run at 5m/s whereas your friend runs at 3m/s. • In 1 second, how many meters do you travel towards your friend? • In 1 second, how many meters does your friend travel towards you? • Therefore, in 1 second, how many meters do the two of you travel towards each other in total? • How many seconds will it take for the two of you to first cross each other? • Item 2: Two MRT trains on separate but parallel tracks are traveling towards each other. Train A is 100m long and is traveling at a speed of 100km/hr. Train B is 200m long and is traveling at a speed of 50km/hr. How many seconds will it take from the time that the two trains first meet to the time they have completely gone past each other?

25. Results

26. Results

27. Discussion • Productive Failure design seems tractable within local classroom context since the study was carried out within the timetable and curricula constraints • It seems to suggest shorter-term inefficiencies and failure but longer-term gains on both standard, well-structured items and more higher-order, ill-structured problem-solving items • The assessment experiment reveals that PF also prepares students to better use the structure provided for new concepts • One of the reasons structuring from the outset may not work could be due to our assumption that learners are prepared to use the structure provided!

28. Patterns across the 2 studies • Collaboration in small groups • Engage students in complexity of solving complex, ill-structured problems • Minimize a priori structure by not providing any external support or scaffolds • Delay structure, be it in the form of a contrasting well-structured problem or a consolidation lecture • Shorter-term inefficiency and failure but longer-term productivity

29. A Complexity Theory Perspective • Structure imposes order on the learning & performance space • Short term: efficient • Long term: may lack flexibility and adaptability • The laws of self-organization and complexity is: under certain conditions, as systems (biological, social, neural, etc.) comprising multiple interacting agents (genes, people, neurons, etc.) become increasingly complex over time, there comes a critical point where the system self-organizes and order emerges spontaneously from chaos.

30. A Complexity Theory Perspective • So, order is important! But, how does it come about? • Top-down vs. bottom-up order • (efficiency) (flexible, adaptive)

31. Laws of Self-organization & Complexity (Kauffman, 1995) ORDER CHAOS High Structure Processes Low Structure Processes Efficiency Innovation Self-Organization & Complexity Do we engage learners more in efficient or innovative processes?

32. Adaptive Experts (OAC: Optimal Adaptability Corridor, Schwartz, Bransford, & Sears, 2005) Innovation OAC Routine Experts Novices Efficiency Traditional Approach Balanced Approach My Proposal EFFICIENCY INNOVATION Implications for Adaptive Expertise (Hatano & Inagaki, 1986)

33. A Working Hypothesis underpinning Productive Failure… In the longer run, an innovation-dominant approach would be more optimal for the development of adaptive expertise than a balanced approach.

34. THANK YOU manu.kapur@nie.edu.sg