Educational Policy Research: Lessons, Puzzles, and New Directions Richard J. Murnane Harvard Graduate School of Educatio

1. Educational Policy Research:Lessons, Puzzles, and New DirectionsRichard J. MurnaneHarvard Graduate School of Education & NBER World Bank Presentation May 19, 2010

2. U.S. Economy-Wide Trends in Routine and Non-Routine Task Input: 1969-1998 (1969=0)

3. Key Elements of Expert Thinking: • A great deal of well organized knowledge about the problem (not memorized facts, but well understood relationships). • Skill at pattern recognition • Initiative • Metacognition

4. Key elements of Complex Communication • Observing and listening. • Eliciting critical information. • Interpreting the information. • Conveying the interpretation to others.

5. Why is a population skilled at expert thinking and complex communication important for economic growth? • These skills are increasingly important to firms competing by being innovative and/or by providing high quality products and services. • Unequal access to these skills exacerbates earnings inequality, which, in turn, may threaten political stability. • These skills are also critical to being a productive contributor to civic life in a democracy.

6. Implications for Education (1) • The growing importance of expert thinking and complex communication does not reduce the importance of reading and mathematical skills. • they are critical tools for developing deep knowledge in a domain • key is developing the disposition to use these tools to continue to learn, and to communicate effectively. • Do assessments examine individuals’ skill at using reading and math skills for problem-solving?

7. Implications for Education (2) • Expert Thinking and Complex Communication are not new subjects to add to the curriculum. They should be at the center of instruction in every one of the existing subjects. • But this means changing instruction (and perhaps changing assessments) • Tendency instead is to add new courses or purchase additional inputs such as new books or new computer software.

8. Expert Thinking and Social Studies in Social Science (questions asked of students in two social studies classes) What was the date of battle of the Spanish Armada? Student in class 1: 1588. Q. How do you know this? It was one of the dates I memorized for the exam. Q. Why is the event important? I don’t know. Student in class 2: It must have been around 1590. Q. How do you know this? I know the English began to settle in Virginia just after 1600, although I’m not sure of the exact date. They wouldn't have dared start overseas explorations if Spain still had control of the seas. It would have taken a little while to get expeditions organized, so England must have gained naval supremacy somewhere in the late 1500's. Q. Why is the event important? It marks a turning point in the relative importance of England and Spain as European powers and colonizers of the New World. What were the differences in instruction and in children’s experiences in the two social studies classes? This example is taken from Bransford, Brown and Cocking (eds.)

9. Four Guidelines for Improving Educational Outcomes (and some puzzles to pay attention to) • Lower the Cost of School Enrollment • Change Children’s Daily Experiences in School • Improve Incentives • Create More Schooling Options for Children from Poor Families

11. Guideline 1: Lower the Cost of School Enrollment • Reduce Commuting Time • Duflo (2001) on Indonesia • Burde & Linden (2009) on Afghanistan • Currie & Moretti (2003) on U.S. • Reduce Out-of-Pocket Educational Costs • Duflo et al. (2006) on Kenya • Dynarski (2003) on U.S. • Reduce Opportunity Costs • Fiszbein & Schady (2009) on Conditional Cash Transfers • Puzzle • Increasing attendance does not always increase achievement (Hanushek & Woessman, 2009; Fitzbein & Schady, 2009)

12. Guideline 2: Change Children’s Daily Experiences in School • More resources (new books, smaller classes, computer software & Hardware) by themselves do not change children’s daily experiences (Hanushek; Barrera-Osorio & Linden, 2008). • Better Teaching • Challenges • Pre-service credentials do not predict teaching effectiveness well. • Conventional in-service professional development does not alter teachers’ instructional methods or children’s experiences. • Components of effective professional development • Focus on instructional methods for teaching particular skills • School-based focus in order to make instruction more consistent. • Teachers need to observe effective instruction of students similar to theirs. • A process, not an event. • Accountability. • Puzzle: Most school systems devote resources to in-service training of teachers. • Evidence that in-service training enhances students’ skills? • Evidence that in-service training consistently changes instruction? • Critical research challenge: Examining conditions under which teacher training makes a difference.

13. Guideline 3:Improve Incentives (for teachers) • Base compensation on factors teachers can control that influence student achievement: • teacher attendance (Duflo et al., 2008) • working at low-performing school (Clotfelter et al., 2008; Steele et al., 2010) • Base compensation on students’ achievement gains • Some Positive Evidence (Muralidharan & Sundararaman, 2009; Lavy, 2009) • Puzzles and Cautions • Half of variation in “value-added” consists of variation within teachers across years (McCafferty et al, 2009) • Rank order of teachers’ value-added depends on choice of test and timing of test in a particular subject (Papay, 2010) • Dysfunctional responses may increase over time (Jacob & Levitt, 2003; Figlio & Getzler, 2002). • A new direction: Focus on school as an organization – evidence from Jackson & Bruegman (2009) • Hypothesis: Performance incentives are more likely to elicit desirable responses if coupled with capacity building. Focus on doing this at the school level.

14. Guideline 3: Improve Incentives (for students) • Long-run incentives are critical • Education paying off in labor markets • Long-run incentives may not be enough, given short time horizon of teenagers. • Puzzles about responses to short-run Incentives • Different responses in Different Settings (Kremer et al., 2009) • Heterogeneous responses in the same setting (males v females, better prepared v. less well prepared):Angrist & Lavy, 2009; Angrist et al., 2009; Riccio et al., 2010 • Student responses depend on What is Rewarded (Fryer, 2010) • positive effects on student test scores to payment for reading books, for regular attendance, and for behaving well. • No responses to substantial payment for good test scores – students do not know how to achieve this objective.

15. Guideline 4: Create More Schooling Options for Children from Poor Families • Many policy options: vouchers; charter schools; subsidies to schools to serve poor children. • Policy should favor children from poor families (contrast Colombia secondary school vouchers with Chile school vouchers; Angrist et. al., 2002, 2006; McEwan et al., 2008) • Public funding of choice options should be tied to accountability for performance (Barrera-Osorio & Raju, 2009). • Puzzle: Some schooling options affected long-run outcomes for disadvantaged students, but not test scores • Career academies (Kemple, 2008) • Charlotte-Mecklenburg high school lotteries (Deming, 2009). • So how should performance be measured for accountability? • Possibility of waivers from conventional accountability measures if a group of schools commits to rigorous evaluation using alternative outcome measures?

16. Measuring Progress (1): A Homework Question • Examine the homework that teachers typically assign • Does the homework push students to develop expert thinking skills (non-routine problem solving)? • What about communication skills? • Or does the homework ask students to do the kind of rules-based tasks that computers can be programmed to do? • The answer may tell you a lot about the types of jobs the school is preparing students to do.

17. Measuring Progress (2): NAEP Test Score Trends in Mathematics National Averages for 13 Year Olds (8th Grade) in U.S.

18. Measuring Progress (3): Trend in the relationship between family SES and children’s cognitive skills (Sean Reardon, 2010)

19. Final Words • The Guidelines are just that (not recipes) • The guidelines are complements, not substitutes • Lower the Cost of School Enrollment • Change Children’s Daily Experiences in School • Improve Incentives • Create More Schooling Options for Children from Poor Families • Incentives and capacity-building may be critical partners. • It is important to pay attention to puzzles in the evidence • year-to-year variation in estimated teacher performance in improving students’ scores • schooling options that improve long-run student outcomes, but not scores on standardized tests of reading and mathematics skills • Focusing on improving schools as organizations may be more effective than focusing on improving the performance of individual teachers. • A critical challenge is to convince policymakers of: • the importance of rigorous evaluations • (especially of activities that use a lot of resources such as in-service training) • the importance of planning the evaluation at the same time that policy intervention is being planned • The design of the Progresa CCT initiative in Mexico provides one example of implementing a program in a way that supports a rigorous evaluation.

20. Thank you for your time. Email: Richard_Murnane@Harvard.edu