Assessment, Accountability and Adaptation April 13, 2006

1. Assessment, Accountability and Adaptation April 13, 2006 copies of this presentation can be found at www.business.duq.edu/faculty/davies

2. Education is a product. Economics provides tools for adequately addressing assessment, accountability, and adaptation, but to use the tools, we must first admit that education, while occupying a place of great importance, is fundamentally a product.

3. Assessment Adaptation Accountability How does the market judge products, hold producers accountable, and adapt to consumers’ needs?

4. The cycle begins with the consumer who develops a mental list of important attributes and a acceptable tradeoffs. Together, these form the basis of the consumer’s assessment of the product. • Attributes • Tradeoffs Assessment Adaptation Accountability

5. Consider grocery stores… • The Consumer Has a List of Important Attributes • Store is located in a safe and convenient location. • The produce is fresh. • There is a large selection of brands. • The store has a deli. • Prices are reasonable. • The Consumer Is Aware of Acceptable Tradeoffs • Convenient location is more important than fresh produce. • Lower price is more important than having a deli.

6. The consumer assesses the performance of the store by comparing the store’s attributes to the consumer’s list of important attributes and acceptable tradeoffs. • Attributes • Tradeoffs Assessment

7. The consumer holds the store accountable via repeat purchase. If the store’s attributes are acceptable to consumers, the store’s sales rise. If not, the store’s sales fall. • Attributes • Tradeoffs Assessment Accountability

8. If sales fall, the store will mimic and/or improve upon attributes of stores with rising sales. If the store’s sales rise, competing stores will mimic and/or improve upon this store’s sales. This is adaptation. • Attributes • Tradeoffs Assessment Adaptation Accountability

9. The consumer now re-assesses the store in light of the store’s new attributes and any changes in the consumer’s attributes / tradeoffs. This is outcomes assessment. • Attributes • Tradeoffs Assessment Adaptation Accountability

10. In the market place, what might cause the AAA cycle to break down? • Attributes • Tradeoffs Assessment Accountability Adaptation

11. Break in the Cycle #1: Consumer is unidentified • If the consumer is unidentified, questions arise: • What attributes are important? • What tradeoffs are acceptable? • Attributes • Tradeoffs Assessment With an unidentified consumer:  Appropriate assessment becomes unclear.  Source of accountability becomes unclear.  Appropriate adaptation becomes unclear. Adaptation Accountability

12. Break in the Cycle #2: Accountability not Possible • Repeat purchase – the consumer must have a need to purchase again so as to signal pleasure; • Mobility – the consumer must be able to purchase elsewhere so as to signal displeasure; • Short-run assessment – the consumer must be able to assess product quality prior to the repeat purchase decision. Possibility of accountability requires: If accountability is not possible • Appropriate adaptation becomes unclear. • Incentive for adaptation disappears. • Attributes • Tradeoffs Assessment Adaptation Accountability

13. First: Can we identify the consumer? Second: On what is assessment based? Fourth: What adaptations result? Third: Is accountability possible? How does the AAA cycle play out for primary and secondary education? • Attributes • Tradeoffs Assessment Accountability Adaptation

14. First: Can We Identify the Consumer? • Student? • Attributes: friends, sports, extracurricular activities. • Parents? • Attributes: safety, education, sports, future opportunities. • Colleges? • Attributes: critical thinking skills, math skills, communication skills, personality, drive. • Employers? • Attributes: work ethic, communication skills, ability to function as part of a team. • Society? • Attributes: likelihood of incarceration, likelihood of becoming a welfare recipient.

15. Second: On What is Assessment Based? • If Consumer = Student • Assessment based on the student’s social environment. • If Consumer = Parents • Assessment based on parents’ satisfaction with perceived educational environment. • If Consumer = Colleges • Assessment based on students’ college preparation. • If Consumer = Employers • Assessment based on students’ job skills. • If Consumer = Society • Assessment based on crime and poverty measures.

16. Only three definitions of “consumer” permit accountability. Third: Is Accountability Possible?

17. Fourth: What Adaptations Result? How we define “the consumer” ultimately determines how education will adapt. Definition of “consumer” determines attributes and tradeoffs, which determine assessment, which determines accountability, which determines adaptation.

18. Assessment Adaptation Accountability Fourth: What Adaptations Result when Consumer = Colleges? Assessment is college preparation. Accountability is imposed via acceptance rates. Schools adapt by making students more attractive to colleges so as to boost acceptance rate.

19. Fourth: What Adaptations Result when Consumer = Colleges? Assessment is college preparation. Assessment Unintended Consequence Schools will encourage students who are technically, rather than academically, talented to go to college rather than to prepare for the job market. Adaptation Accountability Accountability is imposed via acceptance rates. Schools adapt by making students more attractive to colleges so as to boost acceptance rate.

20. Assessment Adaptation Accountability Fourth: What Adaptations Result when Consumer = Employers? Assessment is job skills. Accountability is imposed via employment and wage rates. Schools adapt by making students more attractive to employers so as to boost employment and wage rates.

21. Fourth: What Adaptations Result when Consumer = Employers? Assessment is job skills. Assessment Unintended Consequence Schools will encourage students who are academically, rather than technically, talented to prepare for the job market rather than to go to college. Adaptation Accountability Accountability is imposed via employment and wage rates. Schools adapt by making students more attractive to employers so as to boost employment and wage rates.

22. Assessment Adaptation Accountability Fourth: What Adaptations Result when Consumer = Parents? Assessment is educational environment. Accountability is imposed via changing schools. Schools adapt by making themselves more attractive to parents so as to boost enrollment and retention.

23. Fourth: What Adaptations Result when Consumer = Parents? Assessment is educational environment. Assessment Unintended Consequence Schools will develop educational environments that conform to parents’ perceptions of quality education. Adaptation Accountability Accountability is imposed via changing schools. Schools adapt by making themselves more attractive to parents so as to boost enrollment and retention.

24. Can We Assess Education Without Identifying the Consumer? • Let’s assume that: • We can skip the step of identifying the consumer, and • Standardized Testing is the appropriate assessment tool, •  What is the impact of assessment on adaptation?

25. Assessment Adaptation Accountability Fourth: What Adaptations Result when Consumer = Undefined? Assessment is Standardized Testing. Accountability is imposed via funding (“high-stakes testing”). Schools adapt by making students better able to pass the test so as to boost funding.

26. Fourth: What Adaptations Result when Consumer = Undefined? Assessment is Standardized Testing. Assessment Unintended Consequence Schools will prepare students to pass the Standardized Test (“teaching to the test”). Adaptation Accountability Accountability is imposed via funding (“high-stakes testing”). Schools adapt by making students better able to pass the test so as to boost funding.

27. Fourth: What Adaptations Result when Consumer = Undefined? • Let’s assume that: • We can skip the step of identifying the consumer, and • Standardized Testing is the appropriate assessment tool, •  What attributes do school districts control?

28. An oft-cited important attribute is the Student-Teacher ratio This strategic plan stresses the need for strategies aimed at keeping more teachers in the classroom. – Strategic Plan, Texas Higher Education Coordinating Board, 2002. Teacher recruitment is the key to the drive to reduce class size. – Congresswoman Eddie Bernice Johnson, (D-TX), 2005. Utah will undertake several initiatives aimed at keeping more teachers in the classroom. – Utah Governor John Huntsman (R), 2005. Growing numbers of students in our nation’s schools…mean that our need for teachers increases each year. – Meeting the Challenges of Recruitment and Retention, NEA, 2005.

29. Student-Teacher Ratio has no apparent impact on NAEP scores. Each dot represents one state. Reported scores are sums of state averages for math and reading. Source: National Center for Education Statistics, U.S. Department of Education

30. School size has no apparent impact on NAEP scores. Source: National Center for Education Statistics, U.S. Department of Education

31. Spending per Pupil has no apparent impact on NAEP scores. Source: National Center for Education Statistics, U.S. Department of Education

32. Spending per Pupil has no apparent impact on NAEP scores. Source: National Center for Education Statistics, U.S. Department of Education

33. Spending per Pupil has no apparent impact on NAEP scores. Source: National Center for Education Statistics, U.S. Department of Education

34. Median Family Income has no apparent impact on NAEP scores. Source: National Center for Education Statistics, U.S. Department of Education

35. Perhaps standardized tests are inadequate for assessing the quality of education. What about dropout rate? This measure comes closer to reflecting parent/student choice-to-purchase.

36. Student-Teacher Ratio has no apparent impact on dropout rate. Source: National Center for Education Statistics, U.S. Department of Education

37. School Size has no apparent impact on dropout rate. Source: National Center for Education Statistics, U.S. Department of Education

38. Spending per Pupil has no apparent impact on dropout rate. Source: National Center for Education Statistics, U.S. Department of Education

39. Spending per Pupil has no apparent impact on dropout rate. Source: National Center for Education Statistics, U.S. Department of Education

40. Median Family Income has no apparent impact on dropout rate. Source: National Center for Education Statistics, U.S. Department of Education

41. Does anything predict Standardized Test performance or Dropout Rate?

42. For 8th grade, past NAEP scores predict future NAEP scores. Source: National Center for Education Statistics, U.S. Department of Education

43. For 4th grade, past NAEP scores predict future NAEP scores. Source: National Center for Education Statistics, U.S. Department of Education

44. 4th grade NAEP scores predict 8th grade NAEP scores. Source: National Center for Education Statistics, U.S. Department of Education

45. Conclusion Attributes that conventional wisdom suggests are appropriate have little or no impact on Standardized Test Results or Dropout Rates. But, school performance is not random – states that perform well in the future are those that have performed well in the past. Supposition As most of the attributes employed by school districts are systemic, evidence suggests that educational quality may be better influenced via individual attributes.

46. Assessment, Accountability and Adaptation April 13, 2006 copies of this presentation can be found at www.business.duq.edu/faculty/davies