Assessment for Effective Schooling

1. AEA conference Belfast, 10 November, 2011 J.Scheerens@utwente.nl Assessment for Effective Schooling

2. Contents • Theories on effective school improvement • Evaluation and feedback as an improvement mechanism • What is the actual impact of evaluation and assessment? results from meta-analyses • Tentative explanations of non-effectiveness; remedies

3. Control theories • Synoptic planning • Market mechanisms • “retroactive planning” Alternative “non control” theories • Self-organization • Professional bureaucracy and loose coupling

4. Synoptic planning System level variables relevant from the perspective of the synoptic planning model: • national curriculum planning • infrastructure for program evaluation in education • centralized structures, limited school autonomy • school inspection

5. Market mechanisms System level variables relevant from the perspective of creating market mechanisms • free school choice • financial and managerial school autonomy • privatization • vouchers • “high stakes” accountability arrangements

6. Retroactive Planning System level variables relevant from the perspective of retroactive planning • a broad range of accountability provisions • national assessments • school inspection • new public management: free processes control outcomes • (institutionalization of) school self-evaluation • (institutionalization of) school development planning • Facilities for continuous professional development of teachers

7. Self-organization • school autonomy • recognition of educational cultures • recognition of the importance of composition effects

8. Professional Bureaucracy and Loose Coupling • Professionals resist rational techniques as planning and evaluation • Little management required • Teacher autonomy; initial training and enculturation in the profession most important coordination mechanism • Goals and means are loosely coupled

9. Zooming in on retroactive planning • Central place for evaluation and assessment • Tests as operational goals • Test results as basis for learning and improvement • Test results as a basis for task related cooperation and “professional learning communities” and instructional leadership • Teaching to the test

10. Retroactive and Synoptic Planning

11. Key expectations retroactive planning • Output control • Motivational role; judgmental feedback • Informative role: diagnosis, instrumental feedback, improvement

12. Definitions • Evaluation as a general term for evidence based judgment of programs, organizations and people • Monitoring registration and evaluation associated with administrative data • Assessment associated with performance measures (tests) of students • Evaluation and Monitoring may include assessment

13. Feedback A feedback loop to control human behaviour involves four distinct stages. Firstly - Evidence. A behaviour must be measured, captured, and data stored. Secondly - Relevance. The information must be relayed to the individual, not in the raw-data form in which it was captured but in a context that makes it emotionally resonant. Thirdly - Consequence. The information must illuminate one or more paths ahead. Fourthly - Action. There must be a clear moment when the individual can recalibrate a behavior, make a choice, and act. Then that action is measured, and the feedback loop can run once more, every action stimulating new behaviors that inch the individual closer to their goals. (Source Wikipedia)

14. Results from meta-analyses • Results earlier meta-analyses J.S. and co-authors • Results other Meta-analyses • Results recent meta-analysis, Hendriks, Scheerens and Steen, 2011

15. Scheerens et al. 2007 (monitoring & Evaluation at school level) • School self evaluation • School wide performance assessment • School performance feedback • Evaluation of school policies

16. Scheerens et al. 2007 (evaluation & monitoring at classroom level) • Teachers monitoring student work • Monitoring and feedback • Use of evaluative results • Progress monitoring system

17. Results other meta-analyses • Seidel and Shavelson, 2007, report an effect size of .01 (N 84)for evaluation of learning and .02 (N 143)for regulation and monitoring (classroom level) Effect size Fisher’s Z • Creemers and Kyriakides,2008, report an effect size of .13 for evaluation of teaching policies at school level and .18 for assessment • Hattie, 2009, reports much higher effect sizes, see subsequent slides

18. Results John Hattie, 2009

19. Possible explanations of high effect sizes reported by Hattie • Several studies have quite divergent dependent and independent variables • Suggestion In Hattie, 2009, p. 202 that studies did not adjust for student prerequisites • Regional differences, Anglo Saxon versus European studies

20. Results Hattie, minus divergent meta-analyses

21. Results meta-analysis Hendriks at al., 2011(school level) • School self evaluation • School wide performance assessment • School performance feedback • Evaluation of school policies

22. Results meta-analysis Hendriks at al., 2011(evaluation class level) • Teachers monitoring student work • Monitoring and feedback • Use of evaluative results • Progress monitoring system

23. Results meta-analysis Hendriks at al., 2011(Assessment class level) • Use of tests • Frequency of testing • Assessment op pupils • Review of achievement results

24. Discussion of results • Small effects of school and classroom evaluation on student achievement • Somewhat higher effects when feedback is explicitly referred to • Impact of study characteristics; ( higher effect sizes for experimental studies, for language and reading as compared to other subjects and for studies carried out in the USA)

25. Why relatively low impact? (substantive explanations): • Mixed quality of evaluation/assessment instruments • Mixed “completeness” in following PDCA cycle • Unfamiliarity of schools in using tests • Low priority and Resistance

26. How to improve use by schools? • Indicate the ideal situation: characteristics of an evaluation based learning organizations • Optimize supply side, instruments, information provision, professional development and training • Analyze resistance and reasons for non-use by teachers and schools

27. An evaluation based learning organization • certain desirable characteristics of the information (valid, reliable, useable); • certain desirable characteristics concerning the cognitive capacities of the receivers of the information; • facilitating organizational structural conditions, for instance opportunities to discuss the evaluation results, support from the school leader, technical support services; • supportive cultural conditions, i.e. a majority of staff that have favorable attitudes and dispositions and dynamics that maintain and support an overall positive, learning oriented climate.

29. Relevant personality traits

30. Hypotheses: • to the degree that a critical percentage of teachers are externally oriented, and have a low sense of efficacy evaluation results are more likely to be underutilized • to the degree that a critical percentage of teachers show high levels of anxiety, evaluation apprehension and low ambiguity tolerance defensive reactions towards evaluation findings are more likely • to the degree that a critical percentage of teachers is oriented to personal objectives evaluation results are likely to be used as part of power games • to the degree that a critical percentage of teachers is high on introversion unilateral actions are more likely to inhibit collective learning and decision making on the basis of evaluation findings

31. Critical issues for “positive” and “negative” cycles • the “task related” vs the “judgmental” purpose of the school self-evaluation, and providing clear information on this • real or perceived inequalities in the amount of information that is made available at various levels in the hierarchy • the transparency of the evaluative information and the sense of ownership that teachers have with respect to the data

32. Cognitively induced paths to under-utilization • information is global and vague – does not make sense to teachers – is delegated upwards or is caught in a loop of enduring vagueness • feedback lacks an instrumental component – insufficient handles to use at classroom level – non use • feedback is judicial rather than task oriented – defensive reactions –non use/political use

33. Conclusion: important principles for evaluation management • Clarification of the purposes of the (self) evaluative activities, concerning task related and judgmental aspects. • Scaling down the level of ambition with respect to linear and instrumental use of evaluative information. • Breaking inhibiting loops on the basis of higher levels of reflection

34. Overall conclusion • Meta analytic results partly influenced by imperfections of individual studies • Potential of practical applications can be enhanced • “Assessment for School Effectiveness” strong mechanism, no silver bullet, importance of being embedded in other effectiveness enhancing conditions, achievement orientation, instructional leadership and others.