Evaluation in the Learning Organization

1. Evaluation in the Learning Organization Professor Peter Dahler-Larsen, PhD Dept. of Political Science and Public Management University of Southern Denmark

2. An interesting alliance? ?

3. evaluation • Systematic, data-based judgment about public activities and their outcomes, with the intent of influencing practical decisions • Many socio-historical forms and names • Shaped by social conditions of possibility… • Shaping social reality…

4. learning • Rich concept… • Today: popular, should take place all the time, anywhere, ”up for graps” • Cognition + Progression

5. An interesting alliance? SOCIETY ?

6. Evaluation in the learning organization cannot be viewed in isolation from the society in which it is embedded References • Peter Dahler-Larsen (2008): Kvalitetens Beskaffenhed. Syddansk Universitetsforlag 2008 • Peter Dahler-Larsen (2008): Learning-oriented educational evaluation in contemporary society. In: Cousins and Ryan: Handbook of Educational Evaluation. Sage (in press) • Peter Dahler-Larsen (2007): Constitutive effects of performance indicator systems. In: S. Kushner (ed.): Dilemmas of engagement: Evaluation development under NPM and the New Politics. New York: Elsevier. • Peter Dahler-Larsen (2008): Intern og læringsorienteret evaluering. Kognition og Pædagogik, nr. 67 (pp. 12-19).

7. Outline • Evaluation in the Learning Organization • The ideal and its limitations • Evaluation in Reflexive Modernization • Evaluation in the Audit Society

8. Four Purposes of Evaluation • Control/accountability • Learning/development • Increasing knowledge base • Informing the public

9. Organizational learning Evaluation learning

10. Characteristics of learning organizations • A precise language to describe problems • A tolerance for error • Open communication channels • Multiple perspectives • Shared understandings • Curiosity • Experiments • Appropriate pace of change • Cultural assumptions can be challenged (2. order) • Integrated learning cycle • …

11. The ideal… • The conditions for learning + • Continuous self-observation and reflection => • Continuous organizational self-transformation (collective, cognition, progression ie org learning)

12. Hope: And when they see THAT response, the need for external control will disappear!

13. Experiences with learning-oriented evaluation in educational organizations • Time pressure? • Technical/methodological quality • Organizational structures, cultures etc. • Political conflicts

14. Critical limitations of the project of learning-oriented evaluation • deterioration of evaluation? • deterioration of learning? • Empirical? • Progression? • The normative ideal of never-ending flexibility • internal/external divide

15. Evaluation: What do teachers think they are good at? (N=480)

16. Normative mythology

17. Normative mythology

18. Forms of internal/external evaluation

19. Combinations of internal/external evaluation

20. Combinations of internal/external evaluation

21. Combinations of internal/external evaluation

22. Internal = formative?external = summative? • ”When the cook tastes the soup, that´s formative” • ”When the guest tastes the soup, that´s summative”

26. Three views of formative/summative relation • In one evaluation: mutually exclusive

27. Three views of formative/summative relation • In one evaluation: mutually exclusive • In one evaluation: both possible

28. Three views of formative/summative relation • In one evaluation: mutually exclusive • In one evaluation: both possible • Society: both relevant

29. Central control Local learning

30. Central control Local learning

31. Central control Knowledge based Not knowledge based Local learning

32. Central control Knowledge based Not knowledge based Local learning

33. Outline • Evaluation in the Learning Organization • The ideal • The limitations of the ideal • Evaluation in Reflexive Modernization • Evaluation in the Audit Society

34. Reflexive modernization • In an era in which • Progress has been replaced by development; • Side-effects • And multiple unforeseen reactions are the order of the day • Many, centerless reflexivities exist

35. In reflexive modernization, a learning approach must • ”internalize” side effects • acknowledge the role of the public sphere • live with multiple interpretations (some of which are more knowledge-based than others) • understand knowledge as socially productive

36. Outline • Evaluation in the Learning Organization • The ideal • The limitations of the ideal • Evaluation in Reflexive Modernization • Evaluation in the Audit Society

37. The audit society • From ”evaluability assessment” • To ”Evaluation systems”

38. From evaluability assessment… • Is the program well defined? • Clear goals? • Clear program theory? • Well implemented? • Data within reach? • Managerial and professional space?

39. …to evaluation systems • Evaluation is obligatory • Evaluation covers all activities within a given social domain • Evaluation is integrated, abstract, and organizational

40. Why evaluation systems? • ad hoc evaluation is unpredictable • reduction of complexity • risk management

41. Organizational learning Evaluation learning Evaluation capacity Evaluation culture Evaluation system

42. Evaluation capacity Evaluation culture Evaluation system Management ”production”

43. External audit Accreditation Certification Quality control … … Evaluation capacity Evaluation culture Evaluation system Management ”production”

44. Evaluation systems • Abstraction • Resources • De-professionalization • Constitutive effects

45. ”Quality control procedures may function less to make quality observable and more to construct and define quality itself” Michael Power

46. strategies • Research on evaluation systems • Intelligent accountability? • Negotiate • Go public • ”Work the system”