EVAL 6000: Foundations of Evaluation

1. EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2011

2. Agenda • Course overview • Introductions • Activity • Brief lecture introducing evaluation theory • Covered in greater detail in coming weeks • Questions and discussion

3. Course Overview • This course is designed to provide an overview of the practice and discipline of evaluation • Comparative study of theory, research, and practice perspectives • Analysis of core concepts and definitions, rationale and uses, the field’s history and standards, alternative models and approaches, and emerging and enduring issues

4. Course Overview • The website for this course is located athttp://www.wmich.edu/evalphd/courses/eval-6000-foundations-of-evaluation/ • From this site you can access • The course syllabus • Supplementary readings • Weekly lecture notes • Other materials related to the course

5. Course Overview • Required textbooks • Alkin, M. C. (Ed.). (2004). Evaluation roots: Tracing theorists’ views and influences. Thousand Oaks, CA: Sage. • Mathison, S. (2005). Encyclopedia of evaluation. Thousand Oaks, CA: Sage. • Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evaluation: Theories of practice. Thousand Oaks, CA: Sage.

6. Course Overview • Primary learning objectives • A deep understanding of a wide array of evaluation theory and practice perspectives • An in-depth understanding of the origins and history of evaluation as well as its evolution toward an independent discipline • A clear understanding of key evaluation concepts/vocabulary/terminology • A clear understanding of the nature and purpose of evaluation, and the distinctions between evaluation, basic and applied research, and related terms such as assessment and diagnosis

7. Course Overview • Primary learning objectives • An ability to describe, distinguish among, and critically evaluate the usefulness and validity of selected models and approaches to evaluation, and to identify the conditions under which each should be used • A firm grasp of the fundamental logic and methodology of evaluation • A basic understanding of how to integrate traditional methodologies with evaluation-specific methodologies • An evaluative and critical thinking mindset, in general

8. Course Overview • Secondary objectives • Conveying constructive criticism in a professional, balanced, and tactful manner • Facilitating discussion to engage others in dialogue about evaluation theory, method, and practice • Writing clearly and concisely for both academic and non-academic audiences • Giving high quality, professional oral presentations for both academic and non-academic audiences

9. Course Overview • Course components • Attendance & class participation 10% • Critical readings papers 30% • Application paper 20% • Thought paper & presentation 20% • Final examination 20%

10. Course Overview • Schedule of topics • Introduction to evaluation theory • Foundational evaluation concepts • Stage one theories & theorists • Stage two theories & theorists • Stage three theories & theorists • Methods-oriented theories & theorists • Valuing-oriented theories & theorists • Use-oriented theories & theorists

11. Course Overview • Weekly meeting structure • If necessary, a question-and-answer session for prior week’s material (30 minutes) • Lecture (60-90 minutes) • At the end of each lecture is a list of entries from the Encyclopedia of Evaluation that you are expected to study (many will appear in the final examination) • Discussion (30 minutes) • Activity (30-45 minutes)

12. Introductions • Who are you? • Why are you here? • What do you expect to learn? • What prior experiences do you have with evaluation? Research? Design? Measurement? Statistics (or analysis in general)? • Who are we?

13. Activity • Draw the first image that comes to mind when you hear the word evaluation (15 minutes) • In small groups share your images, identify common themes, and write your themes on the flipchart (20 minutes) • Share your group’s themes (15 minutes)

14. Meta-Theory and Theory • A meta-theory is a theory whose subject matter is some other theory • In other words, it is a theory about a theory • A theory is a set of interrelated constructs, definitions, and propositions that present a systematic view of phenomena by specifying relations among variables, with the purpose of explaining and predicting phenomena

15. Evaluation Theory • Evaluation theories describe and prescribe what evaluators do or should do when conducting evaluations and are mostly normative in origin • They specify such things as evaluation purposes, users and uses, who participates in the evaluation process and to what extent, general activities or strategies, methods choices, and roles and responsibilities of the evaluator, among others

16. Classification of Theories • Shadish, Cook, and Levition’s five principles that undergird evaluation • Alkin and Christie’s evaluation theory tree that classifies and describes major theorists’ orientation • Stufflebeam and Coryn’s five categories • Scriven and Fournier’s more general ‘logic of evaluation’

17. Program Evaluation Theory • Shadish, Cook, and Levition’s five principles that undergird evaluation • Theory of practice • Theory of knowledge • Theory of valuing • Theory of use • Theory of social programming

18. Evaluation Theory Tree • Alkin and Christie’s theory tree

19. Broad Category Classification • Stufflebeam and Coryn’s classification system for evaluation models and approaches • Pseudoevaluations • Questions- and methods-oriented • Improvement- and accountability-oriented • Social agenda/advocacy • Eclectic

20. Logic of Evaluation • Scriven’s logic of evaluation is the closest to a meta-theory of evaluation • Establishing criteria • On what dimensions must the evaluand do well? • Constructing standards • How well should the evaluand perform? • Measuring performance and comparing with standards • How well did the evaluand perform? • Synthesizing and integrating information/data into a judgment of merit or worth • What is the merit or worth of the evaluand?

21. General Premises • Factual premises • The nature, performance, or impact of an evaluand or evaluee • Roughly equivalent to description (“what’s so?”) • Value premises • The properties or characteristics (i.e., criteria and standards) which typify a good, valuable, or important evaluand or evaluee of a particular class or type in a particular context

22. Value Premises • General values • The merit-defining criteria by which an evaluand or evaluee is evaluated; the properties or characteristics which define a ‘good’ or ‘valuable’ evaluand or evaluee • Specific values • The standards (i.e., levels of performance; usually an ordered set of categories) which are applied and by which performance is upheld, in order to determine if that performance is or is not meritous, valuable, or significant

23. General Logic

24. Working Logic

25. Encyclopedia Entries • Assessment • Accountability • Auditing • Campbell, Donald T. • Cook, Thomas D. • Criteria • Evaluand • Evaluation • Evaluation Theory • External Evaluation • Formative Evaluation • History of Evaluation • Independence • Logic of Evaluation • Objectivity • Scriven, Michael • Shadish, William R. • Standards • Summative Evaluation • Value-free Inquiry • Value Judgment • Values