EVAL 6000: Foundations of Evaluation

1. EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2012

2. Agenda • Stage Two theories • Joseph Wholey • Robert Stake • Questions and discussion • Encyclopedia of Evaluation entries

3. “Feedback information about how a program is working to improve its operation, is missing from most local and state evaluation activities” — Joseph S. Wholey

4. Biographical Sketch • Ph.D. in Philosophy and M.A. in Mathematics, Harvard University • B.A. in Mathematics, Catholic University • Professor of Public Policy and Planning at the University of Southern California • Senior Advisor for Evaluation Methodology at GAO • Author of several influential textbooks

5. Wholey’s View of Evaluation • Began under the auspices of truth seeking and experimental methods • His later work emphasized instrumental use of evaluation • Overall, his central concern was producing actionable information aimed at those who are in positions to take decisions • His choice of method was quantitative in orientation

6. Wholey’s Influence • Largely at the federal government level • Likely had an influence on 1993’s Government and Performance Results Act (GPRA) and the later Program Rating Assessment Tool (PART) • Better government through timely, useful information on program performance

7. Wholey’s Major Contributions • Evaluability assessment • Performance-oriented evaluation • Results oriented management • Rapid feedback evaluation • Performance monitoring • Service delivery assessment

8. Wholey’s Theory of Social Programming • Social problems solving through incremental program improvement • Culmination of many small improvements eventually results in an improved ability to solve social problems • Largely a theory of management rather than social policy and social change

9. Wholey’s Theory of Knowledge Construction • Essentially logical positivism—observations of program occur through operationalizing dependent variables • Pragmatism—truth depends on what works in practice (as decided by management) • Knowledge claims are valid if practical consequences satisfy decision makers

10. Wholey’s Theory of Valuing • Valuing is a decision to be made by program management • Value claims are a function of performance measurement, comparison of performance to goals, and the use of performance information for policy-making and program management • Criteria of merit are defined by management

11. Wholey’s Theory of Knowledge Use • Favors instrumental use • Rapid assessment methods provide immediately usable information about programs • Instrumental use can occur in the absence of rigorous methodologies

12. Wholey’s Theory of Evaluation Practice • Sequential purchase of information • Evaluability assessment • Rapid feedback evaluation • Performance monitoring • Intensive evaluation • Encourages flexibility in applying the sequential purchase of information • In sequence and methodology

13. “I like the evaluator more in a role of civil servant than of civil philosopher” — Robert E. Stake

14. Biographical Sketch • Ph.D. in Psychometrics from Princeton University • M.A. and B.A. in Mathematics from University of Nebraska • Emeritus Professor of Education and Director of the Center for Instructional Research and Curriculum Evaluation at the University of Illinois • Recipient of numerous awards and honoraria • Published numerous influential books, journal articles, and reports

15. Stake’s View of Evaluation • Unlike prior (and later) theorists, developed his theory on the premise of “discovery” • Implied that the only realities (i.e., truths) are those that are constructed through a process of discovery • Valuing is socially constructed • The value of a program is best determined by those who experience it • Emphasis was on local stakeholders values • Evaluation should be responsive rather than preordinate

16. Stake’s Influence • Developed widely adopted methods for conducting case studies • Widely considered a pioneer of qualitative methodologies • Responsive evaluation • Recognition of and greater concern for localized evaluation with an emphasis on improving local practice

17. Stake’s Major Contributions • Naturalistic generalization • Case study methodology • Responsive evaluation • Naturalistic inquiry • Direct influence on constructivist conceptualizations of epistemology and ontology

18. Stake’s Theory of Social Programming • Local heterogeneity is of vital importance as incremental changes should be initiated by local stakeholders to meet their own needs • Rejects federal solutions to social problem solving—local problem solving is preferable

19. Stake’s Theory of Knowledge Construction • Responsibility for what constitutes valid knowledge should be given to lay persons (popularization) rather than professionalization (knowledge is the purview of scientists) • Denies that evaluators have a privileged position in constructing knowledge claims

20. Stake’s Theory of Valuing • Singular value claims do not exist • The value of programs is different for different persons, for different purposes • Advocates descriptive valuing by local stakeholders

21. Stake’s Theory of Knowledge Use • Emphasizes enlightenment rather than instrumental or other types of use • Use occurs when vicarious experiences that are intuitively understood occur • Practical versus scientific use

22. Stake’s Theory of Evaluation Practice • Responsive evaluation and case studies are logically distinct since case studies can be done without doing responsive evaluation • One can do responsive evaluation without doing case studies • The choice of responsive evaluation or case studies is defined by purpose

23. Encyclopedia Entries • Cronbach, Lee J. • Design, Evaluation • External Validity • Generalization • Logic Model • Performance-Based Monitoring • Performance Indicator • Planning, Evaluation • Program Logic • Program Theory • Qualitative-Quantitative Debate in Evaluation • Rossi, Peter H. • Theory-Driven Evaluation