EVAL 6000: Foundations of Evaluation

EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2012

Agenda • Stage One theories • General characteristics • Michael Scriven • Questions and discussion • Encyclopedia of Evaluation entries

General Characteristics of Stage One Theories • Stage One theories can be characterized as a period of social problem solving • Emphasis was on scientific rigor and valid causal knowledge of the effects of social programs • Reflects the tenor of the 1960s, a period of rapid development in the social sciences

Stage One Theory of Social Programming • Programs largely viewed as immediately implementable solutions to social problems • Rested on assumptions that • Implementation and evaluation would be unambiguous • “Successes” would be adopted • A and B combined, would lead to ameliorating social problems • Concerned primarily with innovations/pilots/demonstrations

Stage One Theory of Knowledge Construction • Priority was given to “truth” • The dominant view was that an external reality probably exists, though it may never be perfectly known • Valid knowledge could be constructed (more or less) • Emphasis on sources of bias and the best methods for reducing them

Stage One Theory of Valuing • Theorists at this stage agree that valuable social programs solve important social problems • They disagree about how value claims are made and whose values should be given priority • Prescriptive versus descriptive valuing

Stage One Theory of Knowledge Use • Dominant assumption was “naïve instrumentalism” • Effective programs would be maintained or scaled-up and ineffective ones would be eliminated (i.e., a rationalist view) • Use would occur almost naturally • Evaluation findings, being scientifically rigorous, would inform decision making

Stage One Theory of Evaluation Practice • Emphasis on assessing a program’s effectiveness at solving important social problems • Primary task is summative claims or judgments • Methods of arriving at summative statements were given priority • Distance is crucial to scientific rigor and integrity

“Bad is bad and good is good and it is the job of evaluators to decide which is which” — Michael Scriven

Biographical Sketch • Born in 1928, in Bealieu, Hampshire, England • Ph.D. in Philosophy, Oxford University • M.A. in Mathematics and Philosophy, Melbourne University • Former President of AERA and AEA • Author or more than 300 publications • Recipient of numerous awards and prizes

Scriven’s View of Evaluation • Scriven argues that a singular statement of value is the sine qua non of evaluation and is the primary task of evaluators • Evaluation is an autonomous discipline, and a vital component of all other disciplines

Scriven’s Influence • Scriven is widely viewed as the first major evaluation theorist • Whether one agrees or disagrees with his views, Scriven has directly and indirectly influenced nearly every other theorist and theory

“Not to know Scriven is to be illiterate in evaluation” — Thomas D. Cook

Scriven’s Major Contributions • Definition of evaluation • Valuing theory • Formative and summative distinction • Goal-free evaluation • Causal logic and cause-probing methods • Logic of evaluation • Theory-free evaluation • Metaevaluation • Key Evaluation Checklist

Goal-Free Evaluation • Premised on examining what effects a program is actually having rather than intended effects • Aimed at bias reduction through avoiding knowledge of program goals • Relies on needs assessment to determine the extent to which a program meet needs

Causal Logic and Reasoning • Modus operandi method (MOM) and general elimination method (GEM) for causal analysis • Causal claims ‘beyond a reasonable doubt’ should be the standard within and outside of science, as opposed to those supported by ‘the balance of the evidence’ • Overdetemination problem

Theory-Free Evaluation • The role of evaluators is to determine only whether programs work, not to explain how they work • To attempt explanations is beyond the capability of most evaluators • Requires the correct (not just the believed) logic or theory of the program

Metaevaluation • An evaluation of an evaluation • Internal and external quality control • Identify strengths, weaknesses, and uses • Appropriate criteria • Validity • Credibility • Utility • Cost-effectiveness • Ethicality

Key Evaluation Checklist • Organized into four parts with 15 checkpoints • Part A: Preliminaries • Part B: Foundations • Part C: Subevaluations • Part D: Conclusions & Implications

Scriven’s Theory of Social Programming • Social programs exist to meet important societal needs • Consumer needs should be given the same primacy in an evaluation • Good programs should be expanded and poor programs should be abandoned

“If I had asked people what they wanted, they would have said faster horses” — Henry Ford

Scriven’s Theory of Knowledge Construction • Realist philosophy and ideology • Valid knowledge is possible, but nearly always incomplete • Practical inference rather than the development of scientific theories about social programs • Socio-organizational controls to reduce bias in knowledge claims

Scriven’s Theory of Valuing • Prescriptive theory premised on some values having higher priority than others • Valuing is no less scientific in its validity than “value-free” science • Relative valuing, rather than absolute valuing, is preferable

Scriven’s Theory of Knowledge Use • Use is “self-evident” • “Naïve instrumentalism” • Primarily concerned with summative statements emphasizing short-term instrumental use

“Evaluation is what it is, the determination of merit and worth, and what it is used for is another matter” — Michael Scriven

Scriven’s Theory of Evaluation Practice • Methodologically pluralistic • Audience of evaluation is predominately “consumers” • Evaluations should serve the “public good” • Emphasizes bias minimization through socio-organizational controls

Encyclopedia Entries • Bias • Causation • Checklists • Chelimsky, Eleanor • Conflict of Interest • Countenance Model of Evaluation • Critical Theory Evaluation • Effectiveness • Efficiency • Empiricism • Independence • Evaluability Assessment • Evaluation Use • Fournier, Deborah • Positivism • Relativism • Responsive evaluation • Stake, Robert • Thick Description • Utilization of Evaluation • Weiss, Carol • Wholey, Joseph

EVAL 6000: Foundations of Evaluation