EVAL 6000: Foundations of Evaluation

1. EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn & Carl D. Westine November 4, 2010

2. Agenda • Eat • American Evaluation Association conference • Hard-won lessons • Theses 1-7 • Finish Part I of evaluation models and approaches (see October 21 lecture notes) • Begin evaluation models and approaches part II • Social agenda and advocacy • Improvement- and accountability-oriented • Eclectic • “Others”

3. American Evaluation Association • Key issues regarding the AEA conference next week • Sign up for at least two time slots at The Evaluation Center and IDPE table • Roles and responsibilities during your slots • Familiarize yourself with all materials • Be able to answer questions (intelligently) • Representing the IDPE while at AEA • How to choose sessions • Getting involved with TIGs • WMU reception on Thursday, November 11, at 7:30 p.m. - 10:00 p.m. at the Grand Hyatt Hotel, Republic A,B,C on the 4th floor. • Questions or concerns?

4. Hard-Won Lessons #1-#7 • Thesis 1: Program evaluation is not a determination of goal attainment • Thesis 2: Program evaluation is not applied social science • Thesis 3: Program evaluation is neither a dominant nor an autonomous field of evaluation • Thesis 4: Side effects are often the main point • Thesis 5: Subject matter expertise may be the right hand of educational and proposal evaluation, but one cannot wrap things up with a single hand • Thesis 6: Evaluation designs without provision for evaluation of the evaluation are unclear on the concept • Thesis 7: An evaluation without a recommendation is like a fish without a bicycle

5. Improvement and Accountability • Fully assess an evaluand’s merit and worth • Expansive and seek comprehensiveness • Consider the full range of questions and criteria to assess and evaluand • Often employ needs assessment as the source of foundational criteria • Look for all relevant outcomes, not just those keyed to objectives

6. Improvement and Accountability • Approach 20: Decision- and accountability-oriented studies • Emphasize that evaluation should be used proactively to improve an evaluand as well as retroactively to judge its value • Founded upon an objectivist philosophy • Stufflebeam’s CIPP model • Context • Input • Process • Product

7. Improvement and Accountability • Approach 21: Consumer-oriented studies • Regards consumer’s welfare as an evaluand’s primary justification and accords that welfare the same primacy in evaluation • Grounded in a deeply reasoned view of ethics and the common good—but also objectivist • Preferred method is independent, external evaluation • Scriven’s Key Evaluation Checklist (KEC) • Values-oriented and -engaged

8. Improvement and Accountability • Approach 22: Accreditation and certification • Studies of institutions, institutional program, and personnel to determine whether they meet requirements of given professions and service authorities and whether they are fit to serve designed functions in society • Usually conducted using guidelines and criteria adopted by some accrediting or certifying body

9. Social Agenda and Advocacy • Aimed at increasing social justice through evaluation • Seek to ensure that all segments of society have equal access to opportunities and services • Advocate affirmative action to give the disadvantaged preferential treatment • Favor constructivist orientation and qualitative methods

10. Social Agenda and Advocacy • Approach 23: Responsive evaluation or client-centered studies • Client-centered • Evaluator works with and for the support of a diverse client group • Continuous interaction, and response to, the evaluative needs of clients and stakeholders • Findings used primarily for improvement

11. Social Agenda and Advocacy • Approach 24: Constructivist evaluation • Grounded in a rejection of the experimental philosophy—deeply paradigm driven • Sometimes known as 4th generation evaluation—because it goes beyond (1) objectives-based, (2) description, and (3) judgment • Intensive participation of stakeholders in the design, conduct, reporting, and application • Emphasis on the “constructions” (i.e., social realities) that different stakeholders bring to bear in assessing an evaluand

12. Social Agenda and Advocacy • Approach 25: Deliberative democratic evaluation • Functions within an explicit democratic framework • Charges evaluators to uphold democratic principles in reaching defensible conclusions • Three main features • Democratic participation • Dialogue to examine and authenticate stakeholder’s inputs • Deliberation to arrive at a defensible assessment of an evaluand’s merit and worth

13. Eclectic • No connection to any particular evaluation philosophy, methodological approach, or social mission • Advanced pragmatic approaches that draw selectively from a wide range of other evaluation approaches • Designed to accommodate needs and preferences of a wide range of clients and evaluation assignments • Unconstrained by a single model or approach

14. Eclectic • Approach 26: Utilization-focused evaluation • Explicitly geared to ensure that evaluations make an impact and are “used” • Evaluation is guided in collaboration with a targeted group of priority users • All aspects are chosen and applied to help targeted users obtain and apply evaluation findings to their intended use and maximize the likelihood that they will • Eclectic because, in the interest of getting findings used, draws on any legitimate evaluation approach

15. Additional Models and Approaches • Participatory evaluation • Stakeholders are engaged in the entire evaluation process (e.g., design, data collection, analysis, reporting) • Assumes that involvement will increase buy-in, credibility, and use

16. Additional Models and Approaches • Goal-free evaluation • Interested only in actual effects and whether impactees’ needs are being met • Ignores (and is blinded to) program goals or intended effects • Rarely practiced