Decision-Oriented Evaluation Approaches

1. Decision-Oriented Evaluation Approaches Presenters: Chris, Emily, Jen & Marie

2. Why did decision-oriented evaluation approaches emerge? • Decision-oriented evaluations emerged in the 1970’s • Due to the fact that the evaluations were being ignored and had no impact • Evaluators wanted to help people who make decisions actually make decisions

3. CIPP Program • CIPP Program is the concept Mr. Stufflebeam • Most staying evaluation model • “Evaluation’s most important purpose is not to prove but to improve” pg. 176 • Core Values - ideals held by a society that must be taken into consideration pg. 177

4. C. I. P. P. • C - Context Evaluation: What is the desired goal? • I - Input Evaluation: How are you going to implement the program? • P - Process Evaluation: How can you improve/modify the process? • P - Product Evaluation: Did it work? What now? • CIPP uses both formative and summative evaluations

5. Developmental Stages of a Program(p. 174-176) • Focusing the Evaluation • Identify the major levels of decision making. • For each level, project the decision situations to be served and describe each one in terms of locus, focus, criticality, timing, and composition of alternatives. • Define criteria for each decision situation by identifying variables to aide in judging alternatives. • Define policies the evaluator must operate within.

6. Developmental Stages of a Program • Collection of Information • Specify: • Source of information • Instruments and methods of collecting information • Sampling procedure • Conditions and schedule for collecting information • Organization of Information • Provide a format for the collected information. • Designate a means for performing the analysis.

7. Developmental Stages of a Program • Analysis of Information • Select the analytical procedures. • Designate a means for performing the analysis. • Reporting of Information • Define the audience. • Specify the means for providing the information. • Specify the format. • Schedule the report.

8. Developmental Stages of a Program • Administration of the Evaluation • Summarize the evaluation schedule. • Define staff and resource requirements and plans for meeting those requirements. • Evaluate the potential for providing information that is valid, reliable, credible, timely, and pervasive. • Specify and schedule the means for periodic updating. • Provide a budget.

9. How can decision-oriented evaluation help at each stage? • Stages are advantageous to the user because it forces reflection • Evaluators must think of evaluation as cyclical • Similar to formative assessment in education

10. What is the Personal Factor? • According to Patton it is the: • identification of “one or more stakeholders who care about the evaluation and are in a position to use it” (p 179). • “presence of an identifiable individual or group of people who personally care about the valuation and the findings it generates” (p 179). • central element of Utilization-Focused Evaluation (UFE)

11. What are the key factors of UFE? 1) Identify the intended users • Consider interest in the study • Consider power in program to be evaluated • Ideally – both would be high • However, interest is more important than power

12. What are the key factors of UFE? 2) Involving primary users in the conduct of the study • Help to identify focus of study, how to use info obtained, design of study, and data collection • Ensures they have a thorough understanding of the methodology 3) Involve primary users in interpreting results and making decisions about judgments, recommendations, and dissemination

13. Advantages of Decision-Oriented Evaluation • Oldest and still frequently used. • Still being written about and used as guides to design individual evaluations or evaluation systems. • Use various means. • Stages of development, personal factor, consider ongoing information needs • Provide information that helps people make decisions.

14. Disadvantages of Decision-Oriented Evaluation • Focus on decisions • Tend to neglect stakeholders with less power • Do not address social equity and equality • Focus on managers and their information needs • Evaluator can become the manager’s “hired gun” • Occasionally, the evaluator can not respond to critical issues if they do not match the issues of the decision maker. • Assume that the important decisions and information to make them can be clearly defined in advance and the program will remain stable while the evaluation is conducted.

15. What are the strengths and limitations of individual approaches?