Program Evaluation Research Method

1. Program Evaluation Research Method Kermelle Hensley Larry Lipscomb Terri Wyrosdick

2. Overview: Program Evaluation Research Method • Program evaluation is research designed to assess the implementation and effects of a program. • Its purposes vary and can include • (1) program improvement, • (2) judging the value of a program, • (3) assessing the utility of particular components of a program, and • (4) meeting accountability requirements

3. Results of program evaluations are often used for decisions about whether to continue a program, improve it, institute similar programs elsewhere, allocate resources among competing programs, or accept or reject a program approach or theory. • Through these uses program evaluation is viewed as a way of rationalizing policy decision-making.

4. Who Uses program Evaluation Research? • Program evaluation is conducted for a wide range of programs, from broad social programs such as welfare, to large multisite programs such as the preschool intervention program Head Start, to program funding streams such as the U.S. Department of Education's Title I program that gives millions of dollars to high-poverty schools, to small-scale programs with only one or a few sites such as a new mathematics curriculum in one school or district.

5. Scientific Research vs. Evaluation • There has been some debate about the relationship between "basic" or scientific research and program evaluation. For example, in 1999 Peter Rossi, Howard Freeman, and Michael Lipsey described program evaluation as the application of scientific research methods to the assessment of the design and implementation of a program. • In contrast, Michael Patton in 1997 described program evaluation not as the application of scientific research methods, but as the systematic collection of information about a program to inform decision-making.

6. in many circumstances the design of a program evaluation that is sufficient for answering evaluation questions and providing guidance to decision-makers would not meet the high standards of scientific research. • Further, program evaluations are often not able to strictly follow the principles of scientific research because evaluators must confront the politics of changing actors and priorities, limited resources, short timelines, and imperfect program implementation.

7. Program evaluations must be designed to maximize the usefulness for decision-makers, whereas scientific research does not have this constraint. • Both types of research might use the same methods or focus on the same subject, but scientific research can be formulated solely from intellectual curiosity, whereas evaluations must respond to the policy and program interests of stakeholders.

8. Negative Impact • Although the main justification for program evaluation is its role in rationalizing policy, program evaluation results rarely have a direct impact on decision-making. • This is because of the political nature of policy decision-making and because people are generally resistant to change.

9. While program evaluations may not directly affect decisions, evaluation does play a critical role in contributing to the discourse around a particular program or issue. • Information generated from program evaluation helps to frame the policy debate by bringing conflict to the forefront, providing information about trade-offs, influencing assumptions and beliefs underlying policies, and changing the way people think about a specific issue or problem.

10. How to conduct a Program Evaluation

11. Paradigms in program evaluation • Positivist Approach – (most common) can be quantitatively measured • Interpretive Approaches – can be qualitatively measured • Critical-Emancipatory Approaches – based on action research for the purpose of social transformation (used in politics)

12. Methods to collect information • Questionnaires, surveys, and checklists • Interviews • Documentation review • Observation • Focus groups • Case studies

13. Four levels of evaluation • Reactions and feelings • Learning • Changes in skills • Effectiveness Usually, the farther you evaluation information gets down the list, the more useful is your evaluation.

14. Major Types of Program Evaluation • Goals-Based Evaluations – evaluate the extent to which programs are meeting predetermined goals or objectives • Process-Based Evaluations – geared to understanding how a program works – how does it produce the result that it does • Outcome Base Evaluations – evaluates if the program is the right program to bring about the outcome needed United Way’s website provides an overview for their outcome-based evaluation (next slide)

15. http://www.unitedway.org/outcomes/ Outcome Measurement Resource Network Welcome to United Way of America's Outcome Measurement Resource Network. As leaders in results-oriented philanthropy, United Way of America and United Way organizations across the country have championed the adoption of outcome measurement by health and human service programs. The Resource Network offers information, downloadable documents, and links to resources related to the identification and measurement of program- and community-level outcomes. The Outcome Measurement Resource Network has 5 sections: Outcome Measurement: What and Why?Introduction to what outcome measurement is and is not, key terms, and the rationale for and benefits of engaging in outcome measurementUWA Outcome Measurement Resources Descriptions, excerpts or downloadable copies of United Way of America outcome measurement resourcesWho is Involved in Outcome Measurement?Selected outcome and performance measurement initiatives within the United Way system; national health, human service, and youth-serving organizations; public sector agencies; and other nonprofit organizationsResource LibraryFull-text resources and links on program outcome measurement, community-level initiatives, and related topics Frequently Asked Questions