Program Evaluation In A Nutshell

1. Program Evaluation In A Nutshell Jonathan Brown, M.A.

2. Two Types of Evaluation • At the most basic level there are two types of evaluation: • Formative: primary purpose is to provide information to improve a program. • Summative: concerned with providing information about program adoption, continuation, or expansion.

3. Two Types of Evaluation Formative Evaluation Relative Emphasis Summative Evaluation Program Life

4. What Are You Assessing? • The evaluation plan must consider the situation or context of the evaluation. • The purpose of the evaluation. • Improvement (formative evaluation). • Accountability (summative evaluation). • Knowledge generation. • Hidden agendas.

5. What Are You Assessing? • The evaluation plan must consider the situation or context of the evaluation. • The program’s structure and context. • The stage of program development. • Administrative and political context. • Conceptual and organizational structure of the program.

6. What Are You Assessing? • The evaluation plan must consider the situation or context of the evaluation. • The resources available for the evaluation. • Personnel • Equipment • Facilities to support data collection, analyses, and reporting. • Is specialised expertise needed?

7. What Are You Assessing? • Typically, evaluation involves assessing one or more of the five following areas: • The need for a program. • The program’s design. • The program’s implementation or process. • The program’s impacts or outcomes. • The efficiency of a program.

8. Needs Assessment • Purpose of a social program is to alleviate a social problem. • Needs assessments assess: • The nature, extent, and distribution of a social problem. • How these features will impact the design of an intervention program.

9. Needs Assessment • These assessments can be conducted when… • Planning a new program. • Restructuring an existing program. • Determining if an existing program is responsive to the current needs of the target population.

10. Assessing Program Theory • Program theory evaluation is an evaluation guided by a program’s explicit theory of how it causes intended outcomes. • Program theory outlines the following: • Target population • Resources • Activities • Outcomes • A program’s theory is represented by Theory of Change Models (TCMs) and Program Logic Models (PLMs).

11. Assessing Program Theory Conditions before the project The Program is a Conditions after the project • With no information about the program, it remains a mysterious black box. • The evaluation will also not be able to attribute the final conditions to the specific aspects of the program.

12. Assessing Program Theory • Assessing program theory can be done by applying SMART principles (Knowlton & Phillips, 2009). • Specific – the program is clear enough to implement and evaluate. • Measurable – indicators can be qualified and/or quantified. • Action oriented – activities will provoke the desired change in targets. • Realistic – the program is plausible and feasible. • Timed – duration of activities and intended outcomes are specified.

13. Process Evaluation • Given that a valid theory on how to intervene on a diagnosed social problem has been done, the program must be implemented well to be successful. • Two forms: • Process evaluation • Program monitoring

14. Process Evaluation • Process evaluation looks at service utilization and program organization. • Service Utilization • The extent to which the intended targets receive services. • Important when participation is voluntary, or participants must learn new procedures, or habits. • Coverage • Bias

15. Process Evaluation • Process evaluation looks at service utilization and program organization. • Program Organization • How well the program is in terms of managing its efforts and using its resources to accomplish its activities. • Service delivery: • None or not enough intervention • Wrong intervention • Unstandardized intervention

16. Outcome Evaluation • Also known as an impact assessment. • Assesses the extent to which a program produces the outcomes it intends. • But the desired changes could be caused by factors unrelated to the program. • Interested in the changes produced by the program above and beyond any external elements.

17. Outcome Evaluation • Program outcomes • The states or conditions of the target that changed as a result of a program. • Observed characteristics of the target population or social conditions. • Occur to different degrees.

18. Outcome Evaluation • Program outcomes • Outcome level – status of an outcome at a point in time. • Outcome change – difference between outcome levels at different points in time. • Program effect – the unique portion of an outcome that can be attributed to only the program.

19. Outcome Evaluation Outcome status with program Post-program outcome level Program effect Outcome Change Outcome Variable Outcome status without program Pre-program outcome level Before Program During Program After Program (Rossi, Lipsey, & Freeman, 2004)

20. Efficiency Assessment • Due to limited resources, program accomplishments must be judged against program costs. • Two types of efficiency assessments: • Cost-benefit analysis • Cost-effectiveness analysis • Such assessments require placing a dollar amount on program activities and benefits.

21. Typical Evaluation Questions • Needs Assessment • What is the nature and magnitude of the problem? • What are the characteristics of the population in need? • What are the needs of the population? • What services are needed? • How much service is needed and over what period of time? • What service delivery arrangements are needed to provide services to the population?

22. Typical Evaluation Questions • Program Theory Assessment • Is the program doing the right thing? • How could the program better address the needs of participants? • How should the program be organized? • What are the best delivery systems for the services? • Is the program specific enough to measure?

23. Typical Evaluation Questions • Process Evaluation • How many persons are receiving services? • Are administrative and service objectives being met? • Are the intended targets receiving services? • Are the necessary program functions being performed adequately? • Are resources being used effectively and efficiently?

24. Typical Evaluation Questions • Outcome Evaluation • Are the goals and outcomes being achieved? • Do the services have adverse side effects? • Are some participants affected more by the services than others? • Is the problem the services are intended to address made better?

25. Typical Evaluation Questions • Efficiency Assessment • Are resources used efficiently? • Is the cost reasonable in relation to the magnitude of the benefits? • Would alternative approaches yield equivalent benefits at less cost?

26. QUESTIONS? Jonathan Brown, M.A. jbrown@asc4solutions.com