Facilitated by Michael Bamberger and Jim Rugh

1. Promoting a RealWorld and Holistic approach to Impact EvaluationDesigning Evaluations under Budget, Time, Data and Political ConstraintsAEApre-conference professional development workshopMinneapolis, October 24, 2012 Facilitated by Michael Bamberger and Jim Rugh Note: This PowerPoint presentation and the summary (condensed) chapter of the book are available at: www.RealWorldEvaluation.org

2. Workshop Objectives 1. The basics of the RealWorld Evaluation approach for addressing common issues and constraints faced by evaluators such as: when the evaluator is not called in until the project is nearly completed and there was no baseline nor comparison group; or where the evaluation must be conducted with inadequate budget and insufficient time; and where there are political pressures and expectations for how the evaluation should be conducted and what the conclusions should say;

3. Workshop Objectives • Identifying and assessing various design options that could be used in a particular evaluation setting; • Some tips on developing logic models; • Ways to reconstruct baseline data; • How to account for what would have happened without the project’s interventions: alternative counterfactuals; • The advantages of using mixed-methods designs; • Considerations for more holistic approaches to impact evaluation.

4. Workshop Objectives Note: In this workshop we focus on project-level impact evaluations. There are, of course, many other purposes, scopes, evaluands and types of evaluations. Some of these methods may apply to them, but our examples will be based on project impact evaluations, most of them in the context of developing countries.

5. Workshop agenda: morning 8:00-8:30 Introduction + Brief overview of the RealWorld Evaluation approach 8:30-9:00 Basic evaluation design scenarios Jim 29-48 9:00-9:30 Small group discussion: personal introductions, including RWE-type constraints you have faced in your own practice 9:30-9:50 break 9:50-10:20 Logic models; reconstructing baselines Jim 51- 77 10:20-10:40 Mixed Method evaluations Michael 78-102 10:40-11:00 Alternative counterfactuals Jim 103-115 11:00-12:00 Small group exercise, Part I: Reading of case studies then preparing an evaluation design when working under budget, time, data or political constraints 12:00-1:00 lunch

6. Workshop agenda: afternoon 1:00-1:45 Impact Evaluations Michael 120-134, Jim 135-155 1:45-2:30 Small group exercise, Part II: 'Clients’ and ‘consultants’ re-negotiate the case study evaluation ToR. Note: In addition to becoming familiar with some of the RWE approaches, the case study exercises will also illustrate the different evaluation agendas and perspectives of evaluation consultants, project implementers and funding agencies. 2:30-2:50 Plenary discussion of main learnings from group exercises and the RealWorld Evaluation approach. Jim 156-159 2:50-3:00 Wrap-up and evaluation of workshop by participants

7. RealWorld EvaluationDesigning Evaluations under Budget, Time, Data and Political Constraints OVERVIEW OF THE RWE APPROACH

8. Typical RealWorld Evaluation Scenario Evaluator(s) not brought in until near end of project For political, technical or budget reasons: • There was no life-of-project evaluation plan • There was no baseline survey • Project implementers did not collect adequate data on project participants at the beginning nor during the life of the project • It is difficult to collect data on comparable control groups

9. Reality Check – Real-World Challenges to Evaluation • All too often, project designers do not think evaluatively – evaluation not designed until the end • There was no baseline – at least not one with data comparable to evaluation • There was/can be no control/comparison group. • Limited time and resources for evaluation • Clients have prior expectations for what they want evaluation findings to say • Many stakeholders do not understand evaluation; distrust the process; or even see it as a threat (dislike of being judged)

10. RealWorld Evaluation Quality Control Goals • Achieve maximum possible evaluation rigor within the limitations of a given context • Identify and control for methodological weaknesses in the evaluation design • Negotiate with clients trade-offs between desired rigor and available resources • Presentation of findings must acknowledge methodological weaknesses and how they affect generalization to broader populations

11. The Need for the RealWorld Evaluation Approach • As a result of these kinds of constraints, many of the basic principles of rigorous impact evaluation design (comparable pre-test -- post test design, control group, adequate instrument development and testing, random sample selection, control for researcher bias, thorough documentation of the evaluation methodology etc.) are often sacrificed.

12. The RealWorld Evaluation Approach An integrated approach to ensure acceptable standards of methodological rigor while operating under real-world budget, time, data and political constraints. This is the cover of the cover of the 1se edition of the RealWorld Evaluation book (2006)

13. Bamberger Rugh Mabry RealWorld Evaluation RealWorld Evaluation Working Under Budget, Time, Data, and Political Constraints Michael Bamberger Jim Rugh Linda Mabry EDITION 2 EDITION This book addresses the challenges of conducting program evaluations in real-world contexts where evaluators and their clients face budget and time constraints and where critical data may be missing. The book is organized around a seven-step model developed by the authors, which has been tested and refined in workshops and in practice. Vignettes and case studies—representing evaluations from a variety of geographic regions and sectors—demonstrate adaptive possibilities for small projects with budgets of a few thousand dollars to large-scale, long-term evaluations of complex programs. The text incorporates quantitative, qualitative, and mixed-method designs and this Second Edition reflects important developments in the field over the last five years. New to the Second Edition: • Adds two new chapters on organizing and managing evaluations, including how to strengthen capacity and promote the institutionalization of evaluation systems • Includes a new chapter on the evaluation of complex development interventions, with a number of promising new approaches presented • Incorporates new material, including on ethical standards, debates over the “best” evaluation designs and how to assess their validity, and the importance of understanding settings • Expands the discussion of program theory, incorporating theory of change, contextual and process analysis, multi-level logic models, using competing theories, and trajectory analysis • Provides case studies of each of the 19 evaluation designs, showing how they have been applied in the field “This book represents a significant achievement. The authors have succeeded in creating a book that can be used in a wide variety of locations and by a large community of evaluation practitioners.” —Michael D. Niles, Missouri Western State University “This book is exceptional and unique in the way that it combines foundational knowledge from social sciences with theory and methods that are specific to evaluation.” —Gary Miron, Western Michigan University “The book represents a very good and timely contribution worth having on an evaluator’s shelf, especially if you work in the international development arena.” —Thomaz Chianca, independent evaluation consultant, Rio de Janeiro, Brazil 2 EDITION RealWorld Evaluation 2nd edition published January 2012

14. The RealWorld Evaluation approach • Developed to help evaluation practitioners and clients • managers, funding agencies and external consultants • Still a work in progress (we continue to learn more through workshops like this) • Originally designed for developing countries, but equally applicable in industrialized nations

15. Most RealWorld Evaluation tools are not new— but promote a holistic, integrated approach • Most of the RealWorld Evaluation data collection and analysis tools will be familiar to experienced evaluators. • What we emphasize is an integrated approach which combines a wide range of tools adapted to produce the best quality evaluation under RealWorld constraints.

16. What is Special About the RealWorld Evaluation Approach? • There is a series of steps, each with checklists for identifying constraints and determining how to address them • These steps are summarized on the following slide and in the more detailed flow-chart on page 5 of the Condensed Overview.

17. The Steps of the RealWorld Evaluation Approach Step 1: Planning and scoping the evaluation Step 2: Addressing budget constraints Step 3: Addressing time constraints Step 4: Addressing data constraints Step 5: Addressing political constraints Step 6:Assessing and addressing the strengths and weaknesses of the evaluation design Step 7: Helping clients use the evaluation See page 5 of the Condensed Summary for more details

18. Planning and Scoping the Evaluation • Understanding client information needs • Defining the program theory model • Preliminary identification of constraints to be addressed by the RealWorld Evaluation

19. Understanding client information needs Typical questions clients want answered: • Is the project achieving its objectives? • Is it having desired impact? • Are all sectors of the target population benefiting? • Will the results be sustainable? • Which contextual factors determine the degree of success or failure?

20. Understanding client information needs A full understanding of client information needs can often reduce the types of information collected and the level of detail and rigor necessary. However, this understanding could also increase the amount of information required!

21. Still part of scoping: Other questions to answer as you customize an evaluation Terms of Reference (ToR): • Who asked for the evaluation? (Who are the key stakeholders)? • What are the key questions to be answered? • Will this be mostly a developmental, formative or summative evaluation? • Will there be a next phase, or other projects designed based on the findings of this evaluation? 21

22. Other questions to answer as you customize an evaluation ToR: • What decisions will be made in response to the findings of this evaluation? • What is the appropriate level of rigor? • What is the scope / scale of the evaluation / evaluand (thing to be evaluated)? • How much time will be needed / available? • What financial resources are needed / available? 22

23. Other questions to answer as you customize an evaluation ToR: • Should the evaluation rely mainly on quantitative or qualitative methods? • Should participatory methods be used? • Can / should there be a household survey? • Who should be interviewed? • Who should be involved in planning / implementing the evaluation? • What are the most appropriate forms of communicating the findings to different stakeholder audiences? 23

24. Before we return to the RealWorld steps, let’s gain a perspective on levels of rigor, and what a life-of-project evaluation plan could look like 24

25. Different levels of rigor depends on source of evidence; level of confidence;use of information Objective, high precision – but requiring more time & expense Quick & cheap – but subjective, sloppy Level 5: A very thorough research project is undertaken to conduct in-depth analysis of situation; P= +/- 1% Book published! Level 4: Good sampling and data collection methods used to gather data that is representative of target population; P= +/- 5%Decision maker reads fullreport Level 3: A rapid survey is conducted on a convenient sample of participants; P= +/- 10%Decision maker reads 10-page summary of report Level 2: A fairly good mix of people are asked their perspectives about project; P= +/- 25%Decision maker reads at least executive summary of report Level 1: A few people are asked their perspectives about project; P= +/- 40%Decision made in a few minutes Level 0:Decision-maker’s impressions based on anecdotes and sound bytes heard during brief encounters (hallway gossip), mostly intuition; Level of confidence +/- 50%; Decision made in a few seconds 25

26. CONDUCTING AN EVALUATION IS LIKE LAYING A PIPELINE Randomized sample selection Reliability & validity of indicators Questionnaire quality Depth of analysis Reporting & utilization Quality of data collection QUALITY OF INFORMATION GENERATED BY AN EVALUATION DEPENDS UPON LEVEL OF RIGOR OF ALL COMPONENTS

27. Relevance & validity of indicators Randomized sample selection Questionnaire quality Depth of analysis Reporting & utilization Quality of data collection AMOUNT OF “FLOW” (QUALITY) OF INFORMATION IS LIMITED TO THE SMALLEST COMPONENT OF THE SURVEY “PIPELINE”

28. High rigor 4 3 2 Low rigor Time during project life cycle Determining appropriate levels of precision for events in a life-of-project evaluation plan Same level of rigor Final evaluation Baseline study Mid-term evaluation Special Study Needs assessment Annual self-evaluation 28

29. RealWorld EvaluationDesigning Evaluations under Budget, Time, Data and Political Constraints EVALUATION DESIGNS

30. Some of the purposes for program evaluation • Formative: learning and improvement including early identification of possible problems • Knowledge generation: identify cause-effect correlations and generic principles about effectiveness. • Accountability: to demonstrate that resources are used efficiently to attain desired results • Summative judgment: to determine value and future of program • Developmental evaluation: adaptation in complex, emergent and dynamic conditions -- Michael Quinn Patton, Utilization-Focused Evaluation, 4th edition, pages 139-140

31. Determining appropriate (and feasible) evaluation design Based on the main purpose for conducting an evaluation, an understanding of client information needs, required level of rigor, and what is possible given the constraints, the evaluator and client need to determine what evaluation design is required and possible under the circumstances.

32. Some of the considerations pertaining to evaluation design 1: When evaluation events take place (baseline, midterm, endline) 2. Review different evaluation designs (experimental, quasi-experimental, other) 3. Determine what needs to be done to “fill in the missing pieces” when the “ideal” evaluation design/scenario is not feasible.

33. scale of major impact indicator An introduction to various evaluation designs Illustrating the need for quasi-experimental longitudinal time series evaluation design Project participants Comparison group baseline end of project evaluation post project evaluation 33

34. OK, let’s stop the action to identify each of the major types of evaluation (research) design … … one at a time, beginning with the most rigorous design. 34

35. First of all: the key to the traditional symbols: • X = Intervention (treatment), I.e. what the project does in a community • O = Observation event (e.g. baseline, mid-term evaluation, end-of-project evaluation) • P (top row): Project participants • C (bottom row): Comparison (control) group Note: the 7 RWE evaluation designs are laid out on page 8 of the Condensed Overview of the RealWorld Evaluation book 35

36. Design #1: Longitudinal Quasi-experimental P1 X P2 X P3 P4 C1 C2 C3 C4 Project participants Comparison group baseline midterm end of project evaluation post project evaluation 36

37. Design #2: Quasi-experimental (pre+post, with comparison) P1 X P2 C1 C2 Project participants Comparison group baseline end of project evaluation 37

38. Design #2+: Randomized Control Trial P1 X P2 C1 C2 Project participants Research subjects randomly assigned either to project or control group. Control group baseline end of project evaluation 38

39. Design #3: Truncated Longitudinal X P1 X P2 C1 C2 Project participants Comparison group midterm end of project evaluation 39

40. Design #4: Pre+post of project; post-only comparison P1 X P2 C Project participants Comparison group baseline end of project evaluation 40

41. Design #5: Post-test only of project and comparison X P C Project participants Comparison group end of project evaluation 41

42. Design #6: Pre+post of project; no comparison P1 X P2 Project participants baseline end of project evaluation 42

43. Design #7: Post-test only of project participants X P Project participants • Need to fill in missing data through other means: • What change occurred during the life of the project? • What would have happened without the project (counterfactual)? • How sustainable is that change likely to be? end of project evaluation 43

44. See Table 2.2 on page 8 of Condensed Overview of RWE

45. “Non-Experimental” Designs [NEDs] • NEDs are impact evaluation designs that do not include a matched comparison group • Outcomes and impacts assessed without a conventional statistical counterfactual to address the question • “what would have been the situation of the target population if the project had not taken place?”

46. Situations in which an NED may be the best design option • Not possible to define a comparison group • When the project involves complex processes of behavioral change • Complex, evolving contexts • Outcomes not known in advance • Many outcomes are qualitative • Projects operate in different local settings • When it is important to study implementation • Project evolves slowly over a long period of time

47. Some potentially strong NEDs • Interrupted time series • Single case evaluation designs • Longitudinal designs • Mixed method case study designs • Analysis of causality through program theory models • Concept mapping • Contribution Analysis

48. Any questions? 48

49. TIME FOR SMALL GROUP DISCUSSION 49

50. Self-introductions • What constraints of these types have you faced in your evaluation practice? • How did you cope with them? 50