Policy Evaluation Primer

1. Policy Evaluation Primer

2. Action Research Cycle

3. Stage One: Problem Identification: Acknowledge the need for change. Can be an existing, even historical, problem, or an emerging issue. Stage Two: Evaluation: Develop and carry out methods for evaluating the breadth and depth of the issue and how it informs practice. Stage Three: Recommendations: Based on the evaluation, provide specific recommendations for change and/or continued evaluation.

4. Stage Four: Application/Practice: Work with the powers that be to take action and institutionalize the recommendations. Stage Five: Reflection: With changes in place, reflect on ways in which new practices affect the work environment. Concurrently, reflect on what you, as an individual researcher, and/or the team learned from the process of the research. Stage Six: Consideration of New Questions: Acknowledge and dialogue about new questions that have emerged from the changes. Have the changes worked? Are there any shortcomings?

5. Problem identification: • What is the problem or issue you wish to examine? • Is the problem an ill-structured or structured one? • Initial situation clearly defined; one desirable goal; clear dominant means to achieve that goal • Many ways to describe problem, complex goal state, values matter, no one clear best goal-state, etc. • Structuring ill-structured problems complicated, but necessary

6. Problem identification: • Who is this a problem or issue for? • Who might be impacted by an eventual resolution of the problem or issue? (e.g. who are the stakeholders?) • What is your unit of analysis?

7. Problem Formulation • “Some ways to formulate complex social problems surely are more useful than others in a given situation, but there is never a single, clearly correct problem definition on which analysis can converge. Nor is there any purely analytic way of specifying which problems, once formulated, ought to be higher than others on the political agenda.” Lindblom and Woodhouse P.22

8. How do we evaluate policies? • What’s the goal? • To develop improved policies • Academic/’wonk’ process • Political process • To critique policies we disagree with • To find support for policies we agree with • To know what position to declare on a specific policy

9. Who’s the audience? • Are you responding to a request from a politician/decision maker? • Are you working to develop and refine general policies (e.g. academics)? • Are you working to promote specific policies?

10. “If professional analysis deserves to be seen as no more than an input to political interaction and judgment, never a substitute for it, then inquiry and judgment by ordinary people remain at the heart of the policy-making process.” Lindblom and Woodhouse P.22

11. Evaluation requires analysis • To increase our understanding • To solve problems

12. Analysis • A search for understanding: why and how? • Not description • We break a problem down into component parts to simplify analysis • How does a car work? • Take it apart and figure out how each part works? • Not enough– need to figure out the interactions between the parts • Same is true for policies

13. Evaluation in Complex vs. Simple Problems

14. Traditional Scientific Method • Objective is to understand • Value-free puzzle solving exercise • Careful observations and measurements • Form falsifiable hypothesis • Repeated experiments to test hypothesis • If hypothesis is falsified, form new hypothesis • When hypothesis can’t be falsified, it becomes a theory • Only expert opinions matter • Replicability, analytical rigor and peer review determine quality

15. Ideology • Continued belief in theory even when contradicted by empirical evidence • Refusal to test theories empirically • “The terms of the debate can be grounded wholly in how the world ought to be, not in how it is.” Charles Murray

16. Public Policy Problems • System is in a constant state of rapid change, and may be beyond our comprehension. • Complexity, non-linearity, chaos, evolution and unexpected surprises • Not value-neutral: Multiple legitimate perspectives • Ethical values, not objective facts • Uncertain facts • Lack of baseline data • Different sets of facts • Different interpretation of facts

17. Public Policy Problems • High stakes • Urgent decisions • Ill-structured problems • Poorly defined problems • Poorly defined solutions: usually many possibilities • Difficulty of experimentation • sample size of one • ethical issues • constant change • scaling

19. Causality and complex systems • Feedback loops • Positive feedback loop • Land prices increase, speculative demand increases, increased demand increase land prices… • Greenspan says housing bubble exists, people stop speculating, prices fall, people sell, prices fall more… • Negative feedback loops • Milk prices increase, farmers expand production, increased supply drives price back to equilibrium.

22. Ways of Stating Casual Relations Religious Attendance and Divorce Rates • Religious attendance ____________ reduced divorce. “causes” “leads to” “is related to” “influences” “is associated with” “produces” “results in” “reduces the likelihood of…”

23. Post Normal Science • Facts are uncertain • stakes are high • decisions are urgent • Values matter

24. Transdisciplinary • In the university there are disciplines, in the real world there are problems • Communication and understanding across disciplines is essential • Communication with sectors outside of academia is essential

25. Identifying and Gathering Data • Define and structure your problem. Put together outline of what you need to know. Gaps are often first information you need to collect. • Make list of data sources • Best to rely on existing data • How much information do you need?

27. How much information? • Too much or too little information • Time and Cost • “Most policy decisions are made using no more than perfunctory analysis, because decisions cannot wait until ‘all the facts are in.’” Lindblom and Woodhouse p.20 • Optimal ignorance

28. Distinguishing “good” information from “bad” • Source: peer reviewed, grey literature, Internet, government statistics, etc., wikipedia, etc. • Triangulation

30. Questions to ask: • Data collected independently? • Examples • How were the data collected? • Why were the data collected? • When were the data collected? • Who collected the data? • Are only selected data being reported?

31. Quantitative vs. Qualitative • Obsession with quantification and ‘academic purity’ • Numbers and symbols • Interpretation of the facts

32. Literature review: Goals • To demonstrate a familiarity with a body of knowledge and establish credibility. • To show a path of prior research and how current analysis is linked to it. • To integrate and summarize what is known in an area. • To learn from others and stimulate new ideas. Neuman W.L. 2000, p446

33. Interviews • Standardized questionnaires • ‘Elite’ or specialized interviewing • Stakeholder interviews

34. Communication • Your communication must have a specific purpose. You must fully understand this purpose, and know how to articulate it to others; • You must communicate to the appropriate audience, in a format suitable to that audience and to the situation; • Your content must be well organized, concise and to the point; • You must capture and hold the attention of your audience; • Your graphic material must be suitable for the audience, clear and easy to understand, and help to convey your message; • You must be credible, and your information must be credible; and • You must get critical evaluation of polished draft documents or practice presentations