Policy and Evidence: An Uneasy but Essential Partnership

1. Policy and Evidence: An Uneasy but Essential Partnership Mark E. Courtney Fred H. Wulczyn Chapin Hall at the University of Chicago

2. The policy realm is where conflicting values play out most directly • Practice concerns individual-level effects on outcomes that are generally agreed upon • Administration is organized around achieving the goals of policy • Social welfare policy is often the result of highly contested and shifting values • Examples: Willingness to pay • Child protection versus family preservation • Benefit-cost analysis of extending foster care past 18

3. Types of Evidence Needed for Policy Making Characteristics/needs of the population Characteristics and behaviors of systems Capacities Political environment Organizational cultures System dynamics

4. System Dynamics: Patterns Identify Targets of Policy

5. System Dynamics: Patterns Identify Targets of Policy

6. Limits of randomized experiments for informing policy Some policy changes cannot be assessed at all using experiments Mandatory child maltreatment reporting Extending foster care past 18 Constellations of policies are difficult to assess due to design complexities Extending care, health insurance extension, and education vouchers Time required leads to changes in context that can compromise experiments, but actually benefit non-experimental methods …but, experiments are VERY valuable!

7. Principles and Lessons Sound information to guide policy will come as much or more from sound data on populations and services as from evaluation research per se Experiments are best implemented in the context of sophisticated knowledge of populations, service contexts, and system dynamics At best, research can often only help illustrate the tradeoffs of policy choices Do not oversell the implications of findings for policy, lest you get what you asked for Illustrating tradeoffs helps policymakers understand when and how ideology guides their decision making