Welcome!

1. Welcome! • We will begin our webinar at 9:00 am PT • As you log on, do not be surprised if you don’t hear anyone else; participants are placed on mute in order to limit background noise

2. National CASA Performance Measurement Lori Delale-O’Connor Sarah Catherine Williams Child Trends April/May 2014 www.childtrends.org Twitter/childtrends www.facebook.com/childtrends:

3. Presenter Information Lori Delale-O’Connor Scientist Child Trends Having difficulty with the sound? Please send us a message in the chat box.

4. Roadmap Part I: Introductions and Overview of the National CASA project • In-depth interviews • Survey development, testing, and fielding Part 2: Survey Results • Survey completion demographics • Survey data • Potential “priority areas” identified Part 3: Process of Becoming Evidenced-Based • What it means to be evidence-based • “Arrow” and process • National CASA’s work to becoming evidence-based

5. What you will walk away with……. • Knowledge about the current data collection activity of CASA/GAL programs across the country • Understanding of what it means to be “Evidence Based” – and why it matters for your CASA/GAL program • Insight into how National CASA is moving forward with performance measurement and working to build an evidence base

6. Part I: Introductions and Overview

7. Child Trends’ Mission To improve outcomes for children by providing research and analysis to the people and institutions whose decisions and actions affect children. http://www.childtrends.org/

8. EARLY CHILDHOOD YOUTH DEVELOPMENT FERTILITY AND FAMILY STRUCTURE EDUCATION (K-12) Research Areas INDICATORS DATA DEVELOPMENT FATHERHOOD AND PARENTING CHILD WELFARE 8

9. Child Trends’ Work • Develop and track data on children and youth • Help to design, implement, and evaluate child and youth-serving programs • Analyze and synthesize data and evidence • Provide training and technical assistance A key focus of our work is identifying evidence-based programs, practices, and policies

10. National CASA/Child Trends Performance Measurement Goal: Develop key measures to help National CASA better understand performance and move toward developing an evidence base. Phase 1: Scan of current data collection and use (Sept-March) • In-depth interviews • Survey development & testing • Survey fielding Phase 2: Presentation of findings (April-May) Phase 3: Recommendation of performance measures (May-June)

11. Part II: Survey Findings

12. Survey Completion Overview

13. Topics Addressed in the Survey • Data collection responsibility and burden • Data collection systems • Systems used • Meeting needs • Data use • Priorities for future data collection

14. Data collection and level of burden

15. Data collection systems and satisfaction

16. Data use

17. Data support needs

18. Data priorities

19. Participant Feedback

20. Participation What do you think about the priority areas identified in terms of their alignment with CASA? • I think they are an accurate reflection of CASA’s mission and the reach of a volunteer. • I think one or more of them are outside of CASA’s mission and the reach of a volunteer. • I think one or more of them are outside of CASA’s mission, but a volunteer may impact these areas anyway.

21. Participation Do you/local programs you work with already collect data on measures in the priority areas? • Yes, I/they collect data on ALL of these measures. • Yes, I/they collect data on SOME of these measures. • No, I/they do not collect data on ANY of these measures.

22. Participation What potential challenges do you see to getting data on measures in the priority areas? • My program/the programs I work with does not have access to the data needed for these measures. • My program/the programs I work with would need to administer additional assessments (i.e., mental health assessments/screenings) in order to capture data on these measures. • I am not clear on what type of data are needed for these measures. • Programs may interpret the meaning of these measures differently, and therefore data quality would be inconsistent.

23. Participation What potential challenges do you see to reporting on measures in the identified priority areas? • It is a challenge to get the data for these measures, and therefore my program/the programs I work with would most likely be unable to report on these measures. • I/the programs I work with can get the data on these measures, but I am not confident I/they would be able to report on them. • There is a danger that programs could interpret the measures differently, and therefore the data would not be as useful.

24. Part III: Becoming Evidence Based

25. What We Mean by “Evidence-Based”? • Evidence-based programs: • Interventions that have been found to have positive effects or impacts on targeted outcomes • Evidence-based practices: • Intervention strategies or program practices that have been found to improve outcomes for children and youth • Evidence-based policies: • Regulations and laws that have been found to have positive effects on child and youth outcomes

26. High Low Replicated Experimental Levels of Evidence Experimental Quasi-experimental • Evidence-informed, non-experimental

27. Becoming Evidence-Based Collect Data on Performance & Outcome Measures Targeting Implement Program/Approach & Conduct Ongoing Performance Management Conduct Needs Assessment Identify Your Population Select Intervention, Develop Logic Model & Identify Indicators Conduct an Implementation Evaluation Conduct a Quasi-Experimental Outcomes Evaluation Conduct a Randomized-Controlled Impact Evaluation

28. Participant Feedback

29. Participation Where is your CASA/GAL program/the programs that you work with on the arrow? • We/They are not on the arrow at all. • We/They are in the preliminary stages of performing a needs assessment and identifying target population. • We/They are in the more secondary stage of developing a logic model and identifying indicators. • We/They are in the conducting ongoing performance management phase. • We/They are in the more advanced stage of conducting an implementation evaluation, quasi-experimental evaluation, or a randomized control trial (RCT).

30. Food for thought—post-webinar survey question Where do you see your CASA/GAL program/the programs you work with being able to move on the arrow? • Not any further than we/they currently are. • Performing a needs assessment and identifying target population. • Developing a logic model and identifying indicators. • Conducting ongoing performance management. • Conducting an implementation evaluation, quasi-experimental evaluation, or a randomized control trial (RCT).

31. Building the Evidence Base

32. Benefits of Logic Models • Set up a clear path to success or a roadmap for achieving program goals • Provide a visual test of whether inputs, and activities align with outputs and outcomes • Help to get everyone “on the same page” • Facilitate performance management and evaluation

33. LogicModel Example Program Mission Inputs Activities Outputs Outcomes Program Component Program Component Inputs Activities Outputs Outcomes Inputs Activities Outputs Outcomes

34. Outcomes Outcomes Outcomes Outputs Logic Model Example Outputs Outputs Activities Activities Activities Inputs Inputs

35. Outcomes: The observed changes in attitudes, knowledge, or behavior among program participants that result from a program / service / initiative. Output: Measures of evidence of program implementation or quality. Outcomes and Outputs

36. Outputs are sometimes confused with outcomes Outputs Child Outcomes Changes in health Changes in behavior Changes in knowledge Changes in placement Changes in permanency Numbers of volunteers Number of children served

37. Researcher definition of impact: Linking (through study/research) the observed outcomes to a program / service / initiative. Does the program lead to outcomes? Can we control for as many other explanations as possible? Impact could also mean the very long term changes attributable to a program / service / initiative. Child welfare advocates – do children who return home to their parents STAY with their parents? Or do they re-enter foster care at a later date or repeatedly? Impact could also mean the collective outcomes across multiple programs / services / initiatives. Does the same program offered in multiple locations have outcomes that can be aggregated across the locations? What exactly is impact?

38. How does measuring impact differ? Program Outcome(s) Court Appointed Special Advocates Decrease in length of stay in foster care Decrease in trauma symptoms Are other conditions or programs causing these changes? Alternative programming Initiative at local child welfare agency Change in social worker at public child welfare agency

39. National CASA’s next steps for consideration… • Developing a logic model • Identifying a small set of “key measures” for all CASA/GAL programs • Addressing the evaluation challenges of networks structured like National CASA • Conveying potential benefits and collaborating across the network to build an evidence base

40. Contact Information Lori Delale-O’Connor E-mail: ldelale-oconnor@childtrends.org Sarah Catherine Williams E-mail: swilliams@childtrends.org

41. Research in the Service of Children www.childtrends.org 56