Examining Evidence Identifying elements of evidence. How can we do better?

1. Examining EvidenceIdentifying elements of evidence.How can we do better? NASCSP March 1, 2012

2. Guidance About Evidence • “Success should be judged by results, and datais a powerful tool to determine results. We can’t ignore facts. We can’t ignore data.” • President Barack Obama • July 24, 2009 • “The test of a performance management system is whether it’s actually used….Federal managers and employees at all levels must use performance goals and measures to set priorities, monitor progress, and diagnose problems.” • Chief Performance Officer Jeffrey Zients • October 29, 2009 2

3. Use of “evidence” • Two key points: • Results are supported by data • Data from performance measurement systems must be put to use. • In recent OCS guidance, several “models” and references to “evidence” have been identified. • Here is a quick review of these terms.

4. Evidence-Based Practice • Approaches to prevention or treatment that are validated by some form of documented scientific evidence. • This could be findings established through scientific research, such as controlled clinical studies, or other comparable and rigorous methods. • Identifies proven effectiveness, supported by objective and comprehensive research and evaluation.

5. Evidence-Informed Practice • Approaches that use the best available research and practice knowledge to guide program design and implementation within context. • This informed practice allows for innovation and incorporates the lessons learned from the existing research literature.

6. Exemplary Practice • An innovative and consistently applied policy, process, practice, or procedure that takes a comprehensive approach to developing and implementing activities using strategies that are related to the intended service recipients and community. • This practice model is culturally competent, data-driven, measurable, and replicable and incorporates a method for documenting programmatic results.

7. Promising Practice • A practice with at least preliminary evidence of effectiveness in small-scale interventions, or for which there is potential for generating data that will be useful for making decisions about taking the intervention to scale the results diverse populations and settings.

8. Field Tested Practice • A program, activity or strategy that has been shown to work effectively and produce successful outcomes • This is supported to some degree by subjective and objective data sources

9. Where’s the beef (aka data)? What do we know about our measurement tools? Are they • Accessible • Reliable • Valid (appropriate) • Affordable • Scalable (able to measure change)

10. Tools For Measurement • Loan monitoring reports • Mortgage documents Observation log • Pre-post tests • Progress reports • Questionnaire • Rent receipts • Scales and Matrices • Survey • Testing results • Bank accounts • Copy of Diploma or Certificate • Employment records • Escrow accounts • Financial reports • Health or nutrition records • Inspection results • Lease agreements • Legal documents

11. Examples of data sources Identify pros and cons of each source • Agency database • Case notes • Centralized database • Computer spreadsheets • File cabinets • Individual case records • Manual tallies • Public database • School records • Specialized database • Tax Assessor database • Training center • Work plan reports

12. Personnel • If no one is assigned the task, no one will do it. • Consider various agency staff responsibilities • Include partner reports

13. Procedures • Home visits • Office appointments • Phone follow up • Review of case notes/progress reports (scales?)

14. Frequency • Daily • Weekly • Monthly • Quarterly • Biannually • Annually • Upon incident

16. What ROMA Data IS • Data about community needs and resources • This can identify the scope of community issues • Used to prioritize community needs for agency intervention • Data used during strategic planning • Identify resources ($, facilities, staff, etc) the agency has/needs • What results does the agency expect to achieve? • Data collected during implementation of services • How many individuals, families, communities are projected to be served? How many are projected to achieve results? • How many individuals, families, communities were actually served? • Who were they? • Data collected to identify achievement of results • How many made movement toward their goals? • How many achieved the end results previously identified?

17. What ROMA Data IS NOT (yet….) • Evaluation Data • Used in comparing CAA performance with a “control” group to determine the quality of the results • Done by an independent reviewer • Used to validate that results actually impact on identified community needs

18. Moving towards Evidence Source: Child Trends

19. Who’s using ROMA data? • The collection of NPI data for a report to OCS is a comprehensive effort by the CSBG Network to identify the results of the efforts of local CAAs. • This is sometimes called the ROMA report, but we have just seen that ROMA data includes more than these indicators.

20. We use data to: • Measure and report performance • Identify patterns and relationships • Consider new actions based on analysis • Create new outcome-focused goals • Develop new resources

21. So now what? • When you have your data, remember that you have to turn them into information. • They must be interpreted to become evidence • Consider this piece of data and the questions that arise: 35 individuals got a job. • Is that good?   What is the unemployment rate? Opportunity for employment? Characteristics of those who got jobs. Etc.

22. The top 10 • What are the top 10 services provided by your agency? • What are the top 10 outcomes achieved by your customers? • What is the connection between these two lists?

23. What’s next? • Build on ROMA principles to identify ways to collect credible evidence in systematic ways (making the connections among mission, community need, agency strategies and well documented results) • Find ways to compare CAA performance with established standards and performance of other similar programs.

24. Redefining Evidence • We must recognize the value of evidence-based or evidence-informed interventions…. but our definition of what counts as credible evidence should be expanded to allow for continuing improvement and innovation. • ‘‘Evidence-based’’ does not have to mean experimental-based. • We draw on evidence from many kinds of research, including program evaluations, and practice. From Lisbeth Schorr

25. Begin with a results framework. • LisbethSchorr tells us to identify the clear, measurable results sought by a complex intervention as the essential first step toward both successful implementation and to a successful evaluation. • We have a start on this with the NPIs.

26. Expect to compare results • It is essential to have some way of comparing results to establish that the observed change has a high probability of resulting from the practices. • How you will compare results from your programs to those who have not been involved in your programs. The community-specific nature of place-based interventions makes it very hard to find a comparison group that would allow for a clinical “control group.”

27. Other kinds of comparisons It is possible to compare outcomes among the populations those served by a specific initiative to • similar populations in the geographic area before the intervention began for whom baseline data are available, • current populations that did not receive similar services and supports, but for whom data are already available, i.e. does not have to be collected as part of the evaluation, and • national, state or local norms.

28. ROMA to the rescue • The elements of ROMA principles and practices can help agencies establish evaluation frameworks that include the evidence that will be used to support their results • Establish CSBG industry standards • Remember what baseball did that we haven’t yet done:

29. Baseball’s example • Clearly identified the indicator to be measured • Collect data consistently • Analyze the data against records of “wins” • Publish the data and the analysis, so the public recognizes success

30. Suggested Actions • Support knowledge collection, analyses, and evidence syntheses that yield a more complete body of evidence. • Develop network tools and capacities to gather knowledge at greater scale. • Expand the menu of available evaluative techniques that can be matched to different types of interventions. • Combine findings from research, theory, and practice, for informed decision-making • Promote use of a results framework to strengthen measurement for accountability and learning

31. Use day-to-day management of work • Real-time learning can be achieved by using a well developed results framework to track progress toward those results, and using the data for real-time learning.

32. Today’s Logic • As Peter Drucker has pointed out, “The greatest danger in times of turbulence is not the turbulence; it is to act with yesterday’s logic.” • We do not want to rely on fads, hunches, anecdotes, or good intentions. • Nor are we reluctant to identify and end support for the efforts that are ineffective.

33. For more discussion: Barbara Mooney National ROMA Training Project 243 E. High St. Waynesburg, PA 724-852-2272 barbaramooney@windstream.net