Evaluation, assessment & best practices in a family justice center

1. Evaluation, assessment & best practices in a family justice center 12th annual international family justice center conference April 17th-19th, 2012 New orleans, la Carrie petrucci, msw, ph.d., Senior research associate EMT Associates, Inc. - 818.667.9167 cpetrucci@emt.org

2. Presentation overview • Evaluation 101 • Getting the Basics • How a Logic Model Helps You • Thinking about Outcomes and How to Measure them • Evaluating Family Justice Centers • Why is it so Difficult? • Proposed Strategies for FJC evaluations • The CFJI Phase II Evaluation • Why a Collaborative Practice-based Mixed Method Approach is a Good Choice • Determining Best Practices • How to Get There Please ask questions as we go! EMT Associates, Inc.

3. Evaluation 101 A quick review EMT Associates, Inc.

4. Main purpose of evaluation:to determine if a program ‘works’ and for whom it works We need to establish the “counter-factual” EMT Associates, Inc.

5. Tracking the ‘counter-factual’ or what would have happened without the program Two Common Strategies to Capture Counter-factual 2. Historical comparison group 1. Experimental (random assignment) or Quasi-Experimental (not random) prospective comparison group EMT Associates, Inc.

6. Pro’s & con’s of Two types of comparison groups Historical Experimental / Quasi-Experimental Labor intensive to carry out Giving up control of who receives FJC services (in experimental) Lack of cooperation Comparability of groups (in quasi-experimental) Expensive to track data on two groups • Comparability of participants - was same target population served prior to program starting? • Comparability of time periods: How long ago did program start? Were other environmental factors significantly different? • Comparable data: Is same data or any data available? EMT Associates, Inc.

7. Pro’s & con’s of Two types of comparison groups Historical Experimental / Quasi-Experimental If successfully carried out, findings are usually more conclusive than most other designs Can provide the strongest evidence that a program ‘works’ or doesn’t work • Often more feasible approach • Less labor intensive than experimental/ quasi-experimental • Intuitive to broader audience • Findings can be compelling & clear EMT Associates, Inc.

8. What’s the best approach? • Some data is better than no data • Develop a doable strategy, stick to it, & build from there • Consistently collect smaller amount of key data that is reliable rather than trying to collect too much data that is incomplete • Develop/utilize a Client information system /Management Information System • Database such as ETO or off-the-shelf program (Filemaker Pro, Access) • Excel spreadsheet at a minimum • Consider larger scale evaluation on periodic basis (Kettner, Moroney & Martin, 2007) EMT Associates, Inc.

9. General logic model for fjc’s(Townsend, Hunt & Rhodes, 2005) EMT Associates, Inc.

10. Intermediate outcomes (outputs) for fjc’s(Townsend, Hunt & Rhodes, 2005) EMT Associates, Inc.

11. 6 intermediate outcomes(Townsend, Hunt & Rhodes, 2005) • Increased provision of comprehensive services • Increased access to those services • Formal coordination of DV services • Co-location of services • Implementation of a management information system • Increased awareness of both FJC and issues related to DV EMT Associates, Inc.

12. General logic model for fjc’s - OUTCOMES (Townsend, Hunt & Rhodes, 2005) EMT Associates, Inc.

13. General logic model for fjc’s - OUTCOMES (Townsend, Hunt & Rhodes, 2005) EMT Associates, Inc.

14. CFJI Phase I Family Justice CenterLogic Model Source: CFJI Phase I Report, page 5. EMT Associates, Inc.

15. Logic Model EMT Associates, Inc.

16. What is a logic model? A description of your… • Program resources • Service activities and • Immediate, intermediate, and long-term benefits for clients, organizations, or communities It clearly links service activities with more specific client, organization and/or community outcomes EMT Associates, Inc.

17. From the logic model… • You can develop specific ways to document or measure client benefits that you’ve identified • You can then track outcomes to assure clients, organizations, and communities are benefiting in ways you expected or… • You can adjust your services to improve how clients benefit • It’s a “living document” that changes over time as your program changes and grows EMT Associates, Inc.

18. Ways to Measure Outcomes • Counting (how many completed) • Incorporating anchored questions • What is your level of anxiety today on a scale of 1 to 10? • How often did you feel anxious last week – not at all, sometimes, a lot, or all the time? • Customize questions to your setting • Incorporating standardized assessments • See CDC compendium: Measuring Intimate Partner Violence Victimization and Perpetration: A Compendium of Assessment Tools (2006) (can be downloaded) • Incorporating agency-created assessments EMT Associates, Inc.

19. Outcome measurement should be… • Doable by staff • Understandable to clients • Meaningful to staff and to clients • Affordable (in $ and time) • Valuable and meaningful to stakeholders outside the agency • Ideally, measurement also allows you to evaluate your program after the fact using the same data you’ve already collected EMT Associates, Inc.

20. Evaluating family justice centers Key considerations and challenges EMT Associates, Inc.

21. Challenges evaluating family justice centers EMT Associates, Inc.

22. What do we really want to know for evaluation purposes? Family Justice Center cases How do victims / survivors do after contact with FJC? What is the COUNTER – FACTUAL ? What would have happened if there was no contact with FJC among similar participants? Similar cases that do not access Family Justice Centers EMT Associates, Inc.

23. Problem identifying similar cases:multiple points of access Evaluation challenge: Are cases from different access points similar enough to be compared? EMT Associates, Inc.

24. Evaluation challenges in criminal justice process Identifying FJC cases at beginning of criminal justice process Once identified, counting FJC cases at each stage of the process Identifying a similar comparison group Following the trajectory of cases from beginning to end Counting FJC and similar cases from beginning of process to end Multiple data systems across the system Chronological period of time to track cases from beginning to end What about victims/survivors that do not interact with legal system? EMT Associates, Inc.

25. Two evaluation approaches Resource Intensive Rigorous Evaluation Design Sub-Study Approach: Less Resource Intensive / Less Rigorous / Still Informative EMT Associates, Inc.

26. What is best approach to address challenges? • (1) Track all clients that come to FJC as much as possible • (2) Periodically conduct in-depth studies of an unbiased representative sample of cases • All new FJC cases over 1 week, 1 month • (3) Sub-studies at key points in the system • At point of arrest • Among court cases • Choose all new cases in 1 week, 1 month • Non-criminal justice involved • Compare results across sub-study groups EMT Associates, Inc.

27. Addressing real-world challenges with existing resources EMT Associates, Inc.

28. The phase II california family justice initiatIveevaluation Why a Collaborative Practice-based Mixed Method Approach is a Good Choice EMT Associates, Inc.

29. CFJI phase ii evaluation objectives Using multiple data sources to capture multiple perspectives in collaborative, participatory evaluation EMT Associates, Inc.

30. Multiple data sources to reflect multiple perspectives Buy-in from staff essential when they are key to data collection Mixed methods data collection allows evaluation team to build rapport and trust with FJC staff and include perspective of victims / survivors Triangulation of data sources for same indicator enhances validity of the data EMT Associates, Inc.

31. Logic model: objectives and data sources Analyze quantitative & qualitative data together within and across sites to determine key points of variation & similarities EMT Associates, Inc.

32. What is a Collaborative Practice-based Evaluation Research Approach? Collaborative • Project staff AND evaluator contribute their expertise Practice-based • Epstein, Gingerich: putting the needs of practice first “Evaluation research” • Using most rigorous methods possible • Based on existing resources and evaluation purpose • Overlaps with research at methodology level • Purpose more practice-focused than most traditional research

33. Key Elements of Collaborative Practice-based Evaluation Research • First requirement for evaluator is… • “Being there” • Second,… listen, listen, listen to practitioners, clients in the program, other stakeholders • Evaluator needs to… • Be responsive to questions, needs, fears, anxieties • Be open to explain/re-explain especially practice consequences of evaluation strategy • Exercise humility at all times • (Sometimes) wait for initiation of major evaluation strategies to come from practitioners (Petrucci, AEA Expert Panel Presentation, 2007) EMT Associates, Inc.

34. Quick overview of Mixed Methods Mixed Methods vs. Multi-Method Key Decisions Timing, weighting, mixing Timing (concurrent or sequential) Weighting (equal or unequal) Mixing decision (merged, embedded, connected) Use both methods for the same or different research questions? • Where does “mixing” occur? • At data collection phase • At analytical phase • At data interpretation phase • Can be blurry lines between these areas but “mixing” of data must be purposeful and transparent to be of highest quality EMT Associates, Inc.

35. Mixed methods designs 3-Category View (Creswell, 1998) 4-Category View (Clark & Creswell, 2007) Triangulation (quan/ qualgiven equal weight) Embedded (one type given more weight than the other) Explanatory (one type sequentially connected to another for purpose of follow-up or participant selection) Exploratory • Equivalent status designs • Sequential quant/qual • Parallel quant/qual • Dominant-less dominant designs • Multi-level approaches • Individual level data (quant or qual) • Program level data (quant or qual) • Two-Phased Designs EMT Associates, Inc.

36. Enhancing Quality of Data Inferences from Mixed Methods Perspective • “Trustworthiness” of data considered rather than strictly reliability/validity • Credibility: • Prolonged engagement • Triangulation techniques • Sources of data (people with different perspectives) • Methods (quant/qual) • Investigators (different site level visitors) • Peer debriefing • Negative case analysis • Member checks • Transferability of interpretations and conclusions • Thick description • Dependability • Confirmability • Both supported by clear audit trial from original coded data, memo process, and thick descriptions • This increases “internal validity” EMT Associates, Inc.

37. What is gained through collaborative, practice-based, mixed methods approach? • Stronger relationship between evaluation team and project staff • That often leads to higher quality data collection • And in turn, higher quality evaluation product • Flexibility to proactively address unexpected occurrences (that happen in almost every evaluation) • That often leads to stronger implementation of evaluation • A more nuanced picture • Often explains HOW something happens, not just WHAT or IF it happens • Project staff having a higher perceived value of the evaluation process and final report • In part from a sense of ownership EMT Associates, Inc.

38. Why a collaborative practice-based mixed methods approach in family justice center environment? EMT Associates, Inc.

39. Some closingThoughts • A collaborative practice-based mixed methods approach is a useful on-the-ground policy and should be supported in evaluation • Few traditional researchers and even evaluators know about/understand mixed methods • But see NIH document (FINALLY!) on mixed methods • This approach is often the most efficient & effective use of evaluation resources to achieve nuanced findings years faster than traditional research • Preset evaluation policy (e.g., requiring quantitative random designs) can dictate evaluation practice, leaving out mixed methods completely even when there is a clear application for it EMT Associates, Inc.

40. Getting to best practices With concept mapping EMT Associates, Inc.

41. Identifying best practices using concept mapping (if time allows) EMT Associates, Inc.

42. How concept mapping fits with collaborative practice-based mixed methods approach Implementation Advantages Design Advantages Can be done rigorously Incorporates two well established statistical analyses Mixed method approach Flexible but still reliable Concept Systems™ software speeds up analysis process • Collaborative by design • Transparent • Relatively easy to understand • Series of visual maps aid interpretation • Can be done face-to-face or web-based EMT Associates, Inc.

43. Getting to best practices with concept mapping Reasonably low resources & fast turn-around of data Includes diverse voices of staff & victims / survivors Builds on and verifies current knowledge Prioritizes data based on participants’ sorting & rating Incorporates a consensus-building process Approach is flexible yet rigorous EMT Associates, Inc.

44. In Conclusion… • Collaboration means… • Listening & learning from each other • Giving equal weight to all perspectives • In collaborative FJC environment… • Practitioners benefit from understanding basics of evaluation • Evaluators benefit from understanding basics of practice • Evaluating FJC’s will most likely be successful if… • Flexibility, transparency, responsiveness & creativity are incorporated • Voices of experts, staff & victims/survivors are integrated EMT Associates, Inc.

45. Thank you! • Questions / comments • For more information about the CFJI Phase II evaluation, contact: • Carrie Petrucci, MSW, Ph.D.; cpetrucci@emt.org • Wendi Siebold, MA; sps@strategicpreventionsolutions.com EMT Associates, Inc.