Evaluation Update: What Have We Learned So Far? Findings from the Case Studies

1. Evaluation Update: What Have We Learned So Far?Findings from the Case Studies February 4, 2010 Contributors: Brenda Joly Maureen Booth George Shaler Ann Conway

2. Overview • A Review of the Evaluation • What do we hope to learn? • Case Study Findings • Why are case studies useful? • What did we learn? • Other Evaluation Activities • Annual survey • Four additional case studies • Case study follow-up interviews

3. Goals of MLC • To promote the application of QI methods • Prepare for & contribute to national accreditation

4. Evaluation Questions • Are learning collaboratives an effective vehicle for: • Enhancing QI capacity and application? • Achieving targets? • What factors are associated with successful QI initiatives and spread? • How have states positioned themselves for national voluntary accreditation?

6. Why Case Studies? • On the ground” understanding • Opportunity for multiple perspectives • Better appreciation of gains and constraints

7. Case Study Theory

8. Case Study Components • Site visit • Interviews with state and local participants • Document reviews • Observation of Mini-Collaborative meeting where possible

9. Selection of States • Target area • Timing of the Mini-Collaborative • Broadly representative of geography, size and structure • Twelve case studies conducted first round; four remaining case studies in 2010

10. Case Study Design

11. Limitations • Snapshot in time • Mini- Collaboratives at different stages of implementation • MC design and implementation differed among states; sometimes within a state • Constrained on number of LHDs to interview • Potential bias in selection of LHDs

12. Findings • Planning a Mini-Collaborative • Managing a Mini-Collaborative • Implications going forward for NNPHI, states and LHDs

13. Planning a MC • Who selected target? • State sponsors • Statewide body • Local input • Criteria used in selection? • Relevance • Alignment with other state priorities • Need demonstrated by evidence/data • LHDs more engaged when target was relevant to daily practice Target Selection

14. Planning a MC Who planned the MC? State sponsors Broader group of state, faculty and others What was planned in advance? MC model to be used Evidence to support QI intervention Recruitment of credible and skilled faculty Learning curriculum and work plan Tools to assess participant QI knowledge/skills Measures for monitoring progress Extent of planning influences MC effectiveness Advance Planning

15. Planning a MC Goals and Expectations • Goals of MCs varied and evolved • Build skills in QI tools and techniques • Improve quality within target area • Advance accreditation • Combination • Important to align goals with MC timeframe and participant readiness • Expectations for LHD QI Project should be focused, realistic and communicated

16. Planning a MC LHD recruitment strategies Open solicitation to all LHDs Selective recruitment based on prior partnerships, geography, size, and/or need Faculty recruitment Limited to state sponsors Other state agencies with relevant expertise Universities and community resources Prior familiarity jump-starts the MC process Diversity of faculty adds value Composition and Membership

17. Managing a MC Intensity of effort by state sponsors greater than anticipated Especially time consuming in early stage of MC Time commitment reduced when outside faculty involved LHDs under-estimated commitment that was required Lack of clear expectations upfront Unrealistic scope for QI projects Pulled by other priorities Level of Effort

18. Managing a MC Most common QI Tools Plan-Do-Study-Act Root cause analysis Fishbone diagrams Most effective approaches Fewer tools Train on when and how to use Provide immediate application Give a chance to practice, practice, practice Feedback on use QI Tools and Methods

19. Managing a MC Evidence-Base for MC More likely used for outcome targets Limited application in states with process targets Use of National Resources Resources available through NNPHI not fully understood or accessed by states LHDs had limited understanding of MLC or NNPHI LHDs turned to resources in their states, NACCHO or CDC Use of Evidence and Resources

20. Managing a MC Major methods Structured learning sessions Group conference calls or webinars One-on-one conference calls Site visits to LHD participants Effective approaches Clear, concise expectations and curricula Public health and quality improvement expertise Explicit links between theory and practice In person meetings highly valued Site visits from state partners very helpful Not all states actively involved in training or TA; others saw both as essential to learning QI Training and Technical Assistance

21. Managing a MC Prior efforts that strengthened public health infrastructure Commitment to data-driven QI and accreditation Development of partnerships, especially with universities Prior knowledge of each other and/or QI boasted confidence Factors Promoting Effective MC

22. Managing a MC Concurrent QI skill development at state and local levels Focus on QI projects sometimes eclipsed QI skill development Overly ambitious goals and expectations Lack of clear work plans and timelines to monitor progress Competing work priorities Staff turnover/layoffs Limited technical assistance Geographic distances reducing in-person mtgs Factors Impeding Effectiveness

23. Implications National • Flexibility in implementing state grants should be balanced against benefits of more structure and guidance, especially related to • Selection of target area • Use of collaborative approach • Focus on QI • NNPHI’s role and resources in assisting states and LHDs should be more fully communicated • Access to evidence based practices and subject matter experts • Fostering communication among affinity groups • Dissemination of lessons

24. Implications Setting aside time for advance planning is critical Recruitment of credible and skilled faculty raises confidence in the process Assessing the level of QI understanding among participants informs the MC process Having the right number and mix of participants impacts engagement Timely and frequent application of QI tools is essential to learning Communication with and among participants enhances the learning experience A structured approach seems to help, not hinder, effectiveness State

25. Implications The target area and QI project should be relevant to job responsibilities. The perceived value of QI impacts LHD participation and engagement. Local level leadership is essential. Local Level

26. Next Steps Who got the survey? All LHDs within MLC states Mailing lists sent to states for verification How can you help? Publicize importance of survey (promotional blurb available from Muskie) Help with follow up to non-respondents 2010 Annual Survey

27. Next Steps Case studies in remaining four states to be conducted Spring 2010 Illinois Kansas New Hampshire North Carolina Follow-up Interviews with twelve prior case study states in March-May Case Studies

28. Questions??