Children’s Bureau Annual Combined Discretionary Grantees Meeting June 11-13, 2012

1. Implementation Science: Research on and Recommendations for Promoting the Successful Implementation of your Discretionary Grant ProgramElliott Graham, Ph.D.James Bell Associates Children’s Bureau Annual Combined Discretionary Grantees Meeting June 11-13, 2012 Mayflower Renaissance Hotel Washington, DC

2. Implementation Science: What Is It? • Good outcomes influenced as much by implementation process as by specific services/practices (Aarons & Palinkas, 2007). • Research to understand effective implementation in real practice settings collectively referred to as “implementation science” (IS). • IS conceptual models: • Aarons, Hurlburt, & Horwitz (2011) • Bumbarger, Perkins, & Greenberg (2009) • National Implementation Research Network (NIRN) (2005) • Getting to Outcomes (Wandersman et al., 2000)

3. Implementation Science: What Is It? • Implementation science is not rocket science! • It’s really just a fancy term for “good implementation.” • In other words, what does it take to plan, operate, and sustain your program successfully?

4. Overview of NIRN Stages of Implementation • Exploration and Adoption: Assessing match between EBPs and org./community needs and resources. • Program Installation: Tasks that need to be accomplished before the first participant is enrolled (e.g., hiring staff, securing space). • Initial Implementation: Major activities/services begin and confidence in the program is tested. • Full Operation: New services and activities become integrated into the practices, policies, and procedures of staff, the organization, and broader community. • Innovation: Refinement and expansion of new practices and programs. • Sustainability:Long-term maintenance of program in the context of changing organizational, political, economic, and community circumstances.

5. Background and Purpose of JBA’s IS Study • Based on JBA’s current and past work with CB and OFA, and existing IS research. • Key questions: • How useful are existing IS models for supporting the implementation efforts of CB discretionary grantees? • What organizational characteristics, activities, and processes are key to the successful implementation of grant projects? • To what extent are identified implementation factors aligned with existing IS models? Are there other factors not accounted for by the models that contribute to effective implementation?

6. Overview of IS Study • Screening calls conducted in early 2011 with 54 CB discretionary grantees served through JBA’s evaluation TA contract. Participants included project director, evaluator, key front-line staff. • Topics: • Organizational partners • Project descriptors • Major project activities • Implementation challenges and facilitators • Evaluation activities • Dissemination • Project sustainability

7. Overview of IS Study • 17 grantees from several grant clusters selected for further study through site visits/conference calls in May-July 2011: • Diligent Recruitment of Families for Children in the Foster Care System (2010 and 2008 clusters) • Abandoned Infants Assistance: Comprehensive Support Services for Families Affected by SA and/or HIV/AIDS (2009 and 2008) • Nurse Home Visitation (2007) • Comprehensive Family Assessment (2007) • Collaboration between TANF and Child Welfare Services to Improve Child Welfare Program Outcomes (2006), including Cook Inlet Tribal Council) • Post-Adoption Services and Marriage Education (2006) • Model Development or Replication to Identify and Serve Substance Exposed Newborns (2005)

8. Defining Effective Implementation: What Does it Look Like? • We defined implementation effectiveness using criteria derived from IS literature and JBA’s prior work: • Presence of strong program champion(s) or “purveyors” • Active collaboration with project partners • Evidence of meaningful organizational/systemic change • Successful participant recruitment & retention • Full implementation of core project activities (fidelity) • Adaptability to organizational, contextual challenges • Strong evaluation • Effective dissemination strategies • Sustainability plan in place, or meaningful action toward sustainability

9. Discretionary Grants: A Unique Implementation Environment

10. Key Implementation Elements for CB Discretionary Grantees • Phase 2: Project Implementation • Empower Project Champion(s) • Hire/Assign Project Staff • Provide Initial Staff Training • Ongoing Training, Supervision, and Staff Evaluation • Recruit Participants • Retain Participants • Make Necessary Program Mods • Disseminate Project Information • Plan for Project Sustainability • Utilize TA Resources • Implement Program Evaluation • Phase 1: Conceptualization and Planning • Intervention Development: • Intervention Workgroup/Team • Use Data to ID Needs of Target Population • Design, Identify, or Adapt Program Model • Plan Program Evaluation • Infrastructure Development: • Identify Project Champion(s) • Gain/Maintain Internal Org. Support • Build External Org. Partnerships and Commitment Improved Project Plan Improved Grant Proposal Increased Implementation Readiness Improved Project Implementation Improved Participant Outcomes Improved Systems of Care

11. Characteristics of Successful Program Implementers Phase One: Conceptualization and Planning Leadership • Project champion(s) with organizational clout • Supportive and engaged senior management • Formal, permanent, dedicated leadership team

12. Characteristics of Successful Program Implementers Phase One: Conceptualization and Planning Relationships • Committed and engaged external partners • Wide use of informal contacts and relationships

13. Characteristics of Successful Program Implementers Phase One: Conceptualization and Planning Assessment and Evaluation • Use of informal assessment processes (although formal assessments may result in better information) • Careful assessment of match between program model/design and needs of target pop • Formal evaluation plan and logic model prior to start up

14. Characteristics of Successful Program Implementers Phase Two: Project Implementation Leadership and Organizational Environment • Stable project champion(s) • Clear implementation policies/procedures • Adequate resources • Reasonable staff workloads

15. Characteristics of Successful Program Implementers Phase Two: Project Implementation Staff Selection, Training, Supervision • Front-line staff with relevant skills/experience • Front-line staff perceived as caring, authentic, respectful, passionate • Intensive and comprehensive training for new staff • On-the-job coaching and emotional support

16. Characteristics of Successful Program Implementers Phase Two: Project Implementation Program Changes/Adaptations • Careful, deliberative, purposeful approach to change • Backed up by hard data, not just anecdotes • Guided by program experts • Wide use of Federal and other TA resources

17. Characteristics of Successful Implementers: Phase Two: Project Implementation Recruitment and Retention • Community leaders and org. partners involved in recruitment • Recruiters perceived as authentic and credible • Participant retention strategies: • Validate usefulness/value of program • Engage participants actively • Encourage boundary-appropriate bonding/relationships • Celebrate participant successes • Judicious use of incentives – intrinsic incentives more important than extrinsic ones

18. Characteristics of Successful Implementers: Phase Two: Project Implementation Dissemination and Sustainability • Diverse dissemination strategies for different audiences • Early and proactive sustainability efforts • Alternative long-term funding sources • Non-financial resources • Making certain project elements “business as usual”

19. Characteristics of Successful Implementers: Phase Two: Project Implementation Evaluation • Strong implementers tend to have strong evaluations • Evaluation data used consistently to inform program development, improvement, and decision-making • Characteristics of strong evaluations: • Clear process measures/outputs • Use of both quantitative and qualitative methods • Use of standardized assessment instruments when appropriate

20. Recommendations for Federal Grantees Program Conceptualization and Planning • Create a formal leadership team that leads/oversees planning and implementation. • Use formal and informal data sources to inform program selection or design. • Identify or design programs that are responsive to demonstrated needs of target population and based on best available evidence. • Build new or enhance existing org. partnerships and commitments. • Integrate program evaluation into project design and planning. • Identify and empower a Project Champion. • Modify agency policies, procedures, and practices to accommodate implementation.

21. Recommendations for Grantees Project Implementation • Hire or assign project staff with appropriate skills, education, personal qualities. • Conduct intensive, comprehensive initial & ongoing training. • Provide engaged supervision that emphasizes on-the-job coaching, immediate feedback, emotional support. • Use proactive, collaborative participant recruitment/retention strategies. • Make program changes only when necessary and when supported by hard data (not just anecdotes). • Engage in strategic information dissemination with an eye toward sustainability.

22. Recommendations for Grantees Project Implementation (cont.) • Make full use of TA and expert resources. • Implement highest-quality evaluation possible with focus on program development and improvement. • Build sustainability efforts into the program design and implementation process.

23. Next Steps • Which of these recommendations are you already following? To what extent? • Which recommendations are you not yet following? What would you need to do to start following them? • Which recommendations are not relevant to your project, or are beyond your capacity or control to implement?

24. Questions or Comments? Elliott Graham, Ph.D. James Bell Associates 703-528-3230 graham@jbassoc.com