Tracking, Assessment and Evaluation Program

1. Tracking, Assessment and Evaluation Program Kathryn Nearing, PhD Jeffrey Proctor, MBA Lead External Program Evaluators The Evaluation Center, School of Education and Human Development

2. Governance Tracking, Assessment & Evaluation Nearing and Proctor

3. Specific Aims relevant to TEC Research Environment: • Catalyze quality and process improvement and ascertain key program impacts through responsive tracking, assessment and evaluation. (SA4) Resources and Services: • Develop and implement a Program Income System and an evaluative process for cost-effective allocation of resources that supports the planning, conduct, analysis, and dissemination of research results. (SA1) • Optimize the infrastructure for implementing and tracking research and support services. (SA2)

4. ETCD: Four over-arching questions aligned with component themes • What is the evidence that the ETCD pillar effectively recruits and supports the retention of a diverse workforce? (recruitment and retention, inspire academic persistence) • How effectively does the ETCD pillar address the workforce development needs of diverse investigative communities, locally and nationally? (build future workforce, forge collaborations) • Are trainees, scholars adequately prepared for productive careers in clinical translational research? (expand the ETCD portfolio, enhance efficiency) • What is the evidence that we are promoting a culture of effective mentorship? (mentorship)

5. Leveraging Investment of First 5 Years (a few examples) • Needs assessment, workflow analyses informed priorities that will serve as the focus of QPIP initiatives • Existing processes, instruments (e.g., pilot grant tracking expanded to include CSU pilot projects) • Defined indicators, metrics, data points, and feasibility studies regarding their collection, will inform utilization of robust, centralized data systems/tools • Refinement of existing, and development of new, ETCD programs based on evaluation

6. EAC Critiques • Avoid duplication of effort with QPIP • Enhance evaluation of ETCD programs (invest more resources in this area) • Actively contribute to the establishment of national metrics • Move toward a more outcome/impact-oriented evaluation; include an assessment of CCTSI’s impact on health status indicators (utilizing “big data”)

7. Approach: Research Environment • Leverage evaluation findings from first grant cycle to identify priorities • Key Priority: Increase efficiency of study start-up (scientific review, regulatory approval, and enrollment of study subjects) • Leverage the enhanced data infrastructure to assess impact of QPIP initiatives

8. Approach: Research Environment Figure: TEC role in catalyzing quality and process improvement, and assessing the results, to inform data-driven decision making Report Findings to EC & QPIP EC & QPIP Discuss Implications Implement QPIP Initiatives Collect & Analyze Data Reevaluate to Determine Impact of Changes Improved Quality, Cost Effectiveness, Efficiency, Innovation and Safety

9. Approach: Resources and Services • Leverage enhanced data infrastructure, as well as formal needs assessment process, to collect relevant metric-level data systematically • Utilize a balanced scorecard approach to assess value and inform strategic allocation of limited resources

10. Approach: Resources and Services Figure: Balanced Scorecard Approach

11. Approach: ETCD • Strategic planning • Comprehensive examination of data, instruments, key findings • SOW developed collaboratively with steering committee based on established priorities • Implementation of Scope of Work • Revised all instrumentation to support implementation • Progress is reviewed monthly at steering committee meetings

12. ETCD: Four over-arching questions aligned with component themes • What is the evidence that the ETCD pillar effectively recruits and supports the retention of a diverse workforce? (recruitment and retention, inspire academic persistence) • How effectively does the ETCD pillar address the workforce development needs of diverse investigative communities, locally and nationally? (build future workforce, forge collaborations) • Are trainees, scholars adequately prepared for productive careers in clinical translational research? (expand the ETCD portfolio, enhance efficiency) • What is the evidence that we are promoting a culture of effective mentorship? (mentorship)

13. National CTSA Consortium Involvement • Evaluation Key Function Committee • Mentored to Independent Investigator Workgroup • Provided data related to national survey effort • Are collaborating to interview former KL2 scholars to address key evaluation questions • Team Science Affinity Workgroup • Presented two paper presentations as part of CTSA panels at the AEA (workflow analysis, team science) • Pipeline Framework manuscript in development

14. Assessing CCTSI Impact (One Strategy) • Leverage longitudinal pilot grant tracking (e.g., with community engagement awardees) • Identify projects that meet specific criteria: • Adopted/adapted an evidence-informed strategy • Implementation and dissemination occurred in a discrete (well-defined) geographic area/population • Targeting very specific (granular) health indicators • Examine trends in relation to implementation (compare historically and with state)

15. Questions for EAC • Are responses to critiques deemed adequate? • Do you have additional thoughts about how to assess the impact of the CCTSI on community-level health status indicators? • Would you be willing to stay connected as we move such efforts forward (in between EAC meetings)?