Presentation to National Center of Dissemination of Disability Research (NCDDR) Workshop:

1. Towards KT Measurement:What Constitutes KT Outcomes and Outcomes Monitoring in the NIDRR Context(An Evaluation Perspective) Presentation to National Center of Dissemination of Disability Research (NCDDR) Workshop: December 3, 2007 Margaret L. Campbell, Ph.D Senior Scientist for Planning and Evaluation National Institute on Disability and Rehabilitation Research

2. Working Assumptions and Perspective of Presentation • Focus is on KT as a set of strategies for accelerating the uptake of R&D findings to improve systems and benefit end users – i.e., the desired result of KT is “getting to outcomes.” • R&D outcomes do not occur without some type of KT, including publication. • Initial questions posed by NCDDR for this segment of the workshop are in many ways ahead of NIDRR’s development of KT and integration into planning and evaluation. Because of this, it is necessary to reformulate the objectives of this presentation.

3. Presentation Objectives • Illustrate how NIDRR’s existingstrategic framework can be used to clarify what constitutes an outcome and to suggest possible theoretical KT outcomes. • Describe components of NIDRR’s existingevaluation framework, explain current strategies used to monitor and assess the quality of program outcomes, and present quantitative results from recent Annual Performance Assessment Expert Review (APAER) process. • Use examples of “noteworthy” accomplishments from 2007 APAER for Participation and Community Living to illustrate the kinds of outputs and outcomes grantees are reporting and whether they correspond to suggested theoretical KT outcomes.

4. Part 1: Illustrate Utility of NIDRR’s Strategic Framework • NIDRR’s Long-Range Plan and Logic Model help clarify and identify possible theoretically KT outcomes to be tracked and evaluated for quality? • The Long-Range Plan identifies strategic goals and objectives and provides a roadmap to the kinds of program outcomes NIDRR is seeking; • The Logic Model identifies outcome arenas that correspond to strategic goals, types of R&D outcomes, and intermediary and intended beneficiaries.

5. NIDRR’s Strategic Goals, Long-Range Plan – 2005-2009 • To advance knowledge through capacity-building: • By increasing capacity to conduct and use high quality and relevant disability and rehabilitation research and related activities designed to guide decision making, change practice and enhance the lives of individuals with disabilities. • To advance knowledge through research and related activities: • By generating science-based knowledge, technologies and applications in order to contribute to policy; change practice; and enhance the lives of individuals with disabilities. • To advance knowledge through translation by utilizing science-based knowledge, technologies and applications: • By promoting the effective use of science-based knowledge, technologies and applications to contribute to disability and rehabilitation policy; improve practice; and enhance the lives of individuals with disabilities.

6. Brief Background on Logic Models (Making sure we’re on the same page) • What is a logic Model? Graphic blueprint of how a program is supposed to work to achieve benefits for participants. Ideally, a logic model depicts both the “theory of the problem” and the “theory of program.” • Utility of logic modeling? In additional to serving as a valuable communication tool, logic modeling ideallyhelps researchers, program managers and evaluators: • Chart the pathways by which program activities and outputs will have their desired effects or outcomes; • Identify “indicators of success” or milestones along the outcomes pathway that occur with the “span of accountability” of the program.

7. Components of NIDRR’s Logic Model for Research Outcomes (See Handout) • Situation or “theory of the problem” to be addressed • Outcome Arenas -- correspond to strategic goals of the LRP for CB, R&D and KT and point to where program results occur. • Types of Outcomes: • Short-term – advances in understanding, knowledge, skills and learning knowledge that occur through discoveries, theories measures and methods, and interventions, products, devices, etc. • Intermediate – adoption and use of short-term outcomes to change/improve policy, practice & system capacity • Long-term – changes/improvements in overall conditions of the system or population (e.g., increased access and improved rates of employment)

8. Components of NIDRR’s Logic Model for Research Outcomes (Continued) • Together, these outcomes: • Reflect a hierarchy or sequence of anticipated results in order of program implementation; • Depict the program’s change strategy; and • Help to identify areas of possible implementation failure. • Beneficiaries • Intermediate Beneficiaries or Initial Users • Intended Beneficiaries or End-Users

9. Lessons Learned in Implementing NIDRR’s Logic Model • Still a work in Progress. Some of the gaps include: • No specific KT & CB outcomes are identified; • No depiction of the pathways by which outputs of NIDRR’s investments in R&D become outcomes and what is the role of KT is this process; • Similarly, no depiction of the technical and infrastructural developments or “milestones” necessary to support advances in knowledge and changes in policy, practice and systems capacity. CHIR is addressing this challenge by identifying “indicators of success” that approximate and anticipate KT outcomes; • Stated another way, no acknowledgement that outcomes evolve over time and do not emerge fully developed and to scale across a target population of beneficiaries or system.

10. Possible Future Development of NIDRR Logic Model • Ideas for Possible Future Development: • Elaborate CB and KT outcome arenas; • Chart pathways by which KT activities and outputs lead to KT outcomes and facilitate R&D outcomes; and • Identify “indicators of success” that can be used to mark progress towards KT outcomes. • Caveat: • For the above developments to occur, we need more input from stakeholders and more data on the performance of the NIDRR’s new KT centers. • So what can we do in the meantime?

11. Proposed Interim Strategy • We can assume that: • KT outcomes do not differ theoretically from R&D outcomes in their type or sequence or in the intended target populations of users and beneficiaries. • The only differences are in the specific knowledge areas being advanced and to some degree in the methods used to achieve outcomes. • We can use the tools of logic modeling, along with the ongoing refinements in KT NIDRR is making under Pimjai Sudsawad’s leadership, and the experience of other federal and non-federal agencies to suggest possible theoretical KT outcomes that could be tracked through NIDRR’s existing evaluation system to determine their frequency and appropriateness.

12. Building Blocks for Identifying Possible KT Outcomes • CIHR’s initial definition and operationalization of KT and results of recent initiatives described early in this workshop. • On the U.S. side, results of the T.R.I.P. initiative (Translating Research into Practice) sponsored by the Agency for Healthcare Research and Quality (AHRQ) that suggest to accelerate research into policy and practice research users need three types of KT outcomes: • Research results about what works and what doesn’t, • Knowledge about how others have implemented findings within their organizations, and • Knowledge about how the culture and values of an organization affect the uptake and continues use of the findings.

13. Possible KT Outcomes within the NIDRR Context – For Discussion Only • Possible Short-Term Outcomes: • Increased understanding and knowledge of the KT concept and methods, as reflected in results of KT training and use and/or adoption of KT outputs; • Rigorous research results about which KT strategies and methods work and which do not and in which settings and for which target populations. This would be reflected in the increased availability of “best practices.” • Increased knowledge about how the culture and values of research users, including consumers, and organizations affect the uptake of R&D findings.

14. Possible KT Outcomes within the NIDRR Context – For Discussion Only (Continued) • Possible Intermediate Outcomes: • Change in R&D policy as reflected in increased funding of KT initiatives by federal and non-federal organizations; • Change in R&D practice as reflected in increased utilization of KT strategies and methods by practitioners; • Increased capacity to support the exchange, synthesis and ethically sound application of R&D findings to change policy, practice, as reflected in infrastructural developments at a systems level.

15. Possible KT Outcomes within the NIDRR Context – For Discussion Only (Continued) • Possible Long-term Outcome: • Reduced time intervals between the generation of R&D findings and the ethically sound application of findings to eliminate disparities between people with disabilities and the general population in employment, health & function, and participation and community living.

16. Part 2: Describe NIDRR’s Evaluation Framework • Current Status: More emphasis on outcomes monitoring versus outcomes measurement and evaluation • Outcomes monitoring constitutes an ongoing system of reporting and tracking program operations and results typically through the use of administrative information systems and performance indicators, and is used to answer questions that ask for a description of “what’s happening.” • Outcome evaluation extends beyond monitoring to an examination of the extent to which outcomes are caused by the program. Outcomes evaluation requires stronger evidence and seeks to answer questions like “What difference does the program make and what about the program is responsible for the difference”?

17. Components NIDRR’s Performance Monitoring & Evaluation System • Annual Performance Reporting (APR) form -- primary data warehouse for NIDRR’s outcomes monitoring and evaluation system is the. • Performance Measures – out of 7 programmatic GPRA measures 2 pertain directly to KT. • Annual Performance Assessment Expert Review (APAER) process • Phase 1 – external quality review of individual grantee accomplishments documented in APR and related to one of major life domain of NIDRR Logic Model; and • Phase 2 – expert panel review of aggregate accomplishments to assess progress towards NIDRR long-term goals.

18. How NIDRR Currently Measures KT Performance: GPRA Measures • To assess the effective use of NIDRR-funded scientific-based knowledge, technologies, and applications, the agency’s collects data from the APR to determine: • The average number of publications per award based on NIDRR-funded research and development activities in refereed journals.   • Results: For calendar years 2004 to 2006, the average number of peer-reviewed publication per award for the five eligible program categories is 2.71, 3.81, and 2.71, respectively.   • The number of new or improved NIDRR-funded assistive and universally designed technologies, products, and devices transferred to industry for potential commercialization.   • Results for FY 2006 & 2007 available 12-31-07.

19. How NIDRR Assesses Quality of Program Outputs & Outcomes -- Phase 1 of APAER • Purpose:To satisfy GPRA R&D measure #2.1, which calls for "the number of accomplishments developed and/or tested with NIDRR funding that have been judged by expert panelists to be of high-quality and to advance the field," the agency contracts with an outside organization to provide external reviews of individual grantee accomplishments associated a major life domain under the Long-Range Plan. • Focus of 2007 APAER: Participation and Community Living (P/CL) Domain • Data: 84 individual accomplishments (76 outputs and 8 outcomes) documented in the APR from 37 awards were considered eligible for external review based on completeness of information.

20. How NIDRR Assesses Quality of Program Outputs and Outcomes: Phase 1 of APAER (Continued) • Methods: • Reviewers were asked to rate the quality and contribution to the field of each accomplishment using a 5-point scale for outputs and a 3-point scale for outcomes, with each scale ranging from no contribution to outstanding contribution. • Ratings were collected from 12 reviews. To establish a basis for estimating rater effects, 10 accomplishments were rated by all reviewers with the remaining 76 accomplishments overlapping across reviewers and rated by three reviewers. A total of 272 ratings were available for outputs and 34 for outcomes.

21. How NIDRR Assesses Quality of Program Outputs and Outcomes: Phase 1 of APAER (Continued) • Data Analysis: Used Many-faceted Rasch Measurement (MFRM) statistical technique to control for rater effects in arriving at a composite rating for each accomplishment. • Using this approach, it is possible: • To determine if there are significant differences in the ratings across accomplishments and reviewers and then to identify which accomplishments are more highly and which are more lowly rated and which reviewers are more severe and which are more lenient in their ratings; • Adjust the ratings based on reviewer severity and produce a composite rating that is reported in terms of logits (or log-odds probabilities), which represent equal-interval scales that that allow for analysis using parametric statistical techniques.

22. How NIDRR Assesses Quality of Program Outputs and Outcomes: Phase 1 of APAER (Continued) • Quantitative Results: Of the 84 accomplishments rated: • 49 (or 58%) were judged to be of high-quality and advancing the field based on adjusted composite rating, which is equivalent to a rating of substantial or higher on the observed scale. • Of these, only 6 were determined to be outcomes, which may be due primarily to the way performance data were collected in the APR with an emphasis on publications, tools and measures; technology products, and informational products. • Significance: First time NIDRR has had reliable data to satisfy GPRA performance measure 2.1, and to use to establish a baseline against which we can demonstrate progress towards long-term goals, consistent with PART requirements.

23. Part 3: Examples “Noteworthy” Accomplishments: Phase 1 APAER • Preliminary Impressions of Qualitative Data: Of the 43 outputs and 6 outcomes judged to be advancing the field: • The vast majority were the result of R&D activities, with few direct links to KT activities. Again, this may be due to the way performance data are collected in the APR; • None of the 6 outcomes and only 1 of the 43 outputs appears to correspond to the types of suggested KT outcomes presented above; • However, more than half of the outputs describe findings that appeared to be “ready” for the application of KT strategies to accelerate use or uptake based on strength of presented; • And, several of the accomplishments described utilizing KT strategies in the generation of the output – e.g., use of networks or partnerships among researchers, developers, and industry representatives.

24. Part 3: Examples “Noteworthy” Accomplishments – Ready for Application of KT?# 1 • Program Mechanism: RRTC • NIDRR LRP Domain: Employment • Accomplishment Type: Output, Peer-reviewed Lit Review • Objective and Key Findings: • Summarizes research knowledge from fields of disability, economics, health care, and labor force research, to describe the nature of barriers to paid work and economic security for people with mental disorders; and • Describes how findings can be applied to systems change activities that can be pursued at local, state, and federal levels. • Also summarizes NIDRR-funded research that was used by the President’s New Freedom Commission on Mental Health.

25. Part 3: Examples “Noteworthy” Accomplishments – Ready for Application of KT? #2 • Program Mechanism: RRTC • NIDRR LRP Domain: Participation and Community Living • Accomplishment Type: Output, Peer-reviewed Pub • Objective and Key Findings: • Uses most recent available data (2002) to present three per participant expenditure comparisons between Medicaid HCBS waivers (which require that participants have an institutional level of care need) and institutional care: (1) program expenditures; (2) total Medicaid expenditures, and (3) estimated total public expenditures. • Analysis estimates that when compared with Medicaid institutional care in 2002, HCBS waivers produced a national average public expenditure saving of $43,947 per participant.

26. Part 3: Examples “Noteworthy” Accomplishments – Ready for Application of KT? #2 (Continued) • Program Mechanism: RRTC • NIDRR LRP Domain: Participation and Community Living • Accomplishment Type: Output, Peer-reviewed Pub • Potential Policy Implications: • As long-term care policy makers struggle with competing challenges including state budget deficits and pressures to expend home and community based services (HCBS), there is a pressing need for information on the comparative cost of Medicaid HCBS and institutional care.

27. Part 3: Examples “Noteworthy” Accomplishments – Already Applying KT Strategies #3 • Program Mechanism: RRTC • NIDRR LRP Domain: Participation and Community Living • Accomplishment Type: Output, Community of Practice • Objective & Desired Outcomes: • Developed Early Childhood Family Support CoP to foster wisdom-based action, including integration of best available research, for the ultimate benefit of families. Outcomes envisioned: • For families and practitioners that they become informed and inspired to implement wise solutions. • For researchers that they will be guided by families and practitioners as they design future research.

28. Part 3: Examples “Noteworthy” Accomplishments – Already Applying KT Strategies #3 • Program Mechanism: RRTC • NIDRR LRP Domain: Participation and Community Living • Accomplishment Type: Output, Community of Practice • Objective & Desired Outcomes (continued): • For policy leaders that they work in with families, professionals, and researchers to translate best available research and family wisdom into meaningful policy. • KT Strategies: Received RUSH grant to: • Assess the effectiveness of CoP implementation; • Measure gains in knowledge utilization based on random sample of CoP members; and • Utilize mixed-method design to document changes in behavior of stakeholder groups related to using CoP.

29. Conclusions & Next Steps: • NIDRR has only just embarked on the road to measurement of KT outcomes. • Much work needs to be done to get agreement on KT outcomes; chart pathways of KT outcomes; formulate KT outcome goals, and identify indicators of success. • However, significant process has been made in: • Improving performance reporting and data collection systems; • Establishing a performance monitoring system to track KT accomplishments; and • Launching a performance assessment expert review process to rate individual grant accomplishments for contribution to advancing the field and to assess the overall progress towards NIDRR’s long-term goals.

30. Thank you