1 / 17

JACK E. SCOTT, MARGARET BLASINSKY and MARY DUFOUR The Madrillon Group Inc., Vienna, Virginia

PUTTING A VALUE ON BIOMEDICAL RESEARCH PROGRAMS : ADAPTING THE RESEARCH PAYBACK FRAMEWORK FOR APPLICATION IN THE UNITED STATES. JACK E. SCOTT, MARGARET BLASINSKY and MARY DUFOUR The Madrillon Group Inc., Vienna, Virginia G. STEPHANE PHILOGENE Office of Behavioral and Social Sciences Research

veata
Download Presentation

JACK E. SCOTT, MARGARET BLASINSKY and MARY DUFOUR The Madrillon Group Inc., Vienna, Virginia

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. PUTTING A VALUE ON BIOMEDICAL RESEARCH PROGRAMS: ADAPTING THE RESEARCH PAYBACK FRAMEWORK FOR APPLICATION IN THE UNITED STATES JACK E. SCOTT, MARGARET BLASINSKY and MARY DUFOUR The Madrillon Group Inc., Vienna, Virginia G. STEPHANE PHILOGENE Office of Behavioral and Social Sciences Research National Institutes of Health, Bethesda, Maryland and Rachel J. Mandal National Cancer Institute, Bethesda, Maryland American Evaluation Association, Anaheim, California November 5, 2011

  2. OVERVIEW OF PRESENTATION • The Mind-Body Interactions and Health Program • Overview of the Payback Framework • Design and Methods of the MBIH Evaluation • Findings from the Evaluation • Lessons Learned from Applying the Payback Framework

  3. MIND-BODY INTERACTIONS AND HEALTH PROGRAM Rationale: • Cognitions, emotions and stress can affect physical & mental health • Some mind-body interventions can improve health • Mechanisms of these effects not understood MBIH Program: • Established by Congress in 1999 • Ten-year, trans-NIH program directed by OBSSR • 12 partnering NIH Institutes and Centers • $175 million dollars • 15 MBIH research centers • 44 investigator-initiated grants

  4. THE RESEARCH PAYBACK FRAMEWORK • Developed in mid-1990s by Martin Buxton, Stephen Hanney, and Health Economics Research Group (HERG) at Brunel University, London • Two components: • Multidimensional categorization of research benefits • Input-process-output-outcome model of the research process

  5. TYPES OF RESEARCH BENEFITS

  6. PAYBACK FRAMEWORK RESEARCH LOGIC MODEL Stock or Reservoir of Knowledge Internal Reservoir of Knowledge & Expertise (Center Level) Direct Feedback Paths Stage 4: Secondary Outputs: Policy and Product Development Stage 6: Final Outcomes Health & Health Sector and Broader Economic & Social Impacts Stage 0: Topic or Issue Identification Interface A: Project Specification & Selection Stage 5: Adoption by Clinicians & Public Stage 3: Primary Outputs Stage 1: Inputs Stage 2: Processes Interface B: Dissemination Direct Impact from Processes and Primary Outputs to Adoption by Clinicians and Public

  7. MBIH PROGRAM EVALUATIONDESIGN & METHODS • Cross-sectional mixed-methods design • Focus on overall achievements: • Program as a whole • MBIH research centers • MBIH investigator-initiated research projects • Six broad evaluation questions • Data sources: program documents, NIH databases, semi-structured interviews, bibliometric analyses

  8. MBIH RESEARCH CENTER INTERNAL STUDIES PILOT STUDIES • 209 pilot studies (14/15 centers) • Smaller studies led by post-doctoral fellows or junior faculty • 1-2 years duration • Predominantly clinical research (79 percent) versus basic research (11 percent) or both (10 percent) SUB-PROJECTS • 78 sub-projects (11/15 centers) • Larger in scope and scale (similar to R01) led by established investigators • 3-4 years in duration • Clinical research predominant (48 percent), but more basic science (32 percent) and both (20 percent)

  9. PAYBACK BENEFIT CATEGORIESKNOWLEDGE PRODUCTIVITY • Published articles identified from NIH e-SPA database between January 1, 2000 and December 31, 2009 • 429 publications (336 peer-reviewed research articles and 93 non-research articles as defined by Pub-MED) • 171 oral presentations • Comparison of publications and presentations by pilot studies and sub-projects: • Pilot studies: 64 percent of studies led to oral presentation and 47 percent led to a publication • Sub-projects: 47 percent of studies led to oral presentation and 69 percent led to publication

  10. PAYBACK BENEFIT CATEGORIESRESEARCH TARGETING & CAPACITY DEVELOPMENT Research Targeting: NIH Spin-off Grants Research Targeting: Non-federal Grants Could not collect & verify numbers and amounts of non-federal grants 13/15 centers obtained funding from non-federal sources 10/13 obtained funds from foundations 9/13 obtained funds from host institutions • 100 of 287 studies (pilots + sub-projects) led to a new verified NIH spin-off grant • Spin-off grants =$184,781,090 in new NIH funding • Represented $1.95 in new NIH funds for each dollar spent on MBIH centers • Spin-off ratio for 13 centers ranged from $0.07 to $7.93

  11. PAYBACK BENEFIT CATEGORIESRESEARCH TARGETING & CAPACITY DEVELOPMENT CAREER DEVELOPMENT • Investigator awards and honors (2/15) • Graduate and post-graduate training (8/15) • Faculty promotions including tenure (8/15) • Recruitment of new faculty (9/15) • Formation of new collaborations (15/15) TECHNICAL CAPACITY • New research tools, methods and models (11/15) • Used by other investigators at host institution (10/11) • Used by other investigators at other institutions (7/11) • New research infrastructure (10/15) • New equipment or instruments • New services

  12. PAYBACK BENEFIT CATEGORIESINFORMING POLICY AND NEW PRODUCT DEVELOPMENT • Policy considered broadly (both administrative and clinical) • Three indicators: • Research influenced policy formulation/development (9/15) • Research influenced clinical guidelines or treatment recommendations (6/15) • Research influenced medical or healthcare professional education or training (13/15) • Examples: • “White coat” hypertension • National curriculum for mind-body medicine in family residencies

  13. PAYBACK BENEFIT CATEGORIESHEALTH AND HEALTH SECTOR • Requires some degree of adoption of new practices or behaviors by clinical practitioners • Three indicators: • Adoption by clinical practitioners (9/15) • Changes in healthcare service delivery (9/15) • Improvements in health outcomes and quality of life (6/15) • More difficult for PIs to report because they do not usually track this

  14. PAYBACK BENEFIT CATEGORIESBROADER ECONOMIC AND SOCIAL IMPACTS • Most difficult type of benefit to measure • Cost analyses are often not available • Time it can take for a finding to affect clinical practice and health • Attributing an impact to a specific single research project • 4/15 PIs believe that their MBIH research will produce broader economic and social benefits

  15. COMPARING RESEARCH CENTERS ACROSS PAYBACK CATEGORIES • Centers can be compared in terms of individual payback benefit categories, but more interesting to compare centers on all five categories at once • Case study scoring system • Indicators rated on scales scored from 0 to 5 • Ratings on 7 indicators conducted by members of Madrillon evaluation team (82 percent agreement on ratings) • Some indicators combined to represent benefit category • Final ratings for five domains (Research targeting; capacity development; effects on medical or healthcare professional training; effects on policy; and adoption by clinical practitioners)

  16. COMPARISON OF TWO MBIH CENTERSUSING RADAR GRAPHS MBIH Research Center D MBIH Research Center H

  17. LESSONS LEARNED FOR FUTURE APPLICATIONSOF THE PAYBACK FRAMEWORK • Payback Framework well-suited to evaluations of NIH biomedical research programs • MBIH research centers had important impacts in all five benefit categories • Greatest challenge is the case study scoring methodology • As more Payback Framework evaluations are completed, it may become possible to establish benchmarks

More Related