National Institutes of Health

1. National Institutes of Health Use of E-Tools for Organizational Performance Assessments &Evidence-based Management: NIH GPRA & PART Deborah Duran, Ph.D. Systemic Assessments Branch Office of Portfolio Analysis & Strategic Initiatives

2. Overview of Presentation • NIH Overview • Government Performance • Program Performance Monitoring System (PPMS) • Information Data Collection • Knowledge Management • Reporting • Portfolio Analysis for goal replacement

3. NIH Role World’s premier medical research center and steward of medical and behavioral research for the nation NIH Goal Acquire new knowledge to help prevent, detect, diagnose, and treat disease and disability The Nation’s Leading Medical Research Agency

4. NIH Organizational Structure Clinical Center Center for Information Technology Center for Scientific Review Office of the Director Staff Offices Office of Extramural Research Office of Intramural Research Office of Management/Chief Financial Officer Office of Science Policy Office of Communications and Public Liaison Office of Equal Opportunity and Diversity Management Office of Program Coordination Office of Legislative Policy and Analysis Office of Community Liaison Executive Office Office of the Ombudsman/Ctr. for Cooperative Resolution NIH Ethics Office Immediate Office of the Director Office of the Director Program Offices Office of Research on Women’s Health Office of AIDS Research Office of Behavioral and Social Sciences Research Office of Disease Prevention National Institute on Alcohol Abuse and Alcoholism National Institute of Allergy and Infectious Disease National Institute of Arthritis and Musculoskeletal and Skin Diseases National Eye Institute National Heart Lung and Blood Institute National Human Genome Research Institute National Cancer Institute National Institute on Aging National Institute of Biomedical Imaging and Bioengineering National Institute of Child Health and Human Development National Institute of Deafness and Other Communication Disorders National Institute of Dental and Craniofacial Research National Institute of Diabetes and Digestive and Kidney Diseases National Institute on Drug Abuse National Institute of Environmental Health Sciences National Institute of General Medical Sciences National Institute of Neurological Diseases and Stroke National Institute of Nursing Research National Institute of Mental Health National Library of Medicine John E. Fogarty Center for Advanced Study in the Health Sciences National Center for Complementary and Alternative Medicine National Center for Minority Health and Health Disparities National Center for Research Resources

5. NIH Overview • 27 Institutes and Centers (ICs) work individually and collectively • Over ~$29 billion invested annually on medical research • 80% of budget awarded through 50,000 competitive grants to support over 325,000 scientists and research personnel in the U.S. and abroad • 10% of budget supports research led by ~6,000 intramural principal investigators

6. NIH HAS ONE PROGRAM: • Research • For NIH level performance reporting: • GPRA • PART

7. NIH mission: to uncover new knowledge that will lead to better health for everyone. GPRA Performance Driver Map DRIVERS FUNCTIONAL AREAS PERFORMANCE Scientific Research Outcomes • Short/Long term • Low/High risk • Intramural/Extramural Generation of new scientific knowledge of new or improved prevention, diagnosis, and treatment of disease • Knowledge Dissemination • Translation • Field input • Data / Repositories Communication and Transfer of Results Shared knowledge for public benefit Capacity Building and Research Resources • Training • Technology • Facilities • Partnerships Highly trained basic and clinical scientists; leadership in technology and facilities Strategic Management of Human Capital • Performance-based approach • Workplace Management • Adequate Workforce Availability of highly skilled and diverse workforce; enhanced work environment Program Oversight and Improvement Enhanced program management and compliance with relevant laws • Results-oriented Management • Resource Management • Stewardship

8. Core Criteria for Planning and Reporting Performance Goals • Representative (Disease/Disorders/Management) • Trans-NIH Approach • Annual Budget & Performance Target-Based Approach • Balanced Scientific Research Portfolio (Intramural/Extramural, Basic/Clinical, ICs) • R&D Criteria (Quality, Relevance, Performance) • 4 Ps Predictive, Preemptive, Personalized and Participatory • Scientific Difficulty (Low-Medium-High) • Time (Short-Term, Mid-Range, Long-Term)

9. Scientific Risk and Time Matrix Increasing Level of Difficulty 1-3 years 4-7 years 8-10 years Time

10. Annual Targets • Goals span from 1 to 10 years; therefore, assess annual performance • Prospective & Quantifiable Annual Scientific Targets • All IC/Office annual activities or contributions are called Strategic Targets • Reported Targets are called Performance Targets • Monitor all Targets but only submit Performance Targets • Representative Target Reporting • SROs: 1 performance target per goal • Non-SROs: 1-5 Performance targets per goal • Adaptive measure – scientific justification

11. PART Scores

12. Information Knowledge Transitions from Age of Information Technology / Data Management to Age of Knowledge / Collects & Connect • Capture data • Structure data • Analyze information • Design Application

13. Program Performance Monitoring System (PPMS) • Centralizes performance monitoring & reporting • Enhances coordination & communication among collaborators • Reduces workforce burden • Improves analysis of budget / performance information Two Components of PPMS: • Systemic Assessments Branch (SAB) Website (VPS Interface) • Useful links to relevant resources and performance-related information at NIH • Provides single point of access to NIH’s first centralized budget-performance reporting system • Calendar with relevant guidance • Visual Performance Suite (VPS) • Centralized online database for collecting, storing, analyzing and reporting trans-NIH performance and budget data • Maintain information used to plan activities at NIH • Central point of access

14. SAB Homepage – NIH View

15. SAB Homepage – GPRA Goal Performance Reporting

16. INFORMATION DATA COLLECTION

17. Budget Performance Integration (BPI) – Functional Area Level • Information structured to match NIH business process • Scorecard Configured to display budget and performance information • User can drill down to the next level of granularity

18. Functional Level to Goal Level • Goal level displays all goals and funding associated with the functional area • Displayed box illustrates navigation system designed to allow users to: • Navigate to any part of the system • Display summary of information for goal • Upload documents or form discussion using blog tread

19. Goal Data Entry Goal data entry screen has been configured to capture data necessary to support reporting needs-tailored quantitative and qualitative data fields

20. Goal Level to Targets Level

21. Target Data Entry / Tailored Fields • Tailored fields can be added to collect reportable and un-reportable information • Data collected at target levels automatically aggregate to goal levels

22. Budget Reporting • Budget is integrated with workflow of configuring Goals and Targets • Comment box appears/required when threshold for % change is crossed • Email alerts are sent to contacts when a goal is rejected or needs revision

23. Performance Page • Annual performance is collected on the performance page • An automated notification system alerts users when annual performance results are due or missed • Alert contains hyperlink that links users directly to reporting page

24. PART Reporting Page

25. KNOWLEDGE MANAGEMENT

26. Analytic Tool Strategy Map with Attributes • Goal linkage can be represented dramatically on a map • Attributes are tailored for sorting • Attributes provide snapshot of how the elements are performing

27. Decision-Makers’ Tools for Planning and Reporting • VPS produce graphics of budget-performance data in system • Graphical Analyses supports various types of charts & graphs and formats • Summarize budget performance data • Compare dimensions • Filter information • Drill into a dimension • Trending • Modeling • Resource for organizational performance analysis and portfolio management

28. Summarized Data(Core Criteria: Trans-NIH Approach) • X-Axis and Legend can be interchanged dynamically • Chart labels can also be updated

29. Compare Dimensions(Core Criteria: Scientific Risk) • Users can navigate from scorecard to graphics with a single mouse click • Preconfigured graphic can be further analyzed using menus on left panel • Control panel can be hidden to draw a full-screen chart

30. Filter Information • Data can be filtered so only “top n” “bottom n” or “middle n” is displayed • Graphics feature has the ability to query the database by specifying any number of properties of an element in a plan (For example, user can configure graphic to display all goals that has a specific word such as “cancer”).

31. Drill Into a Dimension(Core Criteria: Representative Performance Goals) • Users can drill down to the lowest level of a dimension using a filter • Chart can be display in multiple chart types, in 3D and with table

32. Charting Multiple Measures • Multiple measures can be simultaneously charted on a stacked vertical bar chart

33. REPORTING

34. VPS Reports • Analytic graphics can be saved as jpeg or gif files & imported into PowerPoint • Customized reports for each module or reporting requirement • 12 tailored budget performance reports are currently produced on VPS • Reports can be generated with a button push printed, exported, and/or saved as • MS Word documents • MS Excel spreadsheets

35. Sample VPS Report – Performance Goal Narrative

36. VPS Summary Report – Budget/Performance

37. VPS GPRA Budget Report

38. VPS PART Report

39. PORTFOLIO ANALYSIS FOR GPRA GOAL REPLACEMENT

40. Portfolio Analysis Using Core Criteria

41. Portfolio Analysis Using Funding $ Research, Condition, and Disease Categorization (RCDC) system can help identify • High priority research areas appropriate for representation in a GPRA goal • Potential goal lead

42. RCDC for GPRA $ reporting QVR/RCDC can be used to retrieve RCDC categorized grants based on several criteria (Text search, by IC, FY etc)

43. GPRA goal 3.4a using RCDC/QVR By 2013 identify and characterize at least 2 candidate genes influencing substance use, psychotic disorders using high risk family, twin, special populations Finger Print Concept Name Concept Rank Bipolar Disorder 95% Schizophrenia 85% Alcohol dependence 65% proband 65% Substance Use Disorder 65% endophenotype 60% Genome Scan 58% Family 58% Conduct Disorder 58% Major Depressive Disorder 58% Borderline pers Disorder 58% ADHD 58% Autistic Disorder 58% LD Mapping 58% Psychotic Disorders 50% Twin Multiple Birth 50% Alcoholism 50% Twin Studies 45% Stress Disorders 44% autism spectrum disorder 44% Pathological Gambling 41% Siblings 40% Personality Disorders 40% psychogenetics 40% Project list from intersection Mental Health Alcoholism Drug abuse Genetics Human Genome ICs NIMH, NIDA, NIAAA Key word search

44. Project list GPRA3.4a 5 R01AA015168-03 Alcoholism Susceptibility Genes in High Density Families 5 P50AA011998-09 MARC: Genetic Epidemiology of Alcoholism & Comorbidity 5 R01MH061613-07 Genetic Linkage Studies in Bipolar Disorder Families 5 R01DA013240-07 Twin Study of ADHD, CD and Substance Abuse 5 R01MH042243-16 Genetic Linkage and Association in Bipolar Disorder 3 R01MH060912-06S1 Genetics of Early-Onset Major Depression 1 R01MH079799-01 Using Genetics to Dissect Schizhoprenia, Bipolar Disorder, and Depression 5 R01MH059552-07 Genetics of Early Onset Depression 5 R01AA013320-05 Molecular Epidemiology of Alcoholism 2- Big Sibships 5 R01MH069856-03 Genetics of Bipolar Disorder in Latino Populations 5 R01MH059535-08 Collaborative Genetic Study of Bipolar Disorder

45. Analytic Results It is not a question how much a person knows, but what use s/he can make of what s/he knows. Josiah Gilbert Holland 1866 Data Management Data Analysis Information Knowledge Intuition Systematic

46. Systemic Assessments Branch Office of Portfolio Analysis & Strategic Initiatives (OPASI)National Institutes of Health (OD)9000 Rockville Pike Bldg 2, Rm 1E10Bethesda, MD  20892ph: 301.496.6205fax: 301.480.4716email: durand@od.nih.gov