Peer Review Advisory Committee 2/1/2010 Pete Morton and Paul Sheehy

1. Electronic Research Administration and the Receipt and Referral of Grant Applications Peer Review Advisory Committee 2/1/2010 Pete Morton and Paul Sheehy

2. Context and Drivers • Continuous evolution of Processes and Policies • Interdependencies of Business Processes Evolving needs of NIH’s extramural activities: • IMPAC II is nearing 20 years old • Original System Structure Needs Refreshment • Lower Operations and Maintenance Costs • Greater Flexibility New technologies are now available • IT Systems Development and Business Processes must be closely linked • Do not want to simply re-build the same thing without seeing if business processes would benefit from a change Fundamental shift in the way IT is developed:

3. Three distinct activites IT Refreshment Strategy Three distinct activities: • Develop New Hardware Infrastructure • Servers and SANs • Status: Complete • Develop New Software Infrastructure • Underlying IT applications and data tables • Status: Ongoing • Reengineer Applications “Evergreening” • Incorporate new infrastructure capacities • Refresh IT to incorporate new policies and business processes • Status: Starting

4. Evergreening a Business Function • General Strategy • Identify component business processes • Analyze current processes • Opportunity to identify and assess potential new support functions • Re-engineer IT systems • Approach • Create Business Process Models • “Current / As Is” state vs. “Future / To Be” state • Validate using focus groups of business subject matter experts • Model drives/directs IT re-engineering • Business Process Re-engineering (BPR) • Synthesis of Business Process Modeling and IT Reengineering

5. Business Process Reengineering Benefits • Modeling ensures an excellent understanding of actions and rules between business processes and supporting IT systems • Establishes credibility • Modeling provides a “refreshed examination of business processes” before reengineering the IT support for those processes • Better visualization of inefficiencies and pain points • Holistic view reduces surprises • If the existing model is accurate and adequate, can proceed directly to IT re-engineering • Synthesis allows for faster, cheaper software development • Better defined target

6. Introduction: BPR of CSR Receipt & Referral • As part of a larger effort to examine eRA reengineering, a pilot project to BPR CSR Receipt & Referral was initiated in June 2008 • Mandate • Examine and document current referral business process re: IC and IRG assignment • Measure process performance: efficiency and quality of assignment • Understand referral process’ benefit to NIH’s scientific mission • Understand causes of sub-optimal process performance • Develop heuristic future state business process designed to improve performance and quality of assignment

7. Business Process Reengineering the Receipt and Referral Module of eRA: a Pilot Evergreening eRA

8. Introduction: Division of Receipt and Referral (DRR) • DRR fulfills four business purposes: • Application assignment based on the scientific content of the application • Technical merit review group • Institute or Center • Application quality control • Policy enforcement • Facilitate and manage changes to assignment • All are critical to NIH’s scientific mission

9. Introduction: Business Challenges • Increase in volume of applications • Requirements for consistency, flexibility, transparency, equity, accountability • Aggressive implementation of electronic submission • Sometimes problematic IMPAC II migration to web • Increasing complexity of assignment decisions • Clinical, translational and multi-disciplinary research • Complexity is both review (subject matter, techniques, etc) and referral (relevance to multiple IRGs and/or IC missions) • e.g. A developmental disorder with cardiac & behavioral manifestations

10. Approach • Validate current state business process models developed by DRR and Office of the Chief IT Architect • Establish metrics for the process • Identify areas of challenge for focus • Perform root cause analysis on these areas • Review potential solutions to issues including technology and process changes • Develop a new business process model

11. Methodology: Components Two methodologies needed: • Modeling: To document current processes and heuristic future process • Business Genetics XBML • Metrics: To measure efficiency and effectiveness of current process and provide input to future state processes • Lean Six Sigma

12. Methodology: Activities • Phase 1: Update of existing model • 2006 OCITA current state business process model used as a foundation • Project team, DRR staff, IRG chiefs reviewed, corrected/updated as necessary • Focus on IRG assignment, not IC • Phase 2: Obtaining process metrics • DRR staff and IRG chiefs provided estimates of the normal time taken for each major business process step • “Normal” defined as time required for 80% of applications

13. Findings Evergreening eRA

14. Results: Process Efficiency • 80% of Applications processed within one day • For this 80% of applications, metrics indicate overall efficiency is world class • Work/wait time distribution equals benchmarks regardless of industry • Wait time generated in part by policy • Conclusions • Efforts focused on reducing “wait-time” of grant applications will result in minimal improvement to referral process performance • Process changes should focus on the 20% of applications that take longer to process • However, there is still significant benefit to automating the less complex assignments

15. Results: Process Efficiency (2) • Remaining applications much more variable in time to refer • 20% of total applications • 2/3 due to problems with application itself • 13% of total applications • 1/3 are more complex to assign • 7 % of total • Scientific overlap between IRGs • Complexity of application

16. Distribution of Issues of Problematic 13% of Applications

17. Results: Process Efficiency of Complex Assignments • Factors contributing to complex assignment: • Overlaps/gaps in IC mission • Variation in sponsorship and use of activities • Complexities of locus of review • All require scientific judgment • Quality of assignment • Essential to validity of NIH Peer Review Process • BP and IT changes must retain and enhance quality of assignment going forward (i.e. not bouncing back from SRG) • Assignment delays impact review groups • May delay meetings • May require a new special emphasis panel

18. System Changes: Issues with the eRA System • CSR staff discussions yielded the following issues with eRA: • Performance and reliability of Referral and Review modules • Fragility – Changes tend to introduce new problems • Security model does not allow others to view all referral data • Notes Field • Only place to track referral process e.g. Referral officer and IRG Chief and SRO comments • Lack of carryover between Receipt and Referral (RR) and Review (REV) modules • DRR uses RR, IRG Chiefs and SROs use REV, notes are not visible to each other • Lack of date/time/author stamping • Lack of carryover from ExitRamp • Release of applications and visibility to reviewers in IAR • Web-related issues • Inability to open more than one application window at a time

19. System Changes: DRR Priority List DRR staff-identified priorities: • Support for fully electronic Awaiting Receipt of Application (ARA) notices • IRG Chief push to final assignment • Changes, fixes and improvements in the ACR system • Implement single 2 day extension of the 3 day assignment window • Support for automated comparison of application content • Identification of duplicate applications and revisions • COTS and custom KM products to compare current and previous submissions

20. Recommendations Evergreening eRA

21. Future Model: Changes • Suggested changes to existing processes: • Application Validation • Automate and move validations currently performed by CSR after application receipt to eSubmission process • Expand types of checks and validations applied to application prior to submission beyond the set limited by 424RR form set • Use these checks to either generate warnings for applicant or flag applications for review • Provide a mechanism for applications to validate their application prior to submission – “Pre-submission portal” • Incorporate the ability to create a special, customized workflow for specific applications that require handling outside the normal flow of operations • Modify the 3-day window for application release

22. System Changes: Automation Opportunities Other changes required in order to implement the “To Be” process • Provide system to support validation of an application against all Grants.gov and NIH business rules prior to submission to NIH “Pre-submission portal” • Support for a fully structured electronic “cover letter” • Could be implemented in Commons without need for new OMB form • Enhance ARWS to production ready status • Fully integrate with eRA, especially notes • Make ARWS decisions trainable • Implementation of a flexible, user-controlled workflow system • Expand the eSubmission business rules • Represent Funding Opportunity Announcement as electronic business rules • Represent of referral guidelines as electronic business rules

23. Suggested Priorities • Immediate • Fixes for eRA issues, especially notes • Support IRG Chief push to final assignment and 3 day window extension (currently planned) • Implement Electronic Cover Letter in Commons • Consider providing an optional electronic form to be submitted as an interim measure • Fixes for ACR system

24. Suggested Priorities – cont’d • Medium Term • Implement Pre-Submission Portal • Expand and harden eSubmission business rules • Implement ARWS at production quality • Implement Electronic ARAs • Pilot application comparison tools • Long Term • Implement Flexible Workflow • Implement electronic business rules for: • Referral Guidelines (IC assignment) • Funding Opportunity Announcement (application validation)

25. Software Reengineering Phase Evergreening eRA

26. Next Steps (eRA) Goal: Brief review/update of desired business changes resulting from BPR then rebuild IT support for R&R incorporating desired changes • Form core focus group • Includes staff from Division of Receipt and Referral as well as CSR IRG chiefs and IC representatives • First meeting scheduled for mid-February

27. Next Steps (eRA) -- Continued • Working with core focus group, develop detailed requirements for new R&R module • Design to use new eRA software infrastructure • Initial detailed requirements documented –mid-summer • Develop prototype of new module – early fall • Update requirements as appropriate

28. Next Steps (eRA) -- Continued • Begin iteration 1 development in fall, 2010 • Work with focus group to test initial module and establish priorities for iteration 2 development – spring, 2011 • Begin development of iteration 2 – summer, 2011 • Work with focus group to test iteration 2 module and plan subsequent development if needed – late summer, 2011 • Finalize and deploy new module into production – September 30, 2011

29. Where do we go from here? Evergreening eRA

30. The next step: Review Module • As BPM is complete for the Referral module and software reengineering is starting, the next opportunity is the review module. • Much bigger challenge • peer review policy and procedures are interpreted and executed in a variety of ways • Operational Divisions (OpDivs) under DHHS authority utilize NIH resources but don’t necessarily follow NIH policy • Agency Partners utilize NIH resources but don’t necessarily follow NIH policy • Initial recruiting is underway • Governance • Subject matter experts

31. Review by NIH Governance • Trans-NIH Senior Staff • Extramural Activities Working Group (EAWG) • Working group of NIH Steering Committee with primary responsibility for extramural research and research training • Information Technology Working Group (ITWG) • Working group of NIH Steering Committee with primary responsibility for NIH's enterprise-level IT programs and investments. • Functional Areas • Extramural Program Management Committee (EPMC) • Senior leadership of IC extramural research and research training • Review Policy Committee (RPC) • Chiefs of Review Offices • Review Users Group (RUG) • SROs with interest in electronic research administration (NIH and IC)