1 / 31

Peer Review Advisory Committee 2/1/2010 Pete Morton and Paul Sheehy

Electronic Research Administration and the Receipt and Referral of Grant Applications. Peer Review Advisory Committee 2/1/2010 Pete Morton and Paul Sheehy. Context and Drivers. Continuous evolution of Processes and Policies Interdependencies of Business Processes.

HarrisCezar
Download Presentation

Peer Review Advisory Committee 2/1/2010 Pete Morton and Paul Sheehy

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Electronic Research Administration and the Receipt and Referral of Grant Applications Peer Review Advisory Committee 2/1/2010 Pete Morton and Paul Sheehy

  2. Context and Drivers • Continuous evolution of Processes and Policies • Interdependencies of Business Processes Evolving needs of NIH’s extramural activities: • IMPAC II is nearing 20 years old • Original System Structure Needs Refreshment • Lower Operations and Maintenance Costs • Greater Flexibility New technologies are now available • IT Systems Development and Business Processes must be closely linked • Do not want to simply re-build the same thing without seeing if business processes would benefit from a change Fundamental shift in the way IT is developed:

  3. Three distinct activites IT Refreshment Strategy Three distinct activities: • Develop New Hardware Infrastructure • Servers and SANs • Status: Complete • Develop New Software Infrastructure • Underlying IT applications and data tables • Status: Ongoing • Reengineer Applications “Evergreening” • Incorporate new infrastructure capacities • Refresh IT to incorporate new policies and business processes • Status: Starting

  4. Evergreening a Business Function • General Strategy • Identify component business processes • Analyze current processes • Opportunity to identify and assess potential new support functions • Re-engineer IT systems • Approach • Create Business Process Models • “Current / As Is” state vs. “Future / To Be” state • Validate using focus groups of business subject matter experts • Model drives/directs IT re-engineering • Business Process Re-engineering (BPR) • Synthesis of Business Process Modeling and IT Reengineering

  5. Business Process Reengineering Benefits • Modeling ensures an excellent understanding of actions and rules between business processes and supporting IT systems • Establishes credibility • Modeling provides a “refreshed examination of business processes” before reengineering the IT support for those processes • Better visualization of inefficiencies and pain points • Holistic view reduces surprises • If the existing model is accurate and adequate, can proceed directly to IT re-engineering • Synthesis allows for faster, cheaper software development • Better defined target

  6. Introduction: BPR of CSR Receipt & Referral • As part of a larger effort to examine eRA reengineering, a pilot project to BPR CSR Receipt & Referral was initiated in June 2008 • Mandate • Examine and document current referral business process re: IC and IRG assignment • Measure process performance: efficiency and quality of assignment • Understand referral process’ benefit to NIH’s scientific mission • Understand causes of sub-optimal process performance • Develop heuristic future state business process designed to improve performance and quality of assignment

  7. Business Process Reengineering the Receipt and Referral Module of eRA: a Pilot Evergreening eRA

  8. Introduction: Division of Receipt and Referral (DRR) • DRR fulfills four business purposes: • Application assignment based on the scientific content of the application • Technical merit review group • Institute or Center • Application quality control • Policy enforcement • Facilitate and manage changes to assignment • All are critical to NIH’s scientific mission

  9. Introduction: Business Challenges • Increase in volume of applications • Requirements for consistency, flexibility, transparency, equity, accountability • Aggressive implementation of electronic submission • Sometimes problematic IMPAC II migration to web • Increasing complexity of assignment decisions • Clinical, translational and multi-disciplinary research • Complexity is both review (subject matter, techniques, etc) and referral (relevance to multiple IRGs and/or IC missions) • e.g. A developmental disorder with cardiac & behavioral manifestations

  10. Approach • Validate current state business process models developed by DRR and Office of the Chief IT Architect • Establish metrics for the process • Identify areas of challenge for focus • Perform root cause analysis on these areas • Review potential solutions to issues including technology and process changes • Develop a new business process model

  11. Methodology: Components Two methodologies needed: • Modeling: To document current processes and heuristic future process • Business Genetics XBML • Metrics: To measure efficiency and effectiveness of current process and provide input to future state processes • Lean Six Sigma

  12. Methodology: Activities • Phase 1: Update of existing model • 2006 OCITA current state business process model used as a foundation • Project team, DRR staff, IRG chiefs reviewed, corrected/updated as necessary • Focus on IRG assignment, not IC • Phase 2: Obtaining process metrics • DRR staff and IRG chiefs provided estimates of the normal time taken for each major business process step • “Normal” defined as time required for 80% of applications

  13. Findings Evergreening eRA

  14. Results: Process Efficiency • 80% of Applications processed within one day • For this 80% of applications, metrics indicate overall efficiency is world class • Work/wait time distribution equals benchmarks regardless of industry • Wait time generated in part by policy • Conclusions • Efforts focused on reducing “wait-time” of grant applications will result in minimal improvement to referral process performance • Process changes should focus on the 20% of applications that take longer to process • However, there is still significant benefit to automating the less complex assignments

  15. Results: Process Efficiency (2) • Remaining applications much more variable in time to refer • 20% of total applications • 2/3 due to problems with application itself • 13% of total applications • 1/3 are more complex to assign • 7 % of total • Scientific overlap between IRGs • Complexity of application

  16. Distribution of Issues of Problematic 13% of Applications

  17. Results: Process Efficiency of Complex Assignments • Factors contributing to complex assignment: • Overlaps/gaps in IC mission • Variation in sponsorship and use of activities • Complexities of locus of review • All require scientific judgment • Quality of assignment • Essential to validity of NIH Peer Review Process • BP and IT changes must retain and enhance quality of assignment going forward (i.e. not bouncing back from SRG) • Assignment delays impact review groups • May delay meetings • May require a new special emphasis panel

  18. System Changes: Issues with the eRA System • CSR staff discussions yielded the following issues with eRA: • Performance and reliability of Referral and Review modules • Fragility – Changes tend to introduce new problems • Security model does not allow others to view all referral data • Notes Field • Only place to track referral process e.g. Referral officer and IRG Chief and SRO comments • Lack of carryover between Receipt and Referral (RR) and Review (REV) modules • DRR uses RR, IRG Chiefs and SROs use REV, notes are not visible to each other • Lack of date/time/author stamping • Lack of carryover from ExitRamp • Release of applications and visibility to reviewers in IAR • Web-related issues • Inability to open more than one application window at a time

  19. System Changes: DRR Priority List DRR staff-identified priorities: • Support for fully electronic Awaiting Receipt of Application (ARA) notices • IRG Chief push to final assignment • Changes, fixes and improvements in the ACR system • Implement single 2 day extension of the 3 day assignment window • Support for automated comparison of application content • Identification of duplicate applications and revisions • COTS and custom KM products to compare current and previous submissions

  20. Recommendations Evergreening eRA

  21. Future Model: Changes • Suggested changes to existing processes: • Application Validation • Automate and move validations currently performed by CSR after application receipt to eSubmission process • Expand types of checks and validations applied to application prior to submission beyond the set limited by 424RR form set • Use these checks to either generate warnings for applicant or flag applications for review • Provide a mechanism for applications to validate their application prior to submission – “Pre-submission portal” • Incorporate the ability to create a special, customized workflow for specific applications that require handling outside the normal flow of operations • Modify the 3-day window for application release

  22. System Changes: Automation Opportunities Other changes required in order to implement the “To Be” process • Provide system to support validation of an application against all Grants.gov and NIH business rules prior to submission to NIH “Pre-submission portal” • Support for a fully structured electronic “cover letter” • Could be implemented in Commons without need for new OMB form • Enhance ARWS to production ready status • Fully integrate with eRA, especially notes • Make ARWS decisions trainable • Implementation of a flexible, user-controlled workflow system • Expand the eSubmission business rules • Represent Funding Opportunity Announcement as electronic business rules • Represent of referral guidelines as electronic business rules

  23. Suggested Priorities • Immediate • Fixes for eRA issues, especially notes • Support IRG Chief push to final assignment and 3 day window extension (currently planned) • Implement Electronic Cover Letter in Commons • Consider providing an optional electronic form to be submitted as an interim measure • Fixes for ACR system

  24. Suggested Priorities – cont’d • Medium Term • Implement Pre-Submission Portal • Expand and harden eSubmission business rules • Implement ARWS at production quality • Implement Electronic ARAs • Pilot application comparison tools • Long Term • Implement Flexible Workflow • Implement electronic business rules for: • Referral Guidelines (IC assignment) • Funding Opportunity Announcement (application validation)

  25. Software Reengineering Phase Evergreening eRA

  26. Next Steps (eRA) Goal: Brief review/update of desired business changes resulting from BPR then rebuild IT support for R&R incorporating desired changes • Form core focus group • Includes staff from Division of Receipt and Referral as well as CSR IRG chiefs and IC representatives • First meeting scheduled for mid-February

  27. Next Steps (eRA) -- Continued • Working with core focus group, develop detailed requirements for new R&R module • Design to use new eRA software infrastructure • Initial detailed requirements documented –mid-summer • Develop prototype of new module – early fall • Update requirements as appropriate

  28. Next Steps (eRA) -- Continued • Begin iteration 1 development in fall, 2010 • Work with focus group to test initial module and establish priorities for iteration 2 development – spring, 2011 • Begin development of iteration 2 – summer, 2011 • Work with focus group to test iteration 2 module and plan subsequent development if needed – late summer, 2011 • Finalize and deploy new module into production – September 30, 2011

  29. Where do we go from here? Evergreening eRA

  30. The next step: Review Module • As BPM is complete for the Referral module and software reengineering is starting, the next opportunity is the review module. • Much bigger challenge • peer review policy and procedures are interpreted and executed in a variety of ways • Operational Divisions (OpDivs) under DHHS authority utilize NIH resources but don’t necessarily follow NIH policy • Agency Partners utilize NIH resources but don’t necessarily follow NIH policy • Initial recruiting is underway • Governance • Subject matter experts

  31. Review by NIH Governance • Trans-NIH Senior Staff • Extramural Activities Working Group (EAWG) • Working group of NIH Steering Committee with primary responsibility for extramural research and research training • Information Technology Working Group (ITWG) • Working group of NIH Steering Committee with primary responsibility for NIH's enterprise-level IT programs and investments. • Functional Areas • Extramural Program Management Committee (EPMC) • Senior leadership of IC extramural research and research training • Review Policy Committee (RPC) • Chiefs of Review Offices • Review Users Group (RUG) • SROs with interest in electronic research administration (NIH and IC)

More Related