MR/AM&O Recommendations: Opportunities and Challenges

1. Business & Operations Advisory Committee PresentationMR/AM&O Recommendations: Opportunities and Challenges Arlington, VA November 18, 2004 The information contained in this presentation is sensitive and proprietary and not for distribution without prior approval.

2. Today’s Agenda MR/AM&O Improvement Opportunities Overview Appendix: Improvement Opportunities Groupings

3. The Business Process team analyzed the Merit Review (MR) and Award Management & Oversight (AM&O) processes in FY03-04 Merit Review Phases Award Management & Oversight Phases Program Announcement Proposal Develop.& Submission Intake Plan & Conduct Review Make Decision Make Award Award Management Award Oversight Award Close-Out Areas Examined • Single Disciplinary Proposals and Awards • Proposals and Awards for NSF-Wide Programs • Multidisciplinary Proposals and Awards • Centers and Infrastructure • Large Facility Project Proposals and Awards • Risk Management • Financial Oversight

4. The Team gathered data from NSF leadership, staff and customers • Internal Document Review: Reviewed internal NSF documents to better understand issues and opportunities within the organization regarding MR, AM&O, RA, PA&A and KM. Documents included policies and procedures and internal management documents that are leveraged in the performance of the processes. • External Reports: Reviewed external reports from GAO, NAS, NAPA, OMB, and external contractors/publications to better understand NSF’s performance as assessed by external parties. • Data Mining: Collected and analyzed internal and external data from a variety of systems and sources (EIS, FAS, PARS, FastLane, etc.) in order to determine monthly proposal deadlines and volume, reviewer utilization by Directorate, workload and portfolio balance, success rates and award size, Division and Directorate commitment patterns, budget cycle/submission variability, and historical “mortgage” rates. • 2003 Applicant Survey: Administered a survey to a stratified random sample of 6,185 NSF FY02 proposal applicants to gauge opinions on NSF’s Merit Review and Award Management and Oversight practices (~35% response rate) • Focus Groups:Conducted joint focus groups with PDs, DDs, EO/DADs, PAs, SAs, DGA Grants Officers, BPOs, BPLG and GIIC to gather insights on issues from multiple perspectives within the organization. Focus groups also considered issues related to human capital and tools and technology. • 1-on-1 Interviews: Conducted interviews with staff at all levels of the organization to gather additional data and provide additional depth to the analysis. • Team Exercises:Leveraged input from IPT members to assess the importance and feasibility of findings and opportunities. Collaborated across BP, HC, and TT subteams to better understand and analyze the impacts of findings and recommendations for NSF. Internal Doc Review External Reports Data Mining Applicant Survey Focus Groups 1-on-1 Interviews Team Exercises

5. The Team developed visions and goals for the MR and AM&O processes that focus on service to the research community “To-Be” Vision for Merit Review and Award Management & Oversight Merit Review: “Operate a credible, efficient merit review system that…” Award Management & Oversight: “A collaborative, multi-functional award management process that…” Merit Review “To-Be” Goals Award Management & Oversight “To-Be” Goals • Enables a quick, fair process that is humanizing to the research community • Identifies the people, ideas, and tools with the greatest potential for impact • Facilitates improved service to the research community • Provides true partnership with the research community in all phases of the scientific process • Provides flexible interactions across Directorates and Divisions and encourages consistency in the interpretation and application of NSF policies • Encourages proposals that are highly innovative or that may not fit into traditional programs • Attracts the best reviewers and expands the base of reviewers in terms of quality, diversity, and number • “Makes it easy to do business with NSF” • Applies a level of oversight that balances the degree of an award’s risk with available resources • Minimizes the burden on internal staff and the research community • Fosters partnership with the research community, including academia, private sector and other public sector organizations • Provides transparency, flexibility and ability to respond to changes in the environment • Verifies that projects are in compliance with award agreements and federal regulations • Encourages effective project management

6. Area of Opportunity Summary of Improvement Opportunities Merit Review • Improve the effectiveness and efficiency of the Merit Review process through – broadening the use of Letters of Intent, piloting a “funding-dependent” award category, and providing an electronic survey after issuing a decision • Reduce the workload demands of Program Directors by -- piloting a “customer service center” for all applicant or PI inquiries, automating compliance checking, enhancing tools for reviewer selection and identification, and streamlining the requirements for writing a review analysis • Improve how reviewers are identified, selected, and retained through – providing reviewer training, developing a new reviewer management system, offering incentives to “ad hoc” reviews, and building an NSF-wide system to accept reviewer applications • For large awards, modify the current FastLane proposal submission moduleto accommodate larger proposals Award Management & Oversight • Implement a risk-based approach to award management that allows Program Directors to manage “by exception” and delegate additional responsibilities to support staff • Further integrate business and programmatic functions to enhance quality and reduce workload • Enhance PDs’ ability to manage responsibilities through the use of workload management and workflow tools • Increase the utility of the Project Reporting System by modifying the report template and leveraging PIs in the creation of “nuggets” that are “discovery”- focused • For large awards, evaluate the cost and feasibility of providing additional programand project management support The result was the identification of over 30 potential improvement opportunities for Merit Review and Award Management & Oversight Source: Booz Allen Hamilton Analysis

7. The Team is currently developing implementation plans for the MR and AM&O process improvement opportunities MR and AM&O Process Design Development Review and Revise Design Options FY04, Q3 Analyze Data and Develop Design Options FY04, Q2 Assistant Directors’ Review FY05, Q1 Implementation Planning (Ongoing) Gather Data FY04, Q1 Integrated Process Team Meetings (IPT) Feedback Loops Preliminary Human Capital Requirements Baseline Workload Analysis IPT and Business Analysis Steering Committee (BASC) Review Senior Level Review Develop Implementation Plans Develop Preliminary Design Options Effective Practices Research Preliminary TT Requirements MR/AM&O Baseline Analysis Feedback Loops

8. A number of the improvement opportunities are focused on improving the NSF’s communication with and service to the scientific community Technology Policy/Process • Pilot a "customer service center" for all applicant and awardee inquiries • Issue electronic surveys at the time of decision to assess satisfaction with the quality of the review • Establish standard training that can be completed by a reviewer prior to performing a review • Automate the compliance check function and establish a module that allows a PI to self-audit a proposal prior to submission • Build a module that accepts online reviewer applications • Provide PIs and SROs with personalized webpages at time of award • Pilot a "funding-dependent" award category based on a percentage of a PD's expected budget • Broaden the use of Letters of Intent to determine program fit and begin review planning and coordination prior to proposal receipt • Enhance the technology that supports panels to allow more robust remote participation • Consider compensating "ad-hoc" or mail reviewers for their efforts • Pilot process that allows repeat applicants who are resubmitting full proposals to provide a one-page response to prior review's comments • Require only a total estimated budget at time of submission. PI provides detailed budget only after award decision (not applicable to larger awards)

9. Today, we would like to get your feedback on these opportunities and how we can ensure they are implemented successfully • Based on your experience with NSF and other organizations, would these improvement opportunities move NSF toward becoming a more customer-centric organization? • What do you see as the major benefits or drawbacks of these opportunities? • From your perspective, what are the major challenges associated with implementing these changes at the NSF? (cultural, resource, policy, technology, etc.) • What can we do as a Team to help ensure successful implementation of these opportunities, deriving the maximum benefits to the scientific community?

10. Today’s Agenda MR/AM&O Improvement Opportunities Overview Appendix: Improvement Opportunities Groupings

11. Bolded items identify those presented to the Assistant Directors Improvement opportunities grouped in the Organization/People category address organization structure, culture, and human capital strategy

12. Bolded items identify those presented to the Assistant Directors Process/Policy category includes improvement opportunities that affect the procedures and activities in the MR and AM&O process

13. Bolded items identify those presented to the Assistant Directors Process/Policy category includes improvement opportunities that affect the procedures and activities in the MR and AM&O process (cont.)

14. Bolded items identify those presented to the Assistant Directors Improvement opportunities under the Technology umbrella address key opportunities that were identified throughout the business analysis

15. Bolded items identify those presented to the Assistant Directors Improvement opportunities under the Technology umbrella address key opportunities that were identified throughout the business analysis (cont.)