Setting Priorities for Large Research Facility Projects: National Academies' Recommendations and NSF Report

1. Setting Priorities for Large Research Facility Projects – The National Academies Recommendations and the NSB/NSF Report Mark Coles Deputy Director, Large Facility Projects Office of Budget, Finance and Award Management

2. Introduction • National Academies study “Setting Priorities for Large Research Facility Projects” (Brinkman report) makes specific recommendations for prioritizing, planning, and overseeing the construction and operation of large facilities supported by NSF • The report cites the need to make greater resources available for pre-award planning of a project’s workscope • Implement rigorous post-award oversight through periodic external review AAAC Meeting - Feb. 16, 2005

3. Summary of the NAS Study Recommendations • Development of a 10-20 year facility roadmap • Three levels of criteria for ranking: • Scientific and technical merit • Agency Strategic Criteria • National Criteria • New starts ranked in annual budget request using clear rationale based on roadmap • Enhanced project pre-approval planning and budgeting • Greater independent oversight and review needed • Review effectiveness of Deputy Director for Large Facilities in two years • OSTP should coordinates roadmaps across agencies • NSF Leadership and NSB should pay careful attention to implementation of proposed reforms AAAC Meeting - Feb. 16, 2005

4. NSF/NSB Actions • NSF undertook a detailed analysis of the NAS report recommendations • Formal report prepared after broadly canvassing NSF for input • Committee formed under direction of Joe Bordogna: • John Hunt (chair) – Senior Advisor, Director’s Office • Wayne van Citters – AST Division Director, MPS • Rich Behnke – Section Head,ATM/GEO • Priscilla Nelson – Senior Staff Associate, Director’s Office • Mark Coles – Large Facilities Deputy Director, BFA • Patricia Crumley (secretary) – Program Analyst, Budget Division, BFA • NSF Assistant Directors and Director reviewed • Further NSB input and revision resulted in final version – on NSB web site (October 2004 Summary Report): http://www.nsf.gov/nsb/meetings/2004/1004/summary_report_1004_updt.pdf AAAC Meeting - Feb. 16, 2005

5. Main features of NSB/NSF Report • NSF is developing a facility “roadmap” • Embraces NAS recommendations for greater clarity and transparency to selection and prioritization process • NSF will take steps to enhance the robustness of the pre-construction project development process to improve cost projections for facilities recommended for construction AAAC Meeting - Feb. 16, 2005

6. NSF Roadmap • NSF is now developing a Facility Plan (the “roadmap”) • Report of projects in construction and various stages of development • Includes detailed discussion of the development plans and criteria for projects in advanced stage of pre-construction development • Discussion of the scientific objectives and opportunities that provide the context and compel the need for their development • Will provide discussion of overarching considerations used for cross-disciplinary prioritization (first and second NAS ranking criteria) • Facility Plan will be a public document, updated annually by NSF Director • Facility Plan should become a strategic tool for communicating with research communities and government policy makers • First public draft of plan expected in April or May, 2005 following March NSB meeting • Expect to post for public comment AAAC Meeting - Feb. 16, 2005

7. Process of Large Facility Project Development • Revises existing process defined in NSF’s Facilities Management and Oversight Guide: http://www.nsf.gov/pubs/2003/nsf03049/nsf03049.pdf • Defines a new stage in the pre-construction development process: • “Readiness Stage”: • Has well defined entrance and exit gates • Entrance requires formally defined and publicly distributed development plan • Readiness Stage Development plan included in NSF’s Facility Plan AAAC Meeting - Feb. 16, 2005

8. Project Life-cycle stages • Horizon Stage • Blue sky thinking • Concept Stage • First NSF support for workshops, studies, small scale research • Development Stage • More focused development • Initial facility proposal • Formulation – with NSF – of project development plan for readiness stage • Readiness Stage • Activities that bring project to a “construction ready” status with cost projections that have a high degree of confidence • Begins 1-2 years prior to expected for NSB approval for inclusion in budget request • New start pool of NSB approved projects proposed for funding • Construction • Operations • Renewal, re-competition, termination, etc. AAAC Meeting - Feb. 16, 2005

9. Project Development Plans • Written by project developers in consultation with NSF • Lays out the development trajectory for a project in the Readiness Stage • Technical, managerial activities needed to bring a project to construction readiness • Planning activities that result in mature construction budget estimates and sound projections of expected operations costs • Development budget requirements • NSF adds oversight decision points during Readiness Stage • Decision points (criteria for advance and “off-ramps”): • Developed by NSF • Approved by NSB • Made public – included in Facilities Plan AAAC Meeting - Feb. 16, 2005

10. Readiness stage gatekeepers • Entry: • Community expresses strong endorsement • Advisory committee endorses first and second level criteria • Division and Directorate endorsement • MREFC Panel (NSF Deputy Director, Assistant Directors, OPP, CFO, LFP Deputy – ex officio) • NSB approval • Exit: • Division and Directorate review and endorsement of project, including M&O budget • CFO and LFP Deputy assertion that budget and project plans are “construction ready” • MREFC panel assertion that project meets science objectives and maintains priority • NSF Director concurrence with MREFC panel • NSB approval AAAC Meeting - Feb. 16, 2005

11. Exiting Readiness Stage • Projects exit Readiness Stage because: • Board approved • Fail to advance against project development plan criteria • Eclipse by other projects • Reprioritization • Other factors, as deemed appropriate by NSF Director • Successful exiting projects are in “Candidate for New Start” pool • NSB may re-prioritize pool when candidates are added • Reprioritization rationale will be made public • NSF Director proposes subset of new start pool to OMB for construction funding • OMB approves project inclusion in NSF budget request • Congress appropriates funds AAAC Meeting - Feb. 16, 2005

12. Next steps • Revise NSF’s Facilities Management and Oversight Guide to incorporate these new policies • Challenges: • Describe specific implementation steps that incorporate these principles • Define steps to develop “sufficiently mature” budget projections prior to construction approval • What can be done to minimize risk of potential backlog of approved new starts? • Can this new framework can be used to aid fields (like Astronomy) that have needs for very long lead items as part of facility development? • Advisory Committee input welcome AAAC Meeting - Feb. 16, 2005

13. Defining “construction ready” • Creation of a robust final project “baseline”: • Enabling technical developments are complete • There exist thorough cost and contingency estimates with high degree of certainty that project can be completed within scope • Project management is in place • Project Management Control System (PMCS) in place. • Expensive: 5-15% of total construction budget, but must be done eventually • PMCS alone is typically 1-2% of total project cost • Perhaps overly ambitious definition of “construction ready” AAAC Meeting - Feb. 16, 2005

14. Baseline Development • Need to account for practical considerations, such as difficulty of fully staffing key positions prior to construction approval. • Need to preserve goal of defining scope and budget with “acceptable levels of uncertainty” prior to request for construction funds. • Need to define “acceptable”. For example: • High Energy Physics projects typically allocate 25-45% contingency (45% LHC detectors, 25%: IceCube, LIGO) • Projects with large software component can be higher • Projects with large “build to print” component much less. AAAC Meeting - Feb. 16, 2005

15. Can the framework of NSB/NSF’s new process encompass the natural steps typical of project evolution? • Planning large projects naturally evolves through several stages that are typically well understood: • Conceptual design • Preliminary design • Final design • Construction • Commissioning • Operation • Match NSF’s internal policies to these general notions in M&O Guide • The Guide should also define how solicitations for these proposals may be made, if required AAAC Meeting - Feb. 16, 2005

16. Common baseline evolution • Conceptual design • Functional requirements, definition of systems and functional areas. • Large cost uncertainty. • About 1/3 of total design effort. • Preliminary design • Site-specific design. Principal components, types of equipment, members, sizes defined. • Cost uncertainty much reduced, roughly 25% larger than final design cost uncertainty. • About 1/3 of total design effort. • Final design • Interconnections of components defined. Component vendors defined. Mounting and installation defined. • About 1/3 of total design effort. AAAC Meeting - Feb. 16, 2005

17. Coordination with initiatives in other agencies • Want to make sure that Facilities Plan and the NSF’s Management and Oversight Guide broadly mesh NSF’s strategic planning with activities in other agencies – DOE, NASA • Incorporate project-specific factors, criteria, timing of inter-agency coordination within development plans – will be published as components of Facilities Plan • Natural place to articulate NSF’s intentions for specific international partnerships AAAC Meeting - Feb. 16, 2005

18. Next steps in revision of Facilities Management and Oversight Guide • Identify specific implementation steps that support principles promulgated in NSB/NSF report • NSB review • Expect a period for public reaction and comment • Useful to have advisory committee input on these issues and other problems that exist in current Facilities Guide AAAC Meeting - Feb. 16, 2005

19. Backup materials

20. NAS Study Findings • “There is a lack of funding for disciplines to conduct idea-generating and project-ranking activities and, once ideas have some level of approval, a lack of funding for conceptual development, planning, engineering, and design—information needed when judging whether a project is ready for funding in light of its ranking and for preparing a project for funding if it is selected”. AAAC Meeting - Feb. 16, 2005

21. More planning resources will: • Reduce uncertainties in construction and operations budgets • Reduce uncertainty in construction schedule • Reduce likelihood of de-scoping • Thorough pre-construction risk assessment will create a more robust Project Execution Plan • Create an appropriate framework for cooperation for activities involving international and inter-agency partnerships as an integral part of in-depth planning. • Make project oversight during construction more straightforward, add more definition to the proposed workscope and decision points. • Allow more in-depth consideration of transition to operations and greater certainty in predicted O&M costs. • Avoid a funding hiatus. Overlap some development activities beyond NSB approval and inclusion in annual NSF budget submission. AAAC Meeting - Feb. 16, 2005

22. NSF/NSB Report: • “The National Academies’ Report properly calls attention to the necessity for considerable pre-approval funding for planning and development when it questions whether there is sufficient NSF support for this "bottom up" process. NSF endorses the Report’s recommendations to provide researchers access to funding sufficient to develop compelling research agendas, to refine and prioritize their facility requirements, and to complete research and development on facility designs and needed technologies. The level and form of funding for planning and development will be reviewed, and an evaluation will be made of how project funds are best invested to attain robust plans and schedules with better cost projections, so that only well-defined and thoroughly-costed projects are brought forward for consideration by the Board.” AAAC Meeting - Feb. 16, 2005

23. More NSF/NSB Report: • “The Director and the Board recognize the need to strengthen oversight of the implementation of large facility projects, which will require increased investments of NSF staff time and travel funds. The Report emphasizes the importance of initial planning and definition of technical scope, budget, and schedule, followed by periodic post-award status reviews held on-site by external experts, with implementation of a transparent process for management of changes to a project’s implementation plan.” AAAC Meeting - Feb. 16, 2005