Review of Scientific User Facilities Division

1. Review of Scientific User Facilities Division Committee of Visitors Presented by W. A. Barletta, Chair

2. Top level charge • Assess processes used during FY2010, 2011, & 2012 • to solicit, review, recommend, & document proposal actions • to monitor active projects and programs. • Program elements to assess • Light Sources including the Accelerator & Detector Research Program • Neutron Sources • NanoscaleScience Research Centers & Electron Beam Micro-Characterization Centers • Construction Projects & MIEs • Report to the Summer 2013 BESAC meeting

3. Charge specifics • (1) For scientific user facilities including the accelerator & detector program, • Assess the efficacy • Assess the quality of processes used to: • (a) Solicit, review, recommend, & document proposal actions (b) Monitor active projects, programs & facilities • (2) Within boundaries set by DOE missions & funding, • Comment on how the award process has affected: • (a) Breadth & depth of portfolio elements • (b) National & international standing of portfolio elements

4. BES SUFD Committee of VisitorsDr. William Barletta (MIT/USPAS), Chair • Construction Projects • James Krupnick (LBNL, Ret.) • Angus Bampton (PNNL) • Jeff Hoy (Trident Service LLC) • Maria Dikeakos (DOE PPPL) • Nano-Science & E-Beams • Prof. Donald Tennant (Cornell) • Prof. Beatriz RoldanCuenya (UCF) • Dr. Ernie Hall (GE) • Dr. James Liddle (NIST) • Light Sources, Accelerator & Detector Research • Dr. Simon Bare (UOP) • Prof. Nora Berrah (WMU) • Dr. Gene Ice (ORNL) • Dr. Joel Ullom (NIST) • Dr. David Robin (LBNL) • Neutron Facilities • Prof. Sunil Sinha(UCSD) • Dr. Robert Dimeo (NIST) • Prof. Thomas Russell (UMass) • Dr. John Tranquada (BNL) Chairman in Red BESAC Member in Italics

5. Overarching conclusions • We commend SUFD for effective use of its available funding to construct & operate a system of facilities which deliver world-leading science • The efficacy of SUFD’s processes to review, recommend & document proposal actions are excellent • The efficacy of the processes to monitor & review active projects, programs & facilities are also excellent • SUFD staff members are to be commended for rigorous & effective program management Comment: BES growth over the decade is impressive

6. Overarching conclusions • Within the scope of DOE missions & available funding, SUFD’s award processes continue to enhance • the breadth & depth of portfolio elements as well as • their national and international standing • International competition in scientific user facilities is fierce; maintaining scientific leadership will require increased investments • for the user facilities • for user support

7. SUFD-wide findings & recommendations - 1 • A. Implementation of previous COV recommendations • FINDING: SUFD has effectively addressed the majority of the recommendations made in the previous COV report • FINDING: SUFD travel budget continues to be incommensurate with most effective oversight of its program • RECOMMENDATION: Enhance the effectiveness of program oversight by increasing the flexibility of SUFD managers • to interact with facility managers • to communicate with the facilities staff via increased on-site presence

8. SUFD-wide findings & recommendations - 2 • B. Assessment of COV process effectiveness • FINDING: Providing all data on computers to each reviewer is highly responsive to previous recommendations & greatly improves the COV process. • The electronic documentation was thorough and well organized • RECOMMENDATION: The move toward a full, searchable database is commendable. It should be available to the next COV

9. SUFD-wide findings & recommendations - 3 • C. Facility review process description & effectiveness • FINDING: The 3-year reviews of the facilities are well organized and well executed with appropriate review teams • FINDING: A uniform definition of high impact publications has been established for light sources & neutron sources. • RECOMMENDATION: Finalize the set of uniform definitions of high impact publications for the nanoscience centers

10. SUFD-wide findings & recommendations - 4 • D. General Issues • FINDING: The quality of the scientists at the facilities is the critical asset that ensures excellence & success. • RECOMMENDATION: Place added emphasis on career development as well as on maintaining state-of-the-art experimental apparatus, sample environment & software at all facilities to maximize scientific productivity • FINDING: Different types of facilities serve different scientific communities. They are all needed & important. • RECOMMENDATION: New metrics that account for scientific impact should apply to all the types of scientific user facilities

11. SUFD-wide findings & recommendations - 5 • Recommendations: If an increased fraction of industrial users is desired, this fact should be clearly communicated to the management of facilities • Proposal evaluation should include criteria that value factors such as economic & technological impact • Thorough review of user agreements should be undertaken with a view to removing barriers to industry users

12. Light sources and Accelerator & Detector Research

13. Light source findings - 1 • The review process is comprehensive, balanced, fair, and transparent to all facilities • They are all reviewed using the same major criteria. • The number and breadth of the reviewers for each facility was appropriate • Reviews provided important comments & specific actionable recommendations where appropriate. • The reviews were uniformly forthright, detailed, & contained a summary section at the beginning of the review • The detail in review comments indicated a deep appreciation of important issues that went beyond the documentation solicited. • This indicates the importance of the on-site review.

14. Light source findings - 2 • Review follow-up was not documented between reviews • RECOMMENDATION: A formal yearly follow up of facility recommendations should be documented each year • Ideally this could be a short response saying all issues had been previously addressed when appropriate • Recommendations from the prior reviews were not sent to the facility reviewers • RECOMMENDATION: Make previous review recommendations & facility response available to the facility reviewers at the outset • to allow the review committee to assess how the facility has responded to the prior recommendations.

15. Accelerator & Detector Research - Findings • Resources available are so limited ~$10M (~1% of total SUFD) that the breadth is necessarily small (focusing primarily on FELs) • This figure is too small to maintain scientific leadership • Projects are generally excellent & matched to SUFD needs • At present the portfolio does not include x-ray optics

16. Accelerator & Detector Research - Recommendations • Increase the ADR portfolio to $20M to $30M per year (2-3% of SUFD budget) • Consider the concept of a HUB to advance ADO technology in support of its scientific mission • Add X-ray optics to the ADR portfolio • As part of increasing the portfolio, formalize the proposal solicitations • Continue using workshop reports to guide research initiatives & to shape investment priorities • Develop topic-specific metrics to assess / characterize US capabilities in accelerators, detectors & optics

17. Neutron sources

18. Neutron sources • Finding : Few instruments in ORNL’s Neutron Sciences Directorate operate in the steward-partner model. • RECOMMENDATION: Strongly encourage neutron scattering facilities to explore forming partnerships on instruments with potential partners from other agencies in the cooperative stewardship model • Exploit neutron scattering capabilities to benefit the broadest possible scientific community • Finding: It is unreasonable to frontload facilities with understaffed instrumentation or beamlines   • Recommendation: When an MIE is being considered, a the facility should have well-designed plan to ensure its robust, long-term operation for users

19. Neutron sources & neutron science • Finding: Trends in the science enabled by neutrons have evolved significantly since last studies (1993, 2002) on the future of neutron science in the US • Recommendation: We recommend that BES join with other agencies, such as DOC, NSF, & NIH, to assess current status & future directions for U.S. neutron science • Include factors, such as neutron measurement capacity & capabilities and present & future needs of the U.S. scientific community

20. Nanoscience Research Centers & E-beam Microcharacterization Centers

21. NSRCs & EBCs - General Observations Finding: NSRCs are entering a post-ramp up phase in which they face issues of high demand for facilities & instruments& over subscribed staff Recommendation: Provide guidance to NSRCs to plan for expansion of facilities or expanded operating hours Alternatively adopt higher rejection rates of user proposals

22. NSRCs & EBCs - General Observations Finding: SUFD program managers have done a commendable job at establishing thorough & transparent processes for evaluating ongoing projects The 3-year review cycle might be excessive for timely corrective actions Recommendation: Increase face-to-face to time between DOE officials & management, scientific staff, & user community of the NSRCs &EBCs, including regular (yearly) on-site visits Methods of assessing NSRCs have not evolved since the centers were first commissioned Recommendation: NSRCs (& EBCs) are sufficiently different from light sources to warrant tailored metrics & assessment methods (user surveys)

23. NSRCs & EBCs - General Observations • BES recommended merger of EBMCs & NSRCs in response to atriennial review finding that EBMCs were understaffed & underfunded to carry out their mission. • This recommendation focuses on improvements in synergy & operational efficiency. • Recommendation: SUFD should ensure that facility merger plans focus clearly on these improvements

24. High Utilization Instruments Finding: The unique, world-leading instruments associated with the EBMCs are in high demand but are not utilized optimally Staff funding is currently for 40 hour week, yet the labs are open more than 8 hours per day. Recommendation: Merger plans should include expanded staffing (>8 hr/day) on select tools

25. Industry Participation NSRCs have not attracted a significant number of industry users Recommendations: If an increase in the fraction of industrial users is desired, this fact should be clearly communicated to the NSRCs Proposal evaluation should include criteria that value potential technological or economic impact Thorough review of user agreements should be undertaken with a view to removing legal barriers to industry users

26. CONSTRUCTION PROJECTS

27. Findings regarding project management • Comment: PM processes are robust & successful in delivering projects on schedule, within budget & meeting baseline technical performance parameters • SUFD met, and often exceeded, BES goal of remaining within + / - 10% for cost/schedule performance • Project management performance metrics have become even more visible indicators of DOE performance for external stakeholders • Recommendation: Ensure that CD4 requirements are reasonable, broadly understood by all stakeholders, & fully achievable within the project budget • Effort should be made to manage & align expectations for what constitutes successful initial scientific operations

28. Impact of travel reduction • Comment: SUFD management should determine the correct level of field presence for Program Managers to provide adequate Federal oversight, operational awareness, & fosters strong and open communication between field and HQ elements • On-site presence, graded to project risk, should be appropriately balanced with use of remote communication tools to live within constrained budgets • Recommendation: Mitigate negative impacts of reduced travel funds • Balance field presence with communication technology to maintain robust communication between program managers & the on-site members of IPTs

29. The COV thanks Jim Murphy and the entire SUFD stafffor their candor & responsiveness Special thanks to Linda Cerrone for her help in making this review easier and more pleasant