Evaluating Research Efficiency in EPA: Identifying Best Practices and Implementing NAS Recommendations

1. Evaluating Research Efficiency in EPA Phillip Juengst, Accountability Team Leader

2. Efficiency Measures, the PART, and Research • Where we were: EPA struggled to identify acceptable and meaningful efficiency measures for the PART • Ultimate outcomes and cost effectiveness ($ per outcome) not easily measured for research • Efficiency is important, but research focus should be: • Are we doing the right science? • Are we doing it well? • Are we making an impact?

3. Identifying Best Practices EPA identified significant variation across agencies *Bold indicates similar to EPA-proposed measure(s) not accepted by OMB

4. EPA Research Efficiency Measures • Measures vary by program, making it difficult to assess overall efficiency of research in a consistent and meaningful way • Average time (in days) to process research grant proposals • Average time (in days) to respond to requests for technical support • Number of peer reviewed publications per FTE • Average cost to produce Integrated Science Assessments • Percent variance from planned cost and schedule

5. Steps Toward Performance Improvement • In 2006, EPA began collaborating with OMB and our Board of Scientific Counselors on measures of outcomes and efficiency • Short-term (2007) results: • PART scores up to “Moderately Effective” • EPA green on PMA (research efficiency measures in place) • Broader vision: EPA engaged the National Academy of Sciences, OMB, other agencies, and private sector in a study of research efficiency

6. New Directions for Managing Research Performance • 2008 NAS study provides broad recommendations for all research agencies • Agencies should use expert panels to evaluate “investment efficiency,” beginning with an evaluation of the relevance, quality, and performance of research • To improve program management, agencies should use quantitative “process efficiency” measures based on inputs, outputs, and some intermediate outcomes • OMB should have more consistent and equitable implementation of PART requirements

7. Implementing the NAS Recommendations • Agency dialog about how best to implement NAS recommendations • EPA/OMB led government-wide panel discussion on April 10th to discuss how best to implement NAS recommendations • Approx. 100 representatives attended from more than 25 departments/ agencies, including OMB (Robert Shea and examiners from most branches), NAS, NIH, NSF, DOE, NASA, NIOSH, etc. • Summary and video being posted on EPA’s website, and shared with the research and evaluation communities http://www.epa.gov/ord

8. Next Steps for EPA Developing the right efficiency measures for research Emerging interagency consensus that agencies should be able to “experiment” without risk to PART scores/PMA EPA implementing NAS recommendations: Expanding use of BOSC for assessing “investment efficiency” How efficiently has the program invested and managed resources to achieve its long-term goals? Exploring “% of budget allocated to direct science activities” Endorsed by NAS when balanced by other measures Drives administrative efficiencies at the lab level Linked to long-term goals/EPA mission and balanced by existing quality/outcome measures 7

9. Next Steps for EPA and other Agencies Using research efficiency measures to manage EPA undergoing major reorganization to achieve administrative efficiencies Long-range goal: efficiency improvements help maintain, if not increase, investments in direct science and research outcomes—better EPA decisions that improve human health and the environment EPA will continue to lead and participate in Federal dialog to develop better measures that drive performance improvement Follow-up sessions (WREN?) to discuss implementation of NAS recommendations OSTP’s Science of Science Policy Roadmap 8