Can Feds KISS?

1. Can Feds KISS? Creating a Simple and Smart Strategic Execution and Evaluation “System” Despite Complex Mandates and Requirements Environmental Evaluator’s Network Concurrent Sessions June 23, 2011, 4-5 PM Room 310 Liz Davenport and Thanh Vo Dinh National Oceanic and Atmospheric Administration

2. Network to Learn How Evaluation Can Aid an End-to-End Management Process • How a SMART planning, budgeting, execution, and evaluation system can better manage a complex portfolio • How better integration of performance monitoring, program evaluation, and assessing strategic progress can improve performance management • How Feds can KISS to ensure GPRA Modernization Act improves agency and government-wide performance

3. Context for NOAA’s Strategy Execution and Evaluation (SEE) Process • NOAA manages about 12,800 employees, aided by about 4,000 contractors • NOAA has 6 Line Offices and 12 Staff Offices • There are 127 mandates and authorities applicable to NOAA’s mission • NOAA’s responsibilities span all 50 states, Guam, Puerto Rico, and the Pacific Island Region

4. NOAA Strategic Plan

5. What Is NOAA Creating in the Strategy Execution and Evaluation Process? • Here is SEE…under its first cycle of implementation...

6. Strategy Execution & Evaluation Framework Evaluate Progress to Plan, Performance, Programs, and Strategic Priorities 6 Evaluate Progress 1 Focus Annual Guidance Memorandum Execute Operating Plans to Achieve Priority Outcomes Focus Attention to Achieve Priority Outcomes Align Strategic Plan Implementation Plans 2 Assess Corporate Portfolio 5 Execute Annual Operating Plans 4 Reassess Corporate Portfolio 3 Request Budget Reassess Portfolio to Optimize Performance in Operational Plans Assess Portfolio to Plan Investments Request Budget to Achieve Intended Investment Outcomes

7. From Compliance to Assimilation…Growing Utility of Evaluation Managers consistently use program evaluation requirements to define long-term outcomes, improve short-term success and achieve objectives Agency success is strongly aided by assimilated program evaluation and a performance management framework that makes optimal use of performance data for management

8. Program Evaluation Requirement Driver GPRA Modernization Act Authorizes Implementation to: • Foster coordinated, cross-cutting, government-wide planning • Government-wide strategic goals with annual government-wide (Federal) performance plan • Engage Congress to identify management/performance concerns • Agency strategic plans include Federal performance goals and timed for release with changes in Administration • Improve accountability for results and address management weaknesses • Agency and separate Federal performance plans aligned; looks at past 5 years of performance data • Performance information better utilized due to greater Administration and Congressional engagement • Quarterly reviews reported at www.performance.gov

9. Performance Management and Evaluation “Unit” and “Paradigm” • Adopt Standard, Federal Terms and Definitions • Ensure Agency Clarity Regarding: • Program • Performance Measurement • Performance Measures • Program Evaluation • Comply with New Standard: • Government-wide/cross-agency performance plan, with agreed upon outcomes and accountability within a 2-year time frame

10. Stop and Think! • How can we define “program” for agency performance planning and reporting and for Federal performance planning and reporting, given the need to link to and show progress in achieving Federal goals? • How can we do evaluations and use them to provide feedback to our Agency for these new requirements?

11. Program Evaluation Typology • GAO Defined Types of Federal Program Evaluations Process (or Implementation) Evaluation – assesses extent to which a program is operating as intended…assesses program activities’ conformance to statutory and regulatory requirements, program design, and professional standards or customer expectations Outcome Evaluation – assesses extent to which a program achieves its outcome-oriented objectives…focuses on outputs and outcomes (including unintended effects) to judge program effectiveness but may also assess program process to understand how outcomes are produced Impact Evaluation – (type of outcome evaluation) assesses net effect of a program by comparing program outcomes with an estimate of what would have happened in absence of the program…employed when external factors are known to influence program’s outcomes…to isolate program’s contribution to achievement of objectives Cost-Benefit and Cost-Effectiveness Analyses – compare program’s outputs or outcomes with costs (resources expended) to produce them…when applied to existing programs, considered a program evaluation…cost-effectiveness analysis assesses the cost of meeting a single goal or objective and can be used to identify the least costly alternative for meeting that goal…aims to identify all relevant costs and benefits, usually expressed in dollar terms Performance Measurement and Evaluation,Definitions and Relationships, GAO 11-616SP

12. Stop and Think! • Can these program evaluations and performance measurements serve the new requirements at the same time serving Agency needs for management to meet strategic priorities? (National Ocean Policy…how will it be evaluated?) • Do we need anything else? Or what could be potential tools, approaches, or new paradigms for evaluation?

13. Current Available Tools? • Theory of changes: Logic model for government-wide goals from collective actions • Performance accountability: www.performance.gov • Conflict resolution and coordination of capabilities between partners • Strategic evaluation in concert with performance measurement and program evaluation

14. Discussion Questions • Thanks for letting us share SEE and speculate about GPRAMA impacts on evaluation • Thoughts, experiences , best practices on… • How can we define program for agency performance planning and reporting and for Federal performance planning and reporting given the need to link to and show progress in achieving Federal goals? • How can we do evaluations and use them to provide feedback to our Agency for these new requirements? • Can these program evaluations and performance measurements serve the new requirements at the same time serving Agency needs for management to meet strategic priorities? (National Ocean Policy…how will it be evaluated?) • Do we need anything else? Or what could be potential tools, approaches, or new paradigms for evaluation?

15. Backup Slides

16. Other Discussion Questions • What are your thoughts, experiences, best practices on… • GPRAMA requirements on your Agency’s evaluation and performance management? • Communicating evaluation results, using evaluation to facilitate, increase relationships/interaction with Congress and stakeholders? • Implementing GPRAMA requirements on evaluation within the current budget environment? • How are you building evaluation capacity? (Linking evaluation with performance management and evaluating success/failure of strategic goals/plans)?

17. Program Evaluation Framework Benefits • Effective compliance with evaluation requirements • Learning culture through evaluation feedback • Robust program management • Better investments through adaptive management • Valuable decision-making support tool • Transparent performance, outcomes, and results • Monitor progress toward achieving strategic goals • SMART performance targets • Improved accountability

18. GPRAMA Agency Strategic Plan Changes That May Impact Evaluations Former Process Revised at least every 3 years Covered at least a 5-year period Consulted with Congress Involved other stakeholders Submitted to OMB and Congress Amended Process Revision every 4 years, approximately 1 year after a new Presidential term begins Consultations with relevant Congressional committees at least every 2 years Publication online and President and Congress notified Amended Content Relationship to Federal government priority goals Interagency coordination and collaboration Identification of agency priority goals Description of incorporation of Congressional input Former Content Mission statement Strategic goals Strategies and resources Relationship to performance goals External factors that could significantly affect the goals Program evaluations

19. GPRAMA Agency Performance Plan Changes That May Impact Evaluations Former Process Upcoming fiscal year No set time frame by OMB Submitted to OMB Amended Process Covers 2 fiscal years (current/upcoming) Concurrent with President’s Budget Available online; President and Congress notified Former Content Performance goals in objective, quantifiable, measurable form Cover all program activities Strategies and resources Performance measures Basis for comparing actual results with performance goals Means to verify/validate data Amended Content Relationship to agency strategic and priority goals, and Federal government performance goals Coordination and collaboration Contributing programs and activities Milestones Goal leaders Balanced set of measures Data accuracy and reliability Major management challenges Low-priority programs

20. GPRAMA Agency Performance Plan Changes That May Impact Evaluations GPRA Modernization Act…Agency Quarterly Priority Progress Reviews • At least quarterly, agency head, COO, and PIC review with goal leaders progress toward each priority goal • Review involves contributors in and out of Agency: • Assess how programs and activities contribute to goal • Categorize goals by risk of not being achieved • For those at risk, identify strategies to improve performance

21. GPRAMA Agency Performance Plan Changes That May Impact Evaluations GPRA Modernization Act…Requires High Performance Goals with Expanded Performance Reporting • Set quarterly progress reviews • Require Congressional consultation and engagement in performance management • Require a Performance Portal • Reduce duplicative performance reporting by 10%

22. GPRAMA Agency Performance Plan Changes That May Impact Evaluations GPRA Modernization Act Senate Committee Suggests: • Agency, OMB, and Relevant Congressional Committee Review of Agency Meeting Goals • If Goals Are Not Met: • After 1st FY, require a Goal Performance Improvement Plan • If not met after 2nd FY, submit actions to improve performance (statutory changes, resource transfer, etc.) • If not met after 3rd consecutive FY, must act within 60 days or face statutory changes or possible termination or reduction possible

23. GPRAMA Agency Performance Plan Changes That May Impact Evaluations Strategic Plan must include how program evaluations were used to establish or revise strategic goals and a schedule for future evaluations Performance reports must include summary findings of program evaluations completed during the fiscal year covered by the report Evaluation will benefit performance planning and goal setting as well as quarterly reviews of priority goals