Office of Planning and Accountability Henry M. Doan, Ph.D. November 17, 2009

1. National Institute of Food and Agriculture PROGRAM EVALUATION Office of Planning and Accountability Henry M. Doan, Ph.D. November 17, 2009 INVESTING IN SCIENCE SECURING OUR FUTURE

2. National Institute of Food and Agriculture Program Evaluation “I want to show my funding agency, the Congress and OMB that my project works.” “I want to provide evidence that my project is effective.” ” I want to show that my project is beneficial to the end users.“ “What is working, what is not working for my program/project?” “I want to identify program/project strengths as well as weaknesses.” INVESTING IN SCIENCE SECURING OUR FUTURE

3. National Institute of Food and Agriculture Program Evaluation How complicated is evaluation?” “Would evaluation increase my burden?” “Where do I find resources for evaluating my program/project? “Would evaluation take away resources that are meant for participants/end users?” “Evaluation may produce negative results that would make my program look bad!” INVESTING IN SCIENCE SECURING OUR FUTURE

4. National Institute of Food and Agriculture A Definition of Program Evaluation Program Evaluation is “the systematic collection of information about the activities, characteristics of outcomes of programs to make judgments about the program, improve program effectiveness and/or inform decision about future program development.” Center for Disease Control & Prevention (CDCP) INVESTING IN SCIENCE SECURING OUR FUTURE

5. National Institute of Food and Agriculture GAO Definition Performance Measurement – ongoing onitoring/ reporting of program accomplishments; conducted by program management; answering the “whats.” ---Outcome measurement/monitoring--- Evaluation – studies conducted to assess how well a program works on a periodic or ad hoc basis; answering the “whys” and “hows.” INVESTING IN SCIENCE SECURING OUR FUTURE

6. National Institute of Food and Agriculture Why should we conduct evaluations? Provide feedback for program improvement and external accountability Answer evaluation questions about results and the processes that managers directly control to achieve results Document program effectiveness and value added to society INVESTING IN SCIENCE SECURING OUR FUTURE

7. National Institute of Food and Agriculture Evaluation Process Phase 1 – Getting ready Develop description of program/project; capacity to make a difference Develop a conceptual model, i.e. a program/project logic model Develop evaluation questions and expected results Develop an evaluation design Phase 2 – Conducting the evaluation Collect data Analyze data Report findings INVESTING IN SCIENCE SECURING OUR FUTURE

8. National Institute of Food and Agriculture INVESTING IN SCIENCE SECURING OUR FUTURE

9. National Institute of Food and Agriculture Program Evaluation Conceptual/Logic Model Inputs Activities Outputs Outcomes Resources Funding Staff Volunteers Equipment etc. Constraints: Laws Regs Other require- ments Services Surveying Intervewing Record- reviewing Training Teaching, etc. Products Participants Training sessions conducted Reports generated Publications, etc. Benefits 1.Knowledge acquired 2.Attitudes/values changed Behavior modified about the science, etc. 3.Condition improved INVESTING IN SCIENCE SECURING OUR FUTURE

10. National Institute of Food and Agriculture Types of evaluation questions as they fit into the conceptual/logic model Activities Outputs Inputs Outcomes WHY HOW INVESTING IN SCIENCE SECURING OUR FUTURE

11. National Institute of Food and Agriculture Examples of Evaluation Questions Is the program being delivered as intended? Is the program well managed? Are desired program outcomes obtained? Why is a program no longer obtaining desired outcomes? What is the cost-benefit of the program? INVESTING IN SCIENCE SECURING OUR FUTURE

12. National Institute of Food and Agriculture Examples of Evaluation Design Pre and Post-Test Experimental Design Experimental Group Control Group INVESTING IN SCIENCE SECURING OUR FUTURE

13. National Institute of Food and Agriculture Defining Measurable Results OUTPUTS vs. OUTCOMES Number of training sessions conducted Number of papers presented at professional meetings Number of publications Understanding/knowledge about how outdoor recreation benefit human health Participants applying knowledge about outdoor recreation benefits to improve their productivity/health Understanding and acceptance of new teaching methods Users applying new teaching methods in their work to increase effectiveness INVESTING IN SCIENCE SECURING OUR FUTURE

14. National Institute of Food and Agriculture Performance Measures vs. Indicators How you want your programs to be assessed? What are your programs’ ultimate goals (impact)? What tangibles can you show for program achievements? How do you show programs’ progress toward achieving program goals, having impacts on your program participants? INVESTING IN SCIENCE SECURING OUR FUTURE

15. National Institute of Food and Agriculture Examples Goals : Graduates enhanced professional skills to get jobs in food and agricultural sciences. Performance measures: Increasing the number of professionals in food and agricultural sciences. Performance indicators: degree and/or certificates students received. INVESTING IN SCIENCE SECURING OUR FUTURE

16. National Institute of Food and Agriculture Questions? Comments? INVESTING IN SCIENCE SECURING OUR FUTURE

17. National Institute of Food and Agriculture THANK YOU! Henry M. Doan, Ph.D. 202-401-0791 hdoan@nifa.usda.gov INVESTING IN SCIENCE SECURING OUR FUTURE