SERD Program Assessment and Impact Reporting

1. SERD Program Assessment and Impact Reporting Saleia Afele-Faamuli, Ph.D., M.P.H. National Program Leader Multicultural Alliances, Science Education Resources Development Unit (SERD) 2008 NACTA SERD PD CONFERENCE June 10-13, 2008

2. Why Assessments? • On-going monitoring/checking/managing/measuring performance is a must to: verify goals if they are met. What worked? What didn’t, and why not? Lessons learned. • Increase program effectiveness • Leadership and management for success • Communicate program values/stories

3. Why Assessments?Why Impact Statements? • When all is said and done we need to be able to do all we can to answer this question: “So what?” • Your work can inspire other researchers • You did it! Give credit where it is due. • It’s required—Terms and Conditions

4. Congress Taxpayers Scientists Program evaluators Stakeholders Your NPL Students Your partners Your competition Potential collaborators You Who reads impact statements?

5. Impacts should be: • Brief—Three to five sentences • Results oriented • Able to address a national need • Proofread This is one time when it’s a good idea to brag! In fact, we are counting on you to do so!

6. Example of “Quality” Outcomes • Researchers cloned a gene, GPC-B1, from wild wheat that increases the protein, zinc and iron content in the grain. This trial offers a potential solution to nutritional deficiencies affecting hundreds of millions of children in U.S. and around the world. The researchers found all commercial pasta and bread wheat varieties analyzed so far have a non-functional copy of the GPC gene, suggesting the gene was lost during the domestication of wheat. Re-introducing the functional gene into commercial wheat varieties could increase their nutritional value.

7. Please Don’t • Assume summary = impact • Serve alphabet soup—Acronyms Anonymous • Assume readers know a lot about science—or your particular research field • Say that you have no impacts • Just provide tallies ---Students taking the class ---Meetings held ---Experiments conducted

8. Please Do • Say why this is important. How will your work address a national need? What U.S. statistics back your assertion—CDC, USDA or EPA? • Use numbers to illustrate change: percentages, money saved, time it took for transitions. Remember: report change and growth, not tallies • Use pre/post evaluations

9. Objective versus Subjective • Objective --Students scored 80 percent over pre-test --Farmers made $25,000 with new method --Mites reduced by 25 percent • Subjective --Students received the lesson well --Farmers benefitted and adapted to method --Study will revolutionize bee keeping industry

10. Review RFA • Start with Your Funding Proposal—RFA A) Potential for advancing quality of education Impact—Continuation—Innovation—Products--results B) Proposed Approach and Cooperation C) Institutional capacity building D) Key Personnel E) Cost Effectiveness Criterion in your proposal can help you put your thoughts together--and write better impact statements.

11. CRITERIA A) Potential for advancing quality of education Impact—Continuation—Innovation—Products—results B) Proposed Approach and Cooperation C) Institutional capacity building D) Key Personnel E) Cost Effectiveness ASK YOURSELF Did project meet goals? Were collaborators an asset? Did the approach work? How did the school benefit? Who did what? Was Contribution of value? Did you meet budget? Over/under? Why? Review Your RFA

12. Both RFAs and Impacts • Benefit from action words: reduce, improve, decrease, enhance • Need clearly stated goals • Must address pitfalls and failures • Include plans for continuity of the project

13. Other Tools You Can Use • Pre/Post Tests • Surveys • Program notes • Records • Logic Model

14. So What’s a Logic Model? And how do I use it?

15. Situation Inputs Activities Outputs Outcomes Knowledge Actions Conditions People made changes People learned Tallies go here People are better off The money you spent—The time you provided What is the need? Why do this? What you did EXTERNAL FACTORS – what happens outside your project ASSUMPTIONS– What are the “givens” of your project? The Logic Model

16. What is the need? Why do this? The Logic Model Situation Inputs Activities Outputs Outcomes Knowledge Actions Conditions Assess the needs—but be careful! Are you describing symptoms or problems? Are stakeholders engaged? Consider: Mission, Vision, Values, Mandates, Resources, Local Dynamics, Collaborators and Competitors. Intended outcomes? Check your RFA!

17. Situation Inputs Activities Outputs Outcomes Knowledge Actions Conditions The money you spent—The time you provided The Logic Model Staff Time Volunteer Time Money Research Base Materials Equipment Technology Partners Check your proposal budget! What changed?

18. Situation Inputs Activities Outputs Outcomes Knowledge Actions Conditions What you did The Logic Model Conduct workshops Deliver Services Develop Products Train people Counsel people Facilitate Partner Make referrals Build facilities Your Daily records should help

19. Situation Inputs Activities Outputs Outcomes Knowledge Actions Conditions Tallies go here The Logic Model What do you have? Training CDs Websites Lesson Plans New MA/PhD program New Research Garden New Laboratory Who did you reach? Students Agencies Faculty Customers

20. Situation Inputs Activities Outputs Outcomes Knowledge Actions Conditions People learned People made changes People are better off The Logic Model--IMPACT Knowledge/People learned: Knowledge gained, skills learned, motivations, opinions changed, new student aspirations? Actions/People changed: Actions, behaviors, decisions making skills, policies or social actions Conditions/People are better off:Crops/livestock more productive, pollution reduced, youth diet improved, students learned more, student attained higher degree, farmers saved money, farmers made money THINK: Could this project have national or regional benefits if expanded?

21. Situation Inputs Activities Outputs Outcomes Knowledge Actions Conditions EXTERNAL FACTORS – what was happening outside your project? How did these factors affect your success, and how did they pose challenges? ASSUMPTIONS–What are the “givens” of your project? Consider their effect on situations through outputs… The Logic Model Final thoughts add to your analysis….

22. Let’s do an analysis and impact statement! The NACTA/SERD PD Conference

23. SituationWhy have this conference? • Agricultural teachers need to share teaching methods, problems, philosophies and styles • Agricultural teachers need to network with a broad base of colleagues nationwide • Outstanding agricultural teachers need to be recognized for their contributions Source: NACTA/SERD 2008 Conference website

24. Inputs: Things Put Into The Conference • Utah State University hosted conference • CSREES Staff prepared training • Utah State staff also prepared training • Utah State staff arranged family days/entertainment • Utah State staff arranged with hotels for lodging and logistics • Utah State Staff took care of registration Source: NACTA/SERD 2008 Conference website

25. Actions: What Utah State University Did • Recruited participants • Recruited speakers • Recruited poster presenters—Helped with setup • Conducted Training • Provided recreation for participants/family Source: NACTA/SERD 2008 Conference website

26. Outputs: What Do We Have? • Take home materials—booklet of abstracts • Contacts made for all participants/networking • Training for 250 people • Recognition of outstanding teachers/educators • New Programs, Lesson Plans, Websites Source: NACTA/SERD 2008 Conference website

27. Result 250 attendees Over 130 papers presented Teacher of the year honored Benefit to community Wide number of networking opportunities Gave members great ideas for research and teaching Inspire other teachers; provide example of effective teaching IMPACTS or OUTCOMES: What were the results?

28. Discussion Result • 250 attendees • Over 130 papers presented • Teacher of the year honored Benefit to community • Could a survey prove this is a good networking opportunity? How? • How would you demonstrate the value of just poster or just oral sessions or both? • What U.S. statistics could show the value of honoring teachers? Where would you look?

29. EXTERNAL FACTORS – What was happening outside your project? How did these factors affect your success, and how did they pose challenges? ASSUMPTIONS–What are the “givens” of your project? Consider their effect on situations through outputs… What are the assumptions of this conference? What external factors affect this conference? Final thoughts…

30. Why Assessments? Why Impacts? • When all is said and done, we need to be able to DO ALL WE CAN TO ANSWER THESE QUESTIONS: “SO WHAT?” “WHAT DIFFERENCE WOULD THIS PROJECT MAKE LOCALLY, NATIONALLY, & WORLD-WIDE?” “WOULD THE PROJECT RESULTS OR IMPACTS CONVINCE THE DECISION-MAKERS, FUNDERS, AND STAKEHOLDERS FOR CONTINUED AND ENHANCED SUPPORT?”

31. Thank You Questions?