Simple Survey Resources: Templates, Tabulation & Impact

1. Simple Survey Resources: Templates, Tabulation & Impact Jeff Buckley, Jenna Daniel, & Casey Mull

2. Overview • Situation • Need Being Met • Audience (Agents & Associates) • Basic Overview of the Training • How Can This Be Utilized in Your State?

3. Situation • As a result of shrinking state budgets, the need to effectively document program impact is more critical than ever. • Many states do not have a full time Evaluation Specialist. • County Agents and State Specialists must develop their own evaluation instruments. • Many County Agents and State Specialists may not feel qualified or confident enough to develop their own instruments and/or to tabulate and report data.

4. Need Being Met • A 4-H program evaluation training was created • Implemented by Georgia 4-H State Specialists & CAES Faculty • Piloted at the 2012 Winter School • 4-H Agents & Associates

5. Simple Survey Training • Training Format: • 90 minute session • Interactive • 4-H Agents and Associates • Participants created their own program evaluation

6. Simple Survey Training • Why is it Important to Gather Data? • Who is your audience? • Process Evaluation – Internal Use • Outcome Evaluation – External Audience • How will you share the results? • Impact Statement • News Article • Journal Article • Other

7. Simple Survey Training • What Can a Survey Measure? • Knowledge • Attitude • Behavior • Designing Measurable Objectives • Refer to the handout “Verbs for Writing Measurable Objectives.” • Measurable objectives will translate into more effective constructs and questions.

8. Simple Survey Training • Consider… • What you want your participants to know. • How you want their attitude to change. • What sort of behavior changes you want to see. • Creating A Construct • Pick two outcomes/constructs and develop three “questions” each, OR • Pick three outcomes/constructs and develop two “questions” each. • (The template has space for six “questions”.)

9. Simple Survey Training • Creating A Construct • The food in the cafeteria was hot. • The dining area was clean and comfortable. • There were a variety of fruits and vegetables. • Beverage selections included healthy options. • Vegetables were not over-cooked. • The servers were on task and pleasant.

10. Simple Survey Training • Likert Style Questions • These templates are designed for use with Likert style questions. • Pick the rating scale that works best for your objectives/program. • This is the most challenging part of developing your survey. Allow time. Ask for feedback.

11. Simple Survey Training • Likert Style Questions • Consist of a statement and a rating scale. • Need to conform to one rating scale. • Measurable verbs that fit the appropriate cognitive domain. • One data point per question. • Is what you’re measuring important to the intended audience of the evaluation?

12. Simple Survey Training • Instrument Templates • Traditional • Retrospective Postthenpre • 6 quantitative items • 1 qualitative item • Feedback on the program

15. Simple Survey Training • Tabulation • Excel Spreadsheet • Directions • Worksheet for both instrument templates • “Locked” format

16. Now You’re Ready to Gather Data!

17. Now You’re Ready to Gather Data!

18. Now You’re Ready to Gather Data!

19. Results • Presentations Conducted • Galaxy • Georgia 4-H Winter Conference • Georgia Association of 4-H Agents Conference • Southern Region Volunteer Conference

20. Results • Impact • The percentage of program participants who stated that they could… • develop measurable objectives increased from 60% before to 100% after the program. • design evaluation instruments for their programs increased from 69% before to 91% after. • are likely to develop their own evaluation tools in the future increased from 46% to 89/%. • A domain analysis of the qualitative revealed the following themes… • Participants were very glad to have the templates. • Participants plan to use the tools and knowledge they gained to gather data. • Participants plan to use the data they collect to develop higher quality impact statements.

21. Results • Extension Agent Quotes & Impact Statements • With a mean score of 3.1 (SD=.994), overall respondents from Question 5 indicated they "agreed" they are more likely to get involved in their community as a result of the GPK Leadership Adventure weekend. • Retrospective post then pre: [In Question 2], 84.6% of respondents indicated an increase in their confidence concerning their ability to “design or construct a map of the distribution of major world languages”.

22. Simple Surveys in Florida 4-H • Tabulation of Data • Dropping the data in the sheets is so simple and saves a ton of time.  It is much easier than manually tabulating learning gains • Data is easy to interpret from the sheets • How Data Has Been Used • Data has been used with stake holders to secure program support and has been used in annual reports of accomplishment. • Data will be used in presentations at professional conferences to show outcomes and projected impacts of programming. • 5.   

23. Training Evaluation • Training Program Evaluation Exercise • Confidence in ability • Knowledge gain • Training evaluation

24. How Can You Utilize this Program? • Resources: • www.georgia4h.org/evaluationresources • State Meetings • Webinars • New Agent Training

25. Questions? For more information contact:University of Georgia, State 4-H Office Jeff Buckley, jbuckley@uga.edu Jenna Daniel, jbrown10@uga.edu Casey Mull, mullcd2@uga.edu State 4-H Office – 706-542-4444