Creative or alternative methods for collecting data

1. Creative or alternative methods for collecting data

2. Surveys, end-of-session questionnaires, interviews and focus groups have become popular ways to collect data. • What other ways can you collect data that may be more novel, more appropriate and yield better information? • Remember, always check whether the proposed method will be culturally appropriate. Let your mind roam….

3. Using existing records and forms Record books: • Entries in record books can provide insights about the environment in a club or project area; participants’ perceptions of activities; statements of learning and actions. For example, members may comment on growth in leadership skills, “I have learned how to be more organized” or “I feel more confident in my leadership role”; or “I led the group to…” • You might take note of sections that are filled out and those that aren’t

4. Using existing records and forms Exhibits: • can be counted and compared to previous years’ entries; a team of people can view exhibits to assess outcomes or list new ideas that may be used in program development; judge’s comments can be gathered and summarized; winning entries may be noted

5. Using existing records and forms Application, registration and scholarship forms Forms that are completed by members for awards and trips and other things such as camp counselor applications may contain useful data; for example, answers to questions such as “How has 4-H enhanced your problem solving skills?” or, “How would you describe 4-H to a group of peers?” Or, include questions to solicit data you need

6. Using existing records and forms Committee and participation lists You keep a variety of lists that can be used to track numbers and key demographics of who is (and who is not) signing up for what. You can monitor trends over time and get an idea of how youth are perceiving the program.

7. What other existing records and forms might you use? Take a few minutes and think about the various records and forms you have that either: • already have data that you can use for your evaluation; or • you can modify to collect the data you need (perhaps, by adding a question or two).

8. Using games to collect data Value line or ‘line rating’ • In a value line, people position themselves along a line to denote their position about a topic. • Signs are often posted on the wall (or along a line on the floor) that provide the rating scale (for example, strongly agree to strongly disagree; very helpful to not helpful) • Develop clear strong statements that relate clearly to the topic and what you want to measure • You can use the value line at the end of an event; or use it at the beginning and again at the end to collect pre-post data or with a sample of clubs

9. Example value line Program: Developing club leadership. (Example provided by Mary Klemme) Part of your educational plan is to conduct some workshops about creating a positive environment for youth leadership development. You want to know how clubs are functioning in this area currently and how effective your workshops are. You decide to use a value line exercise with a sampling of clubs both before and after your workshops.

10. Count the number of people standing in the different areas and record the numbers. You can also use this process to facilitate discussion about why people took various positions.

11. Games – Card Sort • Print brief explanations of program outcomes (or whatever you are seeking information about and wish people to rate or rank) on 3 x 5 cards. Make one set of cards for each participant. Or, let participants generate the items (e.g., benefits they gained from the program) and create one set of cards for each person. • Write the rating scale (excellent-poor) or ranks (1st, 2nd, 3rd…) on baskets placed on a table. • Ask participants to sort and put their cards into the appropriate basket. • You may wish to prioritize the cards in each basket. • This can be done individually or as small groups working together. • Simple key words or graphic images can be used to ease literacy requirements.

12. Games… How might you use such games as: • Matching • Jeopardy • Scattagories • Etc. to collect evaluation information?

13. Using photography to collect data • Before and after photos of community service activities; project activity; skill development program, camp, etc. • Photos taken over the course of a project to show development and changes • Engage youth in taking and interpreting their own photos

14. Photography Example. You want to know what the camp experience means to youth. You randomly select a group of campers who are asked to take photos during camp. The photographers then gather in small groups to discuss their photos, why they took the photo, what it means to them, etc. Comments and remarks are recorded.

15. Photolanguage A qualitative data collection technique that uses photographs to help individuals express themselves more fully when asked questions. The photographs are typically black-and-white images in order to allow viewers to focus on the topic or essence of the photos. Select photos related to the evaluation topic that will stimulate conversation. For example, if you want to know what youth learned during the Winter Leadership Camp, you might say, “Select a photo that helps you describe what you learned during the leadership camp.”

16. Photolanguage continued… Record what participants say; analyze the data You can use photolanguage to assess needs, document outcomes and experiences. See an article in the Journal of Extension, “Photos can inspire a thousand words” by White, Sasser, Bogren and Morgan, June 2009 http://www.joe.org/joe/2009june/iw1.php

17. Using creative expression to collect evaluation data Various forms of creative expression can be used to collect evaluative data to document changes in knowledge, attitudes, skills, feelings and behaviors. • Drawing • Drama • Role-playing • Story telling; storyboards • Music

18. Drawing is a natural mode of expression for boys and girls. It is a nonverbal Language and form of communication that can be analyzed for structure, quality and content. • You might ask participants to draw a picture of themselves doing their favorite club activity, after-school activity, camp activity.

19. Storyboards as creative expression • A series of pictures, illustrations or graphics that tell a story about a program, its impact and what youth experienced. • Youth might create their storyboards individually or as a group. • They might draw their storyboards on paper or using computer technologies. • You might give each person/group a specific aspect to draw (relative to your evaluation questions) or let them identify topics they feel would provide useful evaluation information. • Once the storyboards are complete, share and discuss. Record comments as additional evaluative data.

21. Journaling Journals are personal and allow youth to think about their experiences, what they are feeling and thinking and provides a useful ongoing account of an activity or experience. Helping young people journal effectively means more than giving them a notebook. When using journaling to collect evaluation data, be clear about what you want to know and give them a list of questions to address. Suggest a structure for the journal (or provide pre-printed pages). Space might be designated for days events; feelings about the events; reflection on what was learned; what actions they will take. You might consider a group journal where all members write and share their experiences and thoughts in a combined effort.

22. Journaling Do a Google search. There are many resources on journaling on the web. One is from the University of Tennessee 4-HYD, titled Reflection journal at http://www.utextension.utk.edu/4h/sos/whatisit/reflection/reflection_journal.htm

23. Action cards Action cards are index cards on which participants record steps, actions, and accomplishments during the course of a project or program. Individuals or groups write a short entry on each card - a few words or sentences. The cards help to monitor progress, critical incidents and accomplishments. Youth can fill in the cards at regularly scheduled meetings or specified times. Information on the cards become data for later analysis.

24. Group techniques Any of the common group techniques that we use in program planning are also useful for collecting evaluation information: Brainstorming Nominal group technique Delphi technique Affinity diagramming

25. Group technique examples… • A talking circle: youth form a talking circle and provide their comments about their experiences and benefits gained from the program; comments are recorded • A modified world café: participants move from table to table building upon a series of evaluation questions; the table ‘host’ records key points • Brainstorming groups: small groups brainstorm answers to the evaluation questions with a summary list recorded on paper • See also information in the presentation on Interviews – Group interviews

26. Goal setting as data collection Your Goal Participants set their own goal(s) at the beginning of the program (e.g., desired weight change; change in exercise pattern; amount of money to save; increase in public speaking skills, etc). Depending upon the length of the program, you ask them to revisit their goal during the program (record progress) and/or at the end. Progress and achievement are recorded as evaluation data.

27. Fun with stickers! This idea may work well with young children. Count out a fixed number of stickers and place the same number in each of three cups (use any number of cups). Label each cup with “learned a lot”, “learned a little”, didn’t learn anything”, or whatever response options fit your question. Ask each youth to take a sticker from the cup that best answers the question you ask (your evaluation questions). Tally after each question. The youth get a fun sticker and you get evaluation data.

28. To evaluate a program at its conclusion Draw a large head, heart and feet on flipchart paper and post it. Distribute small slips of paper and ask participants to write down the major things they learned or got out of the program. Have them post these in the appropriate position on the diagram and discuss them. Record comments. Or, invite participants to create their own head, heart and feet and fill in the form, either individually or with someone else. You might want to write: ‘What you think’; ‘What you feel’; ‘What you will do’ next to the head, heart and feet. Source: Educating for a Change. Adapted from Marsha Sfeir, a Toronto educator.

29. To evaluate a program at its conclusion Three key points: 1. 2. 3. Another variation for quickly collecting data at the end of a session or program is a card like below: What is still circling in your mind! What is squared away with you that you can apply and use?

30. To evaluate a program at its conclusion Here is another version of a card for quickly collecting data at the end of a session or program:

31. See the PDF on the web site for additional ideas using • Creative expression • Personal stories or testimonials • Video taping or photography • Expert review • Diaries and journals • Logs • Case study

32. Using technology for collecting data • Many of the ideas in this slide set might be adapted to the computer. • Also, see another section on this web site for ‘using technology in data collection’.

33. Analyzing data when using creative methods • While many of the methods covered in these slides seem fun and easy, think about how you will record and analyze the data you collect. • Avoid collecting data unless you have a systematic process for analysis and plan for using the data. • Many of these methods require the use of qualitative data analysis that involves categorizing the data, identifying themes and summarizing the key points. (see the section on qualitative data analysis)

34. Integrating evaluation into your programming • As you can see from this presentation, there are many ways to integrate data collection into your programming so that it doesn’t become an ‘add on’, boring or another questionnaire to complete. • Use your own creativity to develop useful and culturally appropriate data collection methods. Make sure the method aligns with your evaluation purpose and what you want to know!

35. Reflection time What is one thing you learned (or had reinforced)from going through this presentation that you might try in your own evaluation? Good luck with your data collection efforts!