Strategies for Evaluation Data Collection Eric Graig, Ph.D.

1. Strategies for Evaluation Data Collection Eric Graig, Ph.D.

2. Session Objectives • Participants will understand the factors to consider when selecting data collection methods • Participants will become familiar with the data collection methods most often used in evaluation studies • Participants will leave with an understanding of the resources Innovation network and LSC are providing to help develop effective data collection tools

3. More Information http://www.innonet.org/services/LSC_Train

4. Starting Points • Evaluation questions • Evaluation plan

5. Choosing a Data Collection Strategy • Process evaluation • Outcome evaluation • Organizational capacity • Budget • Expertise • Staff time • Desired / Required N • Experiments • Surveys • Standardized instruments • Interviews • Focus groups • Review of existing records

6. Integrate Data Collection into Operations • The best data collection strategy is one that is seamlessly integrated into you day-to-day operations. • Think of your organization as an information system. Where in your workflow can program evaluation information be best captured?

7. Be Systematic • Effective evaluation requires a close attention to detail. You needn’t be scientific but you should strive to be systematic. • Record keeping is primary. • Encourage your left-brain to create forms and protocols. • Write everything down.

8. Example 1

9. Example 2

10. When to Use Surveys • When your evaluation questions are well defined and the range of possible answers is limited • When you need to collect data from a fairly large number of people and you have access to them and can reasonably expect them to cooperate with the survey • When you have the ability to process and analyze the relatively large amount of data you will collect

11. Kinds of Survey Questions • Knowledge questions • Attitude questions • Behavior questions • Subject characteristics

12. Generating Survey Questions Evaluation Questions Evaluation Sub-questions Operational Questions Individual Survey Items

13. Evaluation Questions Evaluation Sub-questions Operational Questions Individual Survey Items Generating Survey Questions • What was learned? • How was identity affected? • Were students satisfied? • Satisfied w/ registration proc. • Satisfied with support received • Satisfied w/ course component • Relevance of content to visits • Quality of the instructors • Convenience of class schedule • Right days • Right time • Right class length

14. Writing Survey Questions • Question formats • Multiple choice • Ordinal scales • Multi-punch questions • Number of choices provided • Even number for a scale • Strive for clarity • Use appropriate language • Avoid words with multiple meanings • Be careful around dates and amounts

15. How often do you attend religious services? • Once a week or more • Several times a month • About once a month • Several times a year • Never Creating Neutral Questions • Dealing with obviously sensitive topics • Dealing with less obviously sensitive topics Some people attend religious services once or several times each week. Others attend only several times a year or not at all. Thinking back over the last year, how often would you say you attended religious services?

16. Survey Delivery • In-person paper and pencil • Mail paper and pencil • Internet • Telephone

17. Questions

18. When to Use Interviews • When you are trying to understand someone else’s subjective experience. • When your evaluation questions are more open-ended and the range of possible answers is large. • When you don’t need to generalize your findings to a larger group.

19. Interview Questions • Avoid questions that lead to ‘motive talk’ • Try to use: • Could you tell me about that • Please tell me more about that • How did that work / come about / feel • Ask interviewees to provide a narrative account of what happened or how they felt

20. Tips for Interviews • Spend time on small talk in order to make the interviewee feel comfortable • Use a written set of questions with probes • Follow the protocol but remain open to exploring other themes as they arise • Record if at all possible

21. When to Use Focus Groups • When you are trying to understand subjective experience • When your evaluation questions are more open-ended • When you sense that respondents may be uncomfortable in a one-on-one interview • When your evaluation questions require you to understand a group dynamic

22. Tips for Focus Groups • Same as for interviews • Plan on 8 to 10 participants and accept that one or two may not show up • Get a good mix of participants • Learn how to prevent particular participants from dominating the group and how to encourage wall flowers to speak up • Consider an introductory exercise to get the people comfortable with one another

23. Questions

24. When to Program Records • When you have them and can access them • When you can vouch for their accuracy and reliability and when you know they are complete • When you can understand them

25. Tips for Analyzing Program Records • Very their accuracy • Find out what, if anything, is missing • Be systematic • Plan on making recommendations about how to improve them to make them more useful for evaluation

26. Questions

27. Additional Resources • InnoNet website www.innonet.org • Asynchronous material • Synchronous course • One-on-one technical assistance