Data Collection Methods

1. Data Collection Methods Dr. Gail Johnson

2. Steps in the Research Process Planning 1. Determining your questions • Selecting an evaluation research design • Identifying your measures and measurement strategy 4. Developing your data collection strategy 5. Identifying your analysis strategy 6. Reviewing and testing your plan

3. Data Collection Options The decision depends upon: • What you want to know • Numbers or stories • Where the data resides • Environment, files, people • Resources available

4. Where does the Data Reside? Where are the best sources of data? Does the data already exist? Do you have to collect new data?

5. Types of Data Collection • Observations • Available data • Self-administered questionnaires • In-person interviews • Telephone surveys • Focus groups

6. Data Collection: General Rules • Use available data if already exists. • Faster, cheaper, easier • But find out how they: • collected the data • defined the variables • ensured accuracy of the data

7. Data Collection: General Rules If you must collect original data: • Establish procedures that are followed • Maintain accurate records of definitions, coding • Remember: all self-reported information may or may not be accurate. Perception of behavior may not be the same as actual behavior! • Pre-test, pre-test, pre-test, pre-test • Verify accuracy of coding, data input

8. Surveys • Mail (e-mail) questionnaires • Phone survey • In-person interviews

9. Survey Options: • Structured • close-ended questions • Unstructured • open-ended questions

10. Structured Questions How satisfied or unsatisfied are you with the graduates of the teachers college? Very Unsatisfied <---- > Very Satisfied How helpful or unhelpful have the agricultural consultants been in working with you in the past year? Very Unhelpful <------- > Very helpful How useful, if at all, has the program evaluation workshop been in helping your evaluate your program? Of little usefulness <------- > Very Useful

11. Unstructured Questions • What are the greatest assets you have observed in the MPA graduates? • What knowledge and skills should MPA graduates have for the 21st century? • What are the three things that you learned from the MPA program that you use most frequently?

12. Writing Questions • Use clear, simple language • Ask only one question at a time • Phrase so all responses are acceptable • To what extent, if at all, • How important or unimportant • Leave exits (no opinion, not applicable)

13. Writing Questions • Encourage a range of responses • Use intensity scales: 5-7 points • I prefer ones with adjectives (excellent/ very good /good /poor /fair) • Avoid yes/no responses • Soften the ends • Always or almost always • Never or almost never

14. Writing Questions • Close the gates • Responses should be mutually exclusive • Time-specific (last week, last month) • Focus on current experiences(avoid memory decay) • Provide sufficient instructions • Check only one, check all that apply • Only ask necessary demographic questions

15. Writing Questions • Use existing questionnaires • Obtain expert review • Pre-test, pre-test, pre-test • Actual completion and debriefing • Final review: dot I’s and cross t’s

16. General Guidelines • Keep it simple, clear, easy, short • Respect respondents time and intelligence • Tell them how they were selected and why their participation is important • Do no harm: keep responses confidential

17. Survey Options: Mail • Usually very structured • Relatively inexpensive • Long time to develop and implement • Easy to analyze • Unforgiving of mistakes • Should take less than < 30 minutes • Better for confidential questions

18. Mail Survey • Improve response rates • Looks good • Easy to complete • Signed letter • Self-addressed, stamped envelop • Incentive

19. Survey Options: Phone • Usually very structured • Simple questions • Time to develop • Easy to analyze

20. Survey Options: Phone • Moderately expensive • Quick way to gather data • But you may have to call a lot of numbers • Interactive • < 15 minutes

21. DiscussionSurvey Options: Phone • When might a phone survey be better than a mail survey? • When might a mail survey be better than a phone survey?

22. Survey Option: Interviews • Structured or unstructured • Can be complex • In-depth • Expensive • Forgiving of mistakes • Can last one hour or more

23. Survey Option: Interviews • When might interviews be better than mail or phone surveys? • When might mail or phone be better than interviews?

24. Focus Groups • Small groups (6-12 people) • Comfortable environment • Good facilitation essential • Facilitator and note-taker • Tape record if possible • Ask few open-ended questions • How many? Until no new themes emerge • Food, incentives, childcare, transportation

25. Focus Groups: Typical Questions • What did you learn from the MPA program that is most helpful to you in your job? • What were the greatest challenges in obtaining your MPA degree? • What should the MPA program teach so that graduates are prepared to meet the current challenges in public administration?

26. Focus Groups • Advantages: • Relatively quick • May be less expensive (staff time) than in-depth interviews • Provides flexibility to make changes in process • Can examine different perspectives • Fun!!

27. Focus Groups Disadvantages: • Can't do quantitative analysis • Analysis: very time consuming • Risk of interpretation bias • Have to control dominant talkers • May be expensive when calculated on a per person basis

28. Overview: Gathering People’s Perceptions • Advantages: • Only they know what they think • Only they can tell you their opinions • Disadvantages: • Self-reported data • Susceptible to bias

29. Combinations It is often helpful to use several different approaches so that the weaknesses of one approach is offset by the strengths of another.

30. Combinations • Quantitative and qualitative data collection • Available data with surveys • Surveys with observations • Observations with available data • Surveys with focus groups

31. Possible Combinations • Include a few open-ended questions on a self-administered survey. • Ask a few close-ended questions in an interview. • Do focus groups to help develop survey questions. • Do focus groups to explore findings from a survey. • Include a survey at end of focus groups.

32. Ethical Issues • Do no harm. Do no harm. Do no harm. • Inform people about potential harm, whether it is about medicine or loss of training opportunities. • Do not force people to participate in any study. • When needed, obtain proper consents. • Respect people’s privacy rights. • When you promise confidentiality, mean it. • You may need to obtain human subjects approval.