Data Collection Toolkit I: Available Data and Observation

1. Data Collection Toolkit I: Available Data and Observation Research Methods for Public Administrators Dr. Gail Johnson Dr. G. Johnson, www.researchdemystified.org

2. Steps in the Research Process Planning 1. Determining Your Questions 2. Identifying Your Measures and Measurement Strategy 3. Selecting a Research Design 4. Developing Your Data Collection Strategy 5. Identifying Your Analysis Strategy 6. Reviewing and Testing Your Plan Dr. G. Johnson, www.researchdemystified.org

3. Toolkit: Available Data • Files/records • Computer data bases • Other reports • Census data • Documents (budgets, policies and procedures, organizational chart, position descriptions, staff, etc.)‏ Dr. G. Johnson, www.researchdemystified.org

4. Data Collection:Agency Records • Agencies may have already collected the data: from clients, community, internal information systems. • Agencies may have already summarized and reported information: internal management reports, budget documents, and reports to public or funding agencies. Dr. G. Johnson, www.researchdemystified.org

5. Existing Reports • Routine and special reports may already exist. • Fully cite any reports used. • Describe the methodology used for each report. • Sometimes the data is old: • Decide whether it is too old. • Decide whether it can be used as a baseline. • Replicate their data collection strategy using the same exact measures Dr. G. Johnson, www.researchdemystified.org

6. Available Computer Data • Obtain a copy of the data files • Obtain a copy of the database structure (how the data set is laid out). • Obtain a copy of the data dictionary (how the data is labeled and coded). • Transfer data with a minimum of effort (no re-typing) Dr. G. Johnson, www.researchdemystified.org

7. Available Computer Data • But how good is the data? • Garbage in, garbage out applies. • Find out procedures used to collect data. • Find out procedures used to verify accuracy. • Estimate the error rate. Dr. G. Johnson, www.researchdemystified.org

8. Available Data Which of these are likely to be available? • Graduation rates for last year • Budget expenditures for school construction • Teacher commitment to teaching • Student performance • Student satisfaction with the learning experience Dr. G. Johnson, www.researchdemystified.org

9. Available Data: Key Issues • If others collected the data, you want to know enough to determine whether the data are: • Valid • Reliable • Accurate Dr. G. Johnson, www.researchdemystified.org

10. Available Data: Key Questions • How did they operationalize their measures? • How did they collect the data? • How did they ensure accuracy in collecting and transcribing data? Dr. G. Johnson, www.researchdemystified.org

11. Available Data • Advantages: • Often cheaper and faster than collecting data yourself. • Challenges: • Exact data you need may not be available • May have difficulty getting access. • May have too many data errors Dr. G. Johnson, www.researchdemystified.org

12. Toolkit: Data Collection Instrument (DCIs)‏ • Systematically collect data from files, documents, or real life (observation in the field) • A form that specifies exactly what data you want to collect • Structured, check-a-box form makes it easy to collect but may not be possible: • The degree of structure depends on situation Dr. G. Johnson, www.researchdemystified.org

13. Data Collection Instrument • Pre-test the DCI and revise • Establish final decision rules • Train coders • Test for inter-rater reliability • This means that different coders actually code the same material in exactly the same way • This is the same as when coding qualitative data, such as content analysis • If too divergent, revise and test again for inter-rater reliability Dr. G. Johnson, www.researchdemystified.org

14. DCI: Example Describe background and performance of teachers: • Go through files: • A. highest degree: __BA/BS ___MA ___PhD • B. date hired:___________ • C. Number of classes taught last year:__________ • D. Performance ratings for past three years: • 2009: __Excellent __ Good ___ Poor ___Unknown • 2008: __Excellent __ Good ___ Poor ___Unknown • 2007: __Excellent __ Good ___ Poor ___Unknown Dr. G. Johnson, www.researchdemystified.org

15. DCI: Example • E. Age: __ <25 ___ 26-35 ___ >36 • F. Gender: ___ Male ___ Female • G. Current salary: _______________ Dr. G. Johnson, www.researchdemystified.org

16. Discussion and Exercise • You want to look at the qualifications of those admitted to the MPA program • How would you operationalize “qualifications”? • Where is the data likely to reside? • At this early stage, what information might you need in order to determine the most efficient and effective data collection option? • Begin a design matrix Dr. G. Johnson, www.researchdemystified.org

17. Toolkit: Observation • Field Work • Environment • People: on the street, in a lab, in a classroom, at work, etc. • Situations and Process • Structured observations use checklists • Unstructured observations capture impressions Dr. G. Johnson, www.researchdemystified.org

18. Observation Examples • Observe classroom activities to measure the amount of time spent on hands-on learning activities. • Observe amount of traffic on a road from a suburb to downtown city. • Observe the number of salmon in local streams during spawning season • Observe the process used for getting a fighter plane ready to re-deploy • Observe customer service skills of front-line workers • Observe grocery prices in rich and poor neighborhoods Dr. G. Johnson, www.researchdemystified.org 18

19. Observation Exercise Take out a piece of paper • Look around the room and write down everything you observe • What are the gaps between what you see and what you expected? • Why is there a gap? • What changes, if any, would you recommend? Dr. G. Johnson, www.researchdemystified.org

20. Observation Process • How can observation be systematic, reliable and accurate? • Collaboration with stakeholders will help to set realistic standards within the culture of the area. • Have more than one observer: their observation reports can be compared. • Develop a check list. • Train the observers so they observe and record the same things. Dr. G. Johnson, www.researchdemystified.org

21. Observation Process • Develop checklist and decision rules • Train the observers • Test for inter-rater reliability • Observers complete their rating sheets independently • Their ratings would be compared • If there are differences, more training and clarification are needed • If little difference, then they are ready to go live Dr. G. Johnson, www.researchdemystified.org

22. Discussion: Housing Conditions • Suppose we want to describe the conditions of various neighborhoods that have had housing programs over the past 10 years. • What are the conditions you would want to researchers to assess? • How might they collect the information about conditions in an unstructured way? • How might they collect the information about conditions in an unstructured way? Dr. G. Johnson, www.researchdemystified.org

23. Which Would Be a Better Strategy? • You want to determine the effectiveness of the program’s efforts in engaging stakeholders in discussions at meetings. • Observe setting, seating arrangements, interactions, and whatever else is interesting. • Use a checklist to capture specific interactions: who speaks, who does not, who interrupts, who does not, who shows up, who does not. Dr. G. Johnson, www.researchdemystified.org

24. Discussion: Observe “Friendly Environment” • You have been asked to look at whether your organization is a friendly environment to women. • If an outsider were to visit your organization, what might they want to observe to determine if it is “friendly” and how might they go about observing it? Dr. G. Johnson, www.researchdemystified.org

25. Combining Different Approaches • It is not unusual to do a preliminary study (or a feasibility study) in a fairly unstructured way • Sometimes you do not know what you want to know until you are in the actual situation • After that initial kicking tires, it is often easier to develop a more systematic and structured way for a larger observational study Dr. G. Johnson, www.researchdemystified.org

26. Types of Observation • Obtrusive: observe with their knowledge • Unobtrusive: observe without their knowledge Dr. G. Johnson, www.researchdemystified.org 26

27. Obtrusive Measures • Observations of behavior with participant knowledge • Ask their: perceptions, opinions and attitudes gathered through interviews, surveys, focus groups • Scores on tests related to the program—they know they are being judged (eg. Dancing with the Stars)‏ • Told: Dr. X will be observing our classroom today Dr. G. Johnson, www.researchdemystified.org

28. Limitations of Obtrusive Observation • The act of observation is likely to change behavior: • Would teachers perform differently if they were being observed? • Would people at meetings be more polite if they were being observed by an outsider? • Would airplane baggage handlers resist the temptation to play basketball with passengers’ luggage if they were being watched? Dr. G. Johnson, www.researchdemystified.org

29. Unobtrusive Observations • Observation without participant knowledge • One-way mirrors • Participant observer • “Nickel and Dimed”: writer worked at low-wage jobs and wrote about that experience • Calling local IRS service center to observe their knowledge of tax code and customer service • Other ways to gather unobtrusive measures Historical/document/archival data • Data already collected for another purpose: secondary sources Dr. G. Johnson, www.researchdemystified.org

30. Observation: Living on Food Stamps • From august 2008 to august 2009, food stamp users rose from 29.5 million to 36.5 million. Half were children. • Living on the equivalent of food stamps for one month is one way to research some aspects of that experience. • Find out the maximum food stamp allowance for our size family Dr. G. Johnson, www.researchdemystified.org

31. Observation: Living on Food Stamps • Research: data collection • Maintaining a record of what is purchased. • Maintaining a diary of everything that is eaten. • No eating out at a restaurant Dr. G. Johnson, www.researchdemystified.org 31

32. Observation: Living on Food Stamps • What is it like to live on food stamps? • Is there enough food to last for the set amount of time? • Does the food meet the minimum standards of what nutritionists believe is a healthy diet? • How does it feel to be restricted? • What else did you observe/experience? Dr. G. Johnson, www.researchdemystified.org 32

33. Observation at Work • How effective is communication at work? • If you were an anthropologist coming into your workplace as a participant observer, how might you design the study? • What are the things you would want to observe? • What do you think you are likely to observe ? • This is your hypothesis • Decide about measures and scope of your research Dr. G. Johnson, www.researchdemystified.org

34. Grounded Theory Concept • Grounded theory begins by observing and gathering data with no acknowledged filters or coding. • Researchers ake sense of the data and then generate a theory to explain what they have observed. • Begin with roots in ground, move up to the top leaves. • Inductive logic. Dr. G. Johnson, www.researchdemystified.org

35. Grounded Theory Concept • This is the opposite of traditional research which uses deductive logic. • We have an idea about how things work, design a data collection instrument to gather that data, and then test our theory. • This is the funnel approach. Dr. G. Johnson, www.researchdemystified.org

36. Observation • Advantages: • Actual situation or behavior (not self-report) • Real-time (avoids memory decay) • Some researchers enjoy this reality • Challenges: • Observer bias • Coding problem • Can be labor intensive and costly Dr. G. Johnson, www.researchdemystified.org

37. Takeaway Lessons • Data collection requires tremendous attention to detail. • It takes time to nail everything down. • Available data: cost-effective if it the data you actually need and is high quality. • DCIs: often used to collect data from files, databases, and real life observations. • Need to know a lot before a DCI can be structured Dr. G. Johnson, www.researchdemystified.org

38. Takeaway Lessons • Observations: “kicking the tires.” • Unobtrusive observation: closer to reality. • Unlike “reality shows” where they know everything is being filmed-and maybe some incentive to create drama. • But limited by our “paradigm” • Things can be invisible when does not conform to our expectations • Not seen because not looked for Dr. G. Johnson, www.researchdemystified.org 38

39. Creative Commons • This powerpoint is meant to be used and shared with attribution • Please provide feedback • If you make changes, please share freely and send me a copy of changes: • Johnsong62@gmail.com • Visit www.creativecommons.org for more information Dr. G. Johnson, www.researchdemystified.org