Making Sense of Qualitative Data

1. Making Sense of Qualitative Data TESC MPA Program ATPS Winter 2010Geri/Gould/McBride

2. Qualitative data defined • Data describing the attributes or properties possessed by an object. • People’s interpretations of situations, others, objects…any phenomena you can imagine. • Usually expressed in words; language and text • Generated by observation, interviews, focus groups, documents • Watching; asking; examining

3. Challenge of Such Data • Labor intensive data collection • Researcher bias • Cost of processing/coding data • Many variables, small N • Not generalizable • Credibility

4. The Benefits • Grounded in a specific context/situation • Events ,real life settings; lived experience • Deep layers of meaning; rich description filled with symbolism, metaphor

5. The risk….. • “Qualitative analyses can be evocative, illuminating, masterful—and wrong. The story, well told as it is, [may] not fit the data. Reasonable colleagues double-checking the case [may] come up with quite different findings. The interpretations of case informants [may] not match those of the researchers.” (Miles and Huberman, Qualitative Data Analysis, 1994:247)

6. What Do You Want? • Be clear to yourself and your client • Qualresearch usually inductive and theory driven, often using purposive samples-NOT a deductive, sampling logic/inferential statistics approach • Outliers, unusual cases, comparative cases that appear to differ in predictable ways • Example: best practices research

7. Your General Approach? • Grounded? (Start data collection with few preconceived notions about what’s going on….no pre-formed coding scheme) • ….or Framed? (specific events, behaviors you intend to look for, with coding scheme already partially developed) • Framed studies benefit from diagrams ….what do you think is happening?

9. Example…My dissertation… • “Do federal programs forced to rely on user fees engage in business-like behavior?” • My data collection and analysis were partially framed…I was looking for actions, structural changes, decisions that had a private-sector orientation • 4 cases, two regulatory, two non-reg • Results: all agencies more business-like, with non-reg more than reg

11. Qual Analysis: The General Process • Data Reduction • Data Display • Conclusion Drawing • These are not linear, but concurrent processes • The less “framed” and more grounded the process, the more they are concurrent: constant comparison

12. Data Reduction • First we transform data from field notes or transcriptions • Write up and/or transcribe field notes • Print • Which of the data are most useful? • Coding and analysis • Some quantitative analysis

13. Manage Your Data • Level of organization needs to be consistent with complexity of your project • Format field notes consistently, index so you can find documents • Make sure you can read them! • Have a sensible system for cross referencing your notes • Document codes that emerge • Write ongoing memos and abstracts

14. Consider…. • For extensive research projects, summarize interviews with a brief cover sheet • Who, what, where, when, importance, summary of key contacts • Coding schemes…must match the complexity of the project • Use similar semantics • Concepts, patterns, memos

15. Basic Goal of this Stage • Identify important phenomena • Begin to speculate about what might be happening

16. Coding….one approach • Start with a sample of the data • Read responses carefully…keep RQ in mind • Find pithy descriptors that describe actions, events, other relevant phenomena • Make rough categories of these descriptors that seem to belong together and code them with a key word.

17. Example • Why do respondents smoke? • Resp.1: It gives me pleasure….I like to blow smoke rings • Possible code: ‘pleasure’, pleas. • Resp. 2: “I like to smoke when I’m with friends; people are more sociable • Possible code: “sociable”, socioab

19. Variations…. • Start categorizing early… • Or ….. • dive deeper into the data and avoid making judgments too early…make tentative observations about what might be happening…. • Keep a list of your codes • And write memos to yourself about what you think is happening

20. Sometimes… • “A little data, and a lot of right brain” is the best approach….. • What is happening? How to make sense of a strange interview?

22. Data Display • Playing with typologies and displays is a part of the analysis process • See Miles and Huberman, Qualitative Data Analysis • Make sense of the data by playing with visual means of representing the patterns that are emerging from the analysis • Processes and outcomes

25. Theorize: cause and effect? • Classic Conditions for Establishing Cause and Effect • Variables Covary. • Covariance is not spurious. • Logical time order. • A lucid explanation is available • Or …clusters of phenomena, identify things that tend often to show up together, even if the causal connection is not clear.

27. Example: analysis of medical errors • What is happening? • “Figure 1 classifies the stage in the diagnostic testing process and the transition points within and between stages at which errors can occur, and presents representative occurrences that fall into each of them.

29. Example…. • Reasons for early introduction of soft foods by young mothers

31. Drawing Conclusions • Summarize the data and results of coding analysis • Patterns and themes • Clusters of similar findings? • Case comparisons • Powerful metaphors • Any data for which your theory can’t provide a reasonable explanation?

32. Verification • Look for disconfirming evidence • Triangulate from multiple sources or methods • Easier with multiple researchers • If it’s just you, double or triple check your data and conclusions

33. Standards • Be true to the data • Don’t get too carried away by particularly eloquent or memorable respondents—this creates a cognitive bias • Always check and recheck both the data and conclusions you draw from it

34. Case: TESC Alumni Relations • Why do colleges and universities have alumni programs?

35. Research questions • What are TESC graduates’ perceptions of TESC’s alumni programs? • What kind of alumni program do they want? • How do they recall their experience as TESC students? • What connects them to the College? • What nourishes that connection? • What can AR do to improve those connections?

36. Approach • Draft questions; approval from Alumni Relations • Zoomerang online survey • 1647 responses • One researcher • Pluses: clear conclusions, grounded in data • Minus: not validated by second researcher

37. Sources • Harris, et al, Mixed Methods Analysis of Medical Error Event Reports: A Report from the ASIPS Collaborative • http://www.ncbi.nlm.nih.gov/bookshelf/br.fcgi?book=aps2&part=A2024

38. Workshop • Code 2 or 3 pages of the data from the responses to the Alumni survey question, • “What was the best part of your experience at Evergreen?” • Code individual responses • What are the most common codes? • What do these data tell you/us about these alumni ? About Evergreen?