Qualitative Data Analysis

1. Qualitative Data Analysis Chapter 11, Research Methods for Public Administrators Dr. Gail Johnson Revised & expanded for PADM 522, Designing Applied Research Dr. Mario Rivera Dr. G. Johnson, www.ResearchDemystified.org

2. Analyzing Qualitative Data A process of making sense of non-numeric data. Data from: • Narrative documents (speeches, newspapers, diaries, reports, etc) • Open-ended interviews • Open-ended questions on a survey • Case study (as the principal method or as embedded in a larger complex of qualitative data and forms of analysis). • Often, qualitative evaluation research incorporates cross-case analysis: For instance in evaluating two sites of the same program, or in assessing a program site against another without the intervention. An example might be a school with an extended learning program to others without it. Dr. G. Johnson, with additional material by Dr. M. Rivera

3. Qualitative Data Analysis Data From • Focus groups transcripts; interview transcripts and notes • Unstructured observations • Document review • Videos, etc. The goal is to identify common themes • Requires a solid plan, attention to detail, good organization and sufficient time • Whether the analysis is done by computer or by hand, it is often necessary to develop a coding scheme so the data can be systematically organized and analyzed • Computer software can help locate and organize data according to the coding scheme created Dr. G. Johnson & Dr. M. Rivera

4. Qualitative Data Analysis • But the analysis—that is, making sense of the data, discovering the story the data reveals—is done by the researcher(s) • The greatest concern is bias/paradigm blinders: hard to recognize things you do not expect, or to avoid finding things you are inclined to find. • Maintaining the Thread • Review the data, make notes as you go along • Read again once all the data collection is completed • Organize the data—according to research questions, by date, by geographic location—what makes sense given the situation • Begin to look for recurrent themes. In one cross-case evaluation of a family-based drug education program, for instance, the instructor (Rivera) found clear commonalities and divergences in program-development in two Navajo communities in New Mexico. Dr. G. Johnson, Dr. M. Rivera

5. Basic Activities: Preliminary Analysis • Identify common words, ideas, themes • Develop spreadsheet or write on index cards • Identify “quotable quotes”—that is, quotes that highlight the key issues: general views, divergent views, a range views. In evaluating an afterschool science program for middle-school students, Dr. Rivera conducted short interviews with students and with their parents, on various occasions. These exchanges produced quotes pertaining to favorite and least favorite parts of the program, parents’ views on program impact on students, etc. In another program evaluation, short quotations from participants supplemented survey results. Dr. G. Johnson, Dr. M. Rivera

6. Qualitative Data Analysis: Coding Process • When applicable (focus groups, content analysis of documents), code the data • Identify common word, issues, themes and go through the material and label them according to that coding scheme • You may go through the material several times • Sometimes a major theme emerges at the end and you will need to go back through to see if it was present in the earlier data • Software is available that may help. Major software are ATLAS.ti and NUD*IST, others like NVivo are specialized for various uses. There is growing literature on the use of computer- assisted qualitative data analysis software (CAQDAS). • Researchers might also use spreadsheets, index cards, color coding of documents, or other devices or combinations of these. Dr. G. Johnson, Dr. M. Rivera

7. Ensuring Quality in Qualitative Analysis • Work to ensure inter-rater reliability • Like content analysis, the researchers should review the same material and apply the coding scheme • Then review and determine if there are difference • Discuss, revise the coding scheme if necessary and retrain the coders • Repeat this test until there is agreement in the coding • Compare results, work out differences, then code all the material • Inter-rater reliability was a major concern in the assessment of teacher dossiers for advancement in tier, in Dr. Rivera’s evaluation of Three-tier Teacher Licensure in New Mexico in 2003-2004. Several trial (but real) assessments of dossiers by 3-person rater teams, of teacher volunteering to be thus evaluated, were tried during 2004-05, a transitional year for the new system. Dr. G. Johnson, Dr. Mario Rivera

8. Qualitative Data Analysis: General Process • Interpreting the data • Making sense of the data: Review salient themes • Are there any seeming relationships between the themes or characteristics? • What are the major points that emerge from the analysis? • What about minor points? • Sometimes an uncommon theme stands out, or is noted by one or two investigators • The researchers need to consider that perspective; always indicate when it is a minority view and be careful that it does not trump majority views • When evaluating programs, principal investigators and program managers are often engaged in conjoint analysis by or with the evaluator. Eliciting a variety of views is important. • These are stakeholders, and stakeholders, when really engaged, make us smarter. Cf. Chen on “stakeholder validity” vs. scientific validity.* *Chen, H. (2005). Practical program evaluation: Assessing and improving planning, implementation, and effectiveness. Thousand Oaks: Sage Publications.

9. One stakeholder engagement technique: the Affinity Diagram • A group process for analyzing qualitative data: • Research team reviews the material • In silence, write down each idea, key work theme on a sticky note • In silence, post on wall • In silence, sort into similar categories • Once the notes are organized, discuss; look for consensus themes • Interpret the data with stakeholders • Reality check: do others involved in program or project agree? • Share preliminary drafts with stakeholders or a small group of the participants to explore the issues you have discovered • Share final draft with experts and cold readers

10. Affinity Diagram • Identify common themes • The Affinity Diagram tool is a quick way to set up a coding framework • It is also a great way to identify the major themes and topics in the final article or report • It has the advantage of capturing all ideas in a room • Everyone’s ideas have equal value when posted • Focus on major themes: • “Some said X, while others said Y . . .” • Avoid generalizations: if not everyone was asked to comment about a particular practice, you do not know if the five who commented represented the views of others or were just the only ones that thought this practice was important.

11. Leading Discussion and Writing Up Results—Could function as a facilitated focus group • Provide a range of perspectives. • Highlight interesting perspectives even if only said by one or two people. • A few offered unique views: “….” • These might be important because “…..” • Do not try to report numbers or percents unless everyone was counted in exactly the same way • For example, 10 people out of the 20 who you interviewed commented that they liked the training program. • But unless you asked all 20 people, you do not know what the other 10 people thought. Dr. G. Johnson, Dr. Mario Rivera

12. Limitations of Qualitative Data Analysis • It can be like a Rorschach test –an inkblot that some see as a butterfly and others see as maple leaf • Reasonable people can read the same material and have very different interpretations. It is important to explore those differences to gain a deeper understanding rather than to win the argument • It is much more difficult to generalize from qualitative analysis than otherwise. Questions of validity, reliability, replicability, etc., inevitably arise with qualitative data. These may be overweighed by the richness of insight gleaned from the data (“thick description”), but these challenges need to be taken into account and addressed with some directness in research reports. • Best when combined with quantitative analysis in mixed-methods research, which allows for triangulation of data and methods alike.

13. Creative Commons • This PowerPoint is meant to be used and shared with attribution • Please provide feedback • If you make changes, please share freely and send me a copy of changes: • Johnsong62@gmail.com • Visit www.creativecommons.org for more information Dr. G. Johnson, www.ResearchDemystified.org