Achieving Appropriate Rigor in Qualitative Research

1. Achieving Appropriate Rigor in Qualitative Research Research Day, February 4, 2011 Mary Katherine O’Connor, Ph.D. School of Social Work, Virginia Commonwealth University mkoconno@vcu.edu

2. Presentation Goals • Propose a multi-paradigmatic heuristic for understanding variety in qualitative research • Detail differential standards for research quality depending upon paradigmatic perspective • Aid in matching research questions, analytical needs and computer-based qualitative analysis packages

3. Two Dimensions of Analysis • Subjective/objective relates to assumptions about the nature of knowing (and, therefore, of science) • Regulation/radical change relates to assumptions about the nature of society (and, therefore, appropriate goals for science)

4. Sociology of Radical Change Radical Structuralist Radical Humanist Subjectivity Objectivity Interpretive Functionalist Sociology of Regulation Burrell & Morgan’s Framework

5. Multiple Paradigms • Fundamentally different perspectives for analysis of social phenomena • Generating different concepts and analytical tools • Different standards for good research practice • Four different types of qualitative research building from different types of research questions and research goals

6. Multiple Paradigms in Qualitative Research

7. Critical Interpretive RadicalChange Critical Positivist Subjective Objective Interpretive Positivist/ Post Positivist Regulation

8. Research Goals

9. RadicalChange Subjective Objective Positivist/ Post Positivist Regulation

10. Positivist Research:UNDERSTANDING FOR THEORY BUILDING THAT CAN RESULT IN THEORY TESTING • Assumptions: Rational, pragmatic methods of natural science are appropriate to study human affairs • Questions: Useful for theory building in the early stages of theory development • Standards: understanding for generalizability including random assignment, control groups, valid and reliable data collection, standardized data analysis

11. RadicalChange Critical Positivist Subjective Objective Regulation

12. Critical Positivist Research:CONSCIOUSNESS RAISING AND CHANGE AT THE SYSTEM LEVEL • Assumptions: Knowledge is for radical/transformative change, emancipation and potentiality; realist but concentrating on structural relationships to understand and generate fundamental conflicts that will be the basis for radical change at the class/structural level • Questions: Focus on structural relationships seeking to provide explanations of the basic interrelationship within the context of total social formation for the purpose of consciousness raising • Designs: critical ethnographies, cooperative or collaborative inquiry, participatory action research, appreciative inquiry, empowerment evaluation • Standards: same as functionalist qualitative inquiry, but in addition, measurable presence of increased sophistication with change or change potential

13. RadicalChange Subjective Objective Interpretive Regulation

14. Interpretive Research:MEANING MAKING • Assumptions: Understanding at the level of subjective experience from perspective of participants, not observer. Relativist in that the world, though ordered, is an emergent social enterprise that is continually being created. • Questions: seeking explanation within individual consciousness and subjectivity from standpoint of the participants rather than the observer • Designs: ethnographies, case studies, phenomenological studies, constructivist studies • Standards: sampling for maximum variation, emergent design, multiple data collection approaches, triangulation, reflexivity, inductive data analysis, thick description of research product

15. Critical Interpretive RadicalChange Subjective Objective Regulation

16. Critical Interpretive Research: CONSCIOUSNESS RAISING AND CHANGE AT THE INDIVIDUAL LEVEL • Assumptions: Critical interpretive with radical change focus, but from a subjectivist, individualistic standpoint. Concentrating on human consciousness and the social structures that inhibit true fulfillment. Goal is release of constraints that hamper individual human development. • Questions: how individual makes sense of life and experiences of societal constraints • Designs: interview studies, narrative research, life histories, autoethnographies, collective biographies • Standards: more artistic or aesthetic including creativity, aesthetic quality, interpretive vitality, and degree of stimulation for change

17. Research Goals: Implications • All qualitative research is not based on the same ontological and epistemological assumptions • All qualitative research cannot be expected to be held to a universal standard of rigor, quality or worth • Key to justification of qualitative processes and products is selection of appropriate research design including data collection and analysis techniques able to answer research question lodged within specific paradigmatic perspective QUESTIONS?

18. Research Design

19. Positivist Design: • An appropriate research design could include structured interviewing, focus group, grounded theory, etc. and should be identified and justified along with a data collection mechanism. • The sampling plan is preferably randomized or at least looking for maximum variation along identifiable dimensions. • An “n” of at least 30 or a justification for a lesser number that is similar to a justification for a lesser sample number in a quantitative project. • A data analytic plan (thematic, content, constant comparison, linguistic, etc.) that matches the question, including whether or not computer based text analysis is being used. When computer analysis is present, the selected program must be able to do the analysis proposed. In most cases the variables for analysis should be identified beforehand. The exception would be in grounded theory designs. • Inter-rater reliability should also be addressed.

20. Critical Positivist Design: • An appropriate research design could include empowerment, appreciative, focus group, etc. and should be identified and justified along with a data collection mechanism. • The sampling plan is preferably randomized or at least looking for maximum variation along identifiable dimensions. • An “n” of at least 30 or a justification for a lesser number that is similar to a justification for a lesser number in a quantitative project. • A data analytic plan (thematic, content, constant comparison, linguistic, etc.) that matches the question, including whether or not computer based text analysis is being used. When computer analysis is present, the selected program must be able to do the analysis proposed. In most cases the variables for analysis should be identified beforehand. • Inter-rater reliability should also be addressed. • Change resulting from the research will need to be measured.

21. Interpretive Design: • An appropriate design could include ethnomethodological, constructivist, phenomenological, etc. and should be identified and justified. If the design is emergent, benchmarks for the emergence should be clear including how sampling and data collection will begin and might change. • Sampling for maximum variation is preferred and should identify the dimensions of variation of interest, the stakeholding groups to be sampled, and what might constitute saturation in determining the end of sampling and data collection. • A data analytic plan and the type of analysis should match the question and if computer-based analysis is being used, the program should be geared to interpretive analysis. Variables for analysis will rarely be identified beforehand; therefore, content analysis and constant comparison would be preferred. • Inter-rater reliability is not favored. Instead, member checking with participants is preferred.

22. Critical Interpretive Design: • An appropriate design could include an interview study, narrative research, life history, etc. and should be identified and justified. If the design is emergent, benchmarks for the emergence should be clear including how sampling and data collection will begin and might change. • Sampling will tend to be for a single case; but if the “n” is greater, maximum variation should identify the dimensions of variation of interest, the stakeholding groups to be sampled, and what might constitute saturation in determining the end of sampling and data collection. • A data analytic plan and the type of analysis should match the question and if computer-based analysis is being used, the program should be geared to interpretive analysis. Variables for analysis will rarely be identified beforehand; therefore, content analysis and constant comparison would be preferred. • Inter-rater reliability is not favored. Instead, member checking with participants is preferred. • Plans for documenting individual change as a result of the process can be noted.

23. Data Analysis & Reporting

24. Paradigmatic Implications • Content Analysis Thematic Analysis Linguistic Analysis Semiotics Constant Comparison • Reporting Count Case Study Narratives

25. Content Analysis Thematic Analysis - reading “chunks” of data for meaning and attaching a meaning label (can then move to word count/distribution analysis) Linguistic Analysis - assessing word usage, message style (can then move to word count/distribution analysis) Semiotics - reading for meaning through use of metaphor (can then move to word count/distribution analysis) Constant Comparison - deconstruction and reconstruction through unitization and categorization (can then move to word count/distribution analysis)

26. Reporting • Count- data display, distribution tables, conceptual or causal maps • Case Study – to chronicle (to record temporally and sequentially as in a history); to render (as in a description or to provide a vicarious experience); to teach (as instructional material); to test (using case as a trial for certain theories or hypotheses) • Narrative – factual (he said/she said); interpretive (novel like); evaluative (elaborative judgments for evaluative testing)

27. Computer Assisted Qualitative Data Analysis Software (CAQDAS) Misconceptions about software: • A primary research instrument • A separate form of analysis • Enhances rigor • Packages have similar paradigmatic assumptions

28. Data Analysis with CAQDAS Practical Considerations • Available technology • Skills and abilities • Time Political Considerations • Institutional expectations (IRB, etc.) • Peer review/professional/funding expectations Paradigmatic Considerations • Basic design assumptions • Package similarities and nuanced differences

29. CAQDAS Disciplinary Backgrounds • Atlas.ti – Interdisciplinary, Psychology, Linguistics, Computer Science • NVivo – Sociology • MAXqda – Political Science

30. CAQDAS Similarities • Unitizing • Coding • Annotating • Retrieving • Querying • Graphic representation • Writing Support

31. CAQDAS Differences Types of data Coding structure Contextualization However…

32. …When considering qualitative data analysis software: Access + Awareness  Paradigmatic Congruence

33. Conclusion

34. When there is congruence between paradigmatic perspective, research goal and selected processes, articulation of what can and should be expected for accountability standards are possible and usefulness can be asserted. Positivist work cannot provide deep, individualized meaning, but can be basis for appropriately targeted quantitative work or provide evidence to support causal hypotheses. Critical Positivist work must engender class or structural changes from objective, generalizable data. Interpretive work is not generalizable, but can provide complex, context-based deep understanding. Critical Interpretive work must engender individual, subjective changes at the intersection of art, spirituality and scientific ways of knowing.

35. Protocol Reviews (or…How do you defend what you are doing? Or assess student work?) Try to determine underlying assumptions in order to establish the paradigmatic location of the research by evaluating the aims or goals. Look for mixing of assumptions that suggest an incorrect understanding that qualitative research is always interpretive research Apply the standards for design and analysis quality consistent with the paradigm.

36. Words to the Wise for the Researcher • Be aware of underlying assumptions in project that establish the paradigmatic location of the research, generally best articulated by project aims or goals. • Avoid mixing of assumptions that suggest an incorrect understanding that qualitative research is always interpretive research. • Articulate and demonstrate the standards for design and analysis quality consistent with the paradigmatic perspective of the project…then choose your software