T he Validity and Reliability of Qualitative Research

1. The Validity and Reliability of Qualitative Research

2. Training Overview • Section 1: Philosophical Orientation: Getting in the Right Mindset • Section 2:Enhancing the Validity of Qualitative Research • Section 3: Enhancing the Reliability of Qualitative Research • Section 4: Applications of Qualitative Research Design • Section 5: Group Work to Apply Learning

3. Section 1 Philosophical Orientation • The philosophical underpinnings of qualitative research (as contrasted against quantitative research) • Implications for sampling • Implications for addressing issues of validity and reliability

4. Quantitative and Qualitative Paradigms Contrasted • Quantitative Research is … • Fundamentally an inferential enterprise that seeks to uncover universal principles • Philosophically and methodologically built or designed around the ability to infer from a sample to a larger population

5. Quantitative and Qualitative Paradigms Contrasted • Qualitative Research is … • Fundamentally an interpretive enterprise that is context-dependent • Philosophically and methodologically built or designed around the ability to interpret (comprehend/understand) a phenomenon from an emic(insider), as well as an etic (outsider) perspective • This inter-subjective (i.e., shared) understanding serves as a proxy for “objectivity”

6. Implications for Research Questions • A client wants to know about how well a program is working for youth: • Quantitative Research Questions (descriptive, explanatory ->inferential) • Qualitative Research Questions (descriptive, explanatory ->interpretive)

7. Implications for Sampling • Sampling Strategies used in Quantitative Research: • Obtaining a random or representative sample (based on probabilities) • Permits the researcher to infer from a segment of the population (from which it is more feasible to collect data) to a larger population

8. Implications for Sampling • Sampling Strategies used in Qualitative Research: • Purposive sampling (to ensure that the researcher has adequately understood the variation in the phenomena of interest) • Theoretical sampling (to test developing ideas about a setting by selecting phenomena that are crucial to the validity of those ideas) • Example: Case study selection in ACY (see, also, handout on sampling techniques)

9. Implications for Handling Threats to Validity and Reliability • In quantitative research, threats to validity are addressed by prior design features (such as randomization and controls)

10. Implications for Handling Threats to Validity and Reliability • In qualitative research, such prior elimination of threats to validity is less possible because: • qualitative research is more inductive, and • it focuses primarily on understanding particulars rather than generalizing to universals. • Qualitative researchers view “threats” as an opportunity for learning - e.g. researcher effects and bias are part of the story that is told; they are not “controlled for”

11. Section 2 Implications: Enhancing the Validity of Qualitative Research • Defining validity within the qualitative paradigm • Major types of validity within the qualitative paradigm • Design considerations

12. Validity Validity is not a commodity that can be purchased with techniques … Rather, validity is, like integrity, character and quality, to be assessed relative to purposes and circumstances. Brinberg and McGrath 1985:13

13. Validity • In general, validity concerns the degree to which an account is accurate or truthful • In qualitative research, validity concerns the degree to which a finding is judged to have been interpreted in a correct way

14. Assessing the Validity of Qualitative Research Can another research read your field (and other types of) notes (i.e., the explication of your logic) and come to the same understandings of a given phenomenon? • Concern about validity (as well as reliability) is the primary reason thick description is an essential component of the qualitative research enterprise • Resources: • Handout: Different Types of Notes • Example: ACY Site Visit Toolkit

15. Major Types of Validity in Qualitative Research • Descriptive Validity • Interpretive Validity • Theoretical Validity • External Validity (i.e., generalizability)

16. Descriptive Validity • Concerned with the factual accuracy of an account (that is, making sure one is not making up or distorting the things one hears and sees) • All subsequent types of validity are dependent on the existence of this fundamental aspect of validity

17. Descriptive Validity Behavior must be attended to, and with some exactness, because it is through the flow of behavior – or, more precisely, social action – that cultural forms find articulation. Geertz 1973:17

18. Interpretive Validity • Interpretive accounts are grounded in the language of the people studied and rely, as much as possible, on their own words and concepts • At issue, then, is the accuracy of the concepts as applied to the perspective of the individuals included in the account

19. Interpretive Validity: Design Consideration • While the relevant consensus about the terms used in description rests in the research community, the relevant consensus for the terms used in interpretation rests, to a substantial extent, in the community studied • An important design element, for increasing interpretive validity, therefore, is to employee, at some level/to some degree, a participatory research approach (e.g., through member checks, peer to peer research model, etc.)

20. Theoretical Validity • Theoretical understanding goes beyond concrete description and interpretation; it’s value is derived based on its ability to explain succinctly the most amount of data • A theory articulates/formulates a model of relationships as they are postulated to exist between salient variables or concepts • Theoretical validity is thus concerned, not only with the validity of the concepts, but also their postulated relationships to one another, and thus its “goodness of fit” as an explanation

21. Major Threats to Validity • Type I error: believing a principle to be true when it is not (i.e., mistakenly rejecting the null hypothesis) • Type II error: rejecting a principle when, in fact, it is true • Type III error: asking the wrong question

22. Triangulation: An Important Theoretical Validity Check • Case example: Parable of the blind men and the elephant

23. Triangulation: An Important Theoretical Validity Check The most fertile search for validity comes from a combined series of different measures, each with its own idiosyncratic weaknesses, each pointed to a single hypothesis. When a hypothesis can survive the confrontation of a series of complementary methods of testing, it contains a degree of validity unattainable by one tested within the more constricted framework of a single method. Webb et al. 1966:174

24. External Validity in Qualitative Research • There is broad agreement that generalizability (in the sense of producing laws that apply universally) is not a useful standard or goal for qualitative research • This is not to say, however, that studies conducted to examine a particular phenomenon in a unique setting cannot contribute to the development of a body of knowledge accumulating about that particular phenomenon of interest • Consensus appears to be emerging that for qualitative researchers generalizability is best thought of as a matter of the “fit” between the situation studied and others to which one might be interested in applying the concepts and conclusions of that study.

25. Enhancing External Validity • Thick descriptions are crucial. • Such descriptions of both the site in which the studies are conducted and of the site to which one wishes to generalize (or apply one’s findings) are critical in allowing one to search for the similarities and differences between the situations. • Analysis of these similarities and differences makes it possible to make a reasoned judgment about the extent to which we can use the findings from one study as a working hypothesis about what might occur in another situation.

26. Multi-site Studies: Another Way to Enhance Generalizability • A finding emerging repeatedly in the study of numerous sites would appear to be more likely to be a good working hypothesis about some as yet unstudied site than a finding emerging from just one or two sites. • A finding emerging from the study of several very heterogeneous sites would be more robust and, thus, more likely to be useful in understanding various other sites than one emerging from the study of several very similar sites. • Heterogeneity may be obtained by creating a sampling frame that maximizes the variation inherent in the sample, specifically in terms of potentially theoretically important dimensions

27. Section 3 Implications: Enhancing the Reliability of Qualitative Research • Defining reliability • Key strategies for enhancing the reliability of qualitative research

28. Reliability • Reliability concerns the ability of different researchers to make the same observations of a given phenomenon if and when the observation is conducted using the same method(s) and procedure(s)

29. Enhancing the Reliability of Qualitative Research • Researchers can enhance the reliability of their qualitative research by: • Standardizing data collection techniques and protocols • Again, documenting, documenting, documenting (e.g., time day and place observations made) • Inter-rater reliability (a consideration during the analysis phase of the research process)

30. Section 4 Applications of Qualitative Research Design • Core Qualitative Methods • Guiding Principles • Qualitative Research Techniques

31. Core Qualitative Methods • Semi- or Un-structured, Open-Ended: • In-depth Interviews (in the field, face-to-face) • Participant Observation (field/site visits) • Archival Research (document review and analysis)

32. Guiding Principles • Qualitative research designs consider ways to foster: • Reflexivity(an ongoing process of reflecting on the researcher’s subjective experience, ways to broaden and enhance this source of knowing, & examining how it informs research) • Iteration(a spiraling process: sequential and repetitive steps in examining preliminary findings for the purposes of guiding additional data collection and analysis) • Intersubjectivity(a process of reaching a shared/ “objective” agreement about how to assign meaning to a social experience - with insiders and outsiders)

33. The Iterative Process of Qualitative Research: A Model Analysis Data Collection Reflection

34. Qualitative Research Techniques • Instrumentation: • Key Informants (question development and piloting of instrument) • Unstructured to Semi-structured • Probing • Data Processing and Analytic Tools

35. Qualitative Research Techniques/ Considerations • Sampling • Single v. Multiple Cases(not an individual) • Expert and Key Informants (identification and recruitment of sample) • Roles of the Researcher (identification and recruitment of sample)

36. Qualitative Research Techniques • Data Collection • Participants as Data Collectors • Field Notes (personal reflections, observations, emerging concepts/theories) • Debriefing (a participant, a participating researcher, a non-participating researcher)

37. Qualitative Research Techniques • Analysis • Key Informant Feedback • Codebooks(specifies definitions and relationships of concepts and terms) • Memos(emerging patterns, concepts; documentation of analytic pathways) • Case Analysis Meeting(a meeting of a research team for the purposes of reflecting on analytic process, tools, and findings) • Matrices or Diagrams(to identify and examine time sequencing, the structure of relationships, conditions of cross case events)

38. Section 5 Group Work to Apply Learning

39. What is a given program achieving with homeless and runaway youth? • Key Methodological Issues (instrumentation, sampling, data collection, analysis) • What More Do You Need to Know? • Initial Methodological Approach and Justification

40. In-House Qualitative Resources • Handouts, OMNI Reports and Proposals • Qualitative Research Design, An Interactive Approach(Maxwell, 1996) Sage, Applied Social Research Methods Series • Qualitative Data Analysis(Miles & Huberman, 1994) • The Quality of Qualitative Research(Seale, 1999) • Focus Groups, Theory and Practice(Stewart & Shamdasani, 1990) Sage, Applied Social Research Methods Series • The Mismeasure of Man (Gould)