2017 Vanderbilt Conference ” Questions of Validity and Reliability for Qualitative Research ”

1. Scott D. Churchill Ph.D.University of DallasPeabody College, Vanderbilt UniversityMarch 24, 2017 2017 Vanderbilt Conference   ”Questions of Validity and Reliability for Qualitative Research” Lessons Learned from Quantitative Methodology

2. Overview of Issues to be Addressed • “Internal Validity” • (reframed for qualitative studies) • A fidelity to the targeted experiences • Avoiding Researcher Bias • “External Validity” • Crossing the threshold from the idiographic to nomothetic • “Reliability” • Finding convergences and divergences • among different researcher’s perspectives

3. Internal Validity Redefined for Qualitative Research: • “Finding our way to the evidence in a way that reflects a fidelity to the targeted experience at the individual level” • So, the question of internal validity – the validity of our findings with regard to the sample studied – has two components: • first, finding participants whose experiences actually represent the experience under investigation; • and secondly, developing a method of reflection that results in findings that correspond faithfully to the experiences reported.

4. (A) The Validity of Qualitative Evidence • To achieve a genuinely “phenomenological” understanding, the aim would be for the phenomenologist to allow the phenomenon to appear concretely in its own self-givenness…. • ... and then to proceed to reflect upon and describe its appearance as best one can without the mediation of theories or hypotheses. • To avoid the charge of subjectivism – where researchers base their “findings” on their own private experiences, the founders of phenomenological method in psychology decided to “outsource” the original description of lived experience.

5. When we ask patients or research subjects to describe their experiences, they do so with many of the same operative defenses and self-deceptions that were inherent in the experiences they described. • Thus if empirically-based phenomenological research is going to aim to achieve fidelity with respect to an original experience using narrative data and narrative methods of analysis, then out of concern for the validity of narration one must consider the possibility of "distortion" in the reflexive movement from the data to the findings. • In practice there will always be a precarious relationship between an experience and its description, for a multitude of reasons.

6. Averill (1983) has proposed one understanding of why verbal reports (protocols) have fallen under suspicion and "searching criticism" by empirical psychologists: • What a person says is under conscious, voluntary control, and hence is subject to dissimulation and conformity to social expectations. And even when a person is not dissimulating in an attempt to present a positive self-image, she or he may not have the ability to report accurately what is taking place internally (p. 1154). • The latter charge was the substance of Nisbett & Wilson's (1977) oft-cited critique of verbal reports as data. As they see it, self-reports amount to "telling more than we can know."

7. If verbal reports do not accurately reveal the cognitive mechanisms underlying human experience, this is not to say that they are not revelatory of the meaning or "intentionality" of human experience. Moreover, we will see that to the extent that verbal explanations are themselves human behaviors, they constitute a legitimate realm of investigation in their own right

8. (B) The Question of Validity in Regards to the Researcher’s Process • The participant’s description functions as a medium…. through which, as meanings of the subject's experience begin to resonate within the researcher's own experience, the researcher gains access to the world of the subject…. and at the same time grasps this world as a function of the subject's presence, or intentionality. • “Intentionality” is the central phenomenon of interest to the phenomenological psychologist

9. The phenomenological approach requires that the researcher enter into direct, personal, living contact with the psychological event being studied. • The words of the research subject open up a world of experience, and it is this world that the researcher directly and vividly intuits through the self-report data. • The task of the researcher is essentially to imagine the subject’s intentional relationship to his or her world of experience through empathy

10. Empathie, seule attitude requise pour comprendre. -- J-P Sartre, L’Idiot de la familie

11. II. External Validity: The Generalizability of Qualitative Findings If “empathy” is used to “open up” the meaning-horizons of qualitative data at the individual level…. “free variation in the imagination” is utilized to attain external reliability so that findings at the individual level can begin to be generalized to wider contexts

12. The move towards universal structures of conscious experiencing requires that the phenomenologist shift the attention away from the individual experience grasped for its own sake, and toward the ‘category’ or class of experiences of which the individual experience is now taken as merely an example. • In quantitative psychology, this would be the move from the sample to the population. • For the phenomenologist, it would be from the 'instance' to the 'category' – from “this” experience of anger, to the category of “all experiences of anger.”

13. The Question of Reliability • "Reliability refers to the consistency of a measuring procedure or instrument. • A method of measurement is reliable if it always produces the same result under the same conditions" (Lewin, 1979). • Reliability thus assumes that one can establish an equivalence of measurement. • In the modern sciences, "measurement" has come to mean quantification according to an established standard or scale.

14. “Experiential Methodology” The original meaning of the term “measurement” was “description” When qualitative researchers conduct research, their “descriptive findings” consist of their own verbalizations of their experience of the participants’ own descriptions of their experience. hence reference to “experiential method”, which refers both to the experience described by our informants, and to the researcher’s empathic experience of their participant’s experience.

15. Within my own situation, that of the [other] whom I am questioning makes its appearance and, in this bipolar phenomenon, I learn to know both myself and others. … it is not a question … of reducing his experiences to mine, or coinciding with him, or sticking to my own point of view, but of making explicit my experience, and also his experience as it is conveyed to me in my own … and to understand one through the other” (Merleau-Ponty, 1945/1962, p. 338).

16. So: how do we establish the “reliability” of a method that requires the researcher to place herself or himself into the position of the participant, by means of the other’s description of their experience? In a comparative study of several qualitative researchers analyzing the same data, what we noticed were “convergences” – as well as divergences – among the individual researcher’s thematizations. The points of convergence, in which researchers appeared to be making the same observation while using slightly different words, established a basis for considering this (phenomenological) method of observation as ‘reliable’. Divergences were reconciled by asking each member of our team of researchers if they could “see” what the other researchers were seeing, by putting themselves into the position of the other researcher’s standpoint. Each qualitative researcher brings his or her own unique perspectives to bear - and the question is thus not whether on first reading we all see the same thing.

17. Thus, the chief point to be remembered with this type of research is not so much whether another position could be adopted (this point is granted beforehand) but [rather] whether a reader, adopting the same viewpoints as [those] articulated by the researcher, can also see what the researcher saw, whether or not he/she agrees with it. That is the key criterion for qualitative research. (Giorgi, 1975, p. 96)

18. All research discloses only a limited truth, that is, a truth limited by the researcher’s procedures and perspective. Phenomenological researchers attempt to articulate those limits reflectively and honestly, and additional limits may be discerned by others whose scholarship and reflections bring additional perspectives and procedures to bear. The validity of research findings – as well as the reliability of our method - is therefore not contingent on whether they are entirely similar to those of other viewpoints.

19. According to the phenomenological approach, it is not possible to exhaustively know any phenomenon, and different viewpoints can be valid (Churchill, Lowery, McNally, & Rao, 1998; Wertz, 1986). In other words, other perspectives, perhaps rooted in different research interests, and their corresponding intuitions, always are possible and contribute in a complementary manner to our knowledge of “the whole.” In the end, the value of the findings depends on their ability to help others gain some insights into what has been lived unreflectively. Other insights from different viewpoints may then supplement, and thereby extend and possibly even radically decenter, what always is essentially a partial knowledge of human life.

20. To quote Jacques Derrida (1985, p. 4): (everything comes down to the ear you are able to hear me with.)