Credibility and Validity in Qualitative Research

1. INF5220 - 8 Lecture 12th of January 2006

2. Silverman on quality • Aim of this course is to instill in you a ”methodological awareness” • Question: Should we just trust qualitative researchers? (eg. based on their ”political credentials” and demonstration of ”involvement”) • Or is there a basic, shared attempt (within the research field) to generate credible and valid knowledge? • Credibility: the extent to which any research claim has been shown to be based on evidence • We will discuss: • Validity • Reliability • Generalisation

3. Validity and reliability • Validity has to do with what we might call truthfulness • Reliability has to do with consistency of ’measurements’ • Example: repeated readings of two termometers in boiling water (100 ºC): • A is reliable, but gives invalid results • B is unreliable, but gives relatively valid results • Are your procedures realiable, are your conclusions valid?

4. Validity (1) • Is the account true? • Are the deviant or contrary cases excluded? • Is a ’glossy picture’ presented? • The common problem of ’anecdotalism’: • that a few exemplary instances are offered, without reasons for selecting them, or a discussion of the typicality or representativeness of them. • Claims to validity should be based on attempts at refutation • Avoid jumping to conclusions just because there is some evidence that seem to lead in an interesting direction. Make an effort to falsify your initial assumptions about the data

5. Validity (2) • Often advocated strategies (but flawed in S’ view): • Triangulation (e.g. combination of different methods that give you different but complementary data) • Remember that results from methods are situated, context-bound • Respondent validation (take your accounts and interpretations back to the subjects) • But do they have a privileged epistomological status? • Silverman’s advice: • The refutability principle • Constant comparative method • Comprehensive data treatment • Deviant case analysis • Using appropriate tabulation

6. Reliability (1) • ”Reliability refers to the degree of consistency with which instances are assigned to the same category by different observers or by the same observer on different occasions” (Hammersley, 1992; 87) • Consistency can be along several dimensions: between different observers, between the results of different data collection methods, across different cases, over time, etc.

7. Reliability (2) • Important to document the procedure. How do you present your data? • Use low-inference descriptors, e.g. verbatim accounts of what people said, extracts from field notes. • On the contrary, ”high-inference descriptors” may be your ’polished’ account of what you see in your material, your interpretations. • Give details on the relevant context of observations and how you recorded and handled the notes. • Have you standardised your data collection/construction to some degree? Describe your field-note conventions (if you used standardised templates for data collection, your research group’s practices for sharing and discussion of analytic memos, etc.)

8. Generalisability (1) • Quantitative research: sampling is about selecting a representative subsection of a population. A good choice helps establish the representativeness and thus the generalisability. • This is not so straightforward in Qualitative research, but we may think about representativeness in several ways: • Complement qualitative studies with quantitative measures • Purposive sampling: think critically about the parameters of the population before you select case • Theoretical sampling: your attempt is not to generalise to populations, but you select samples based on their potential to offer interesting theoretical insights • Usually: a single case, chosed for ’convenience reasons’ (e.g. access)

9. Generalisability (2) • The validity of inferences drawn from on e of more cases does not depend on the representativeness of the cases in a statistical sense, ”but on the plausibility and cogency of the logical reasoning used in describing the results from the cases, and in drawing conclusions from them” (Klein and Myers, citing Walsham) • Four types of generalisations (Walsham, 1995): • The development of concepts • The generation of theory • The drawing of specific implications • The contribution of rich insights

10. What is high quality in research? • Silverman’s criteria (see tables 15.1, 15.2, 15.3). To what degree does it match the list of criteria that I gave you for the Master thesis evaluation? • Silverman suggest four basic criteria: • build useful theories, • be self-critical, • selecting appropriate research methods, • make a practical contribution.

11. Klein and Myers • Criteria for conducting and evaluating interpretive research • Based on the perspective of hermeneutics • Seven principles (page 72): • The fundamental principle of the hermeneutic circle • The principle of contextualisation • The principle of interaction between the researcher and the subjects • The principle of abstraction and generalization • The principle of dialogical reasoning • The principle of multiple interpretations • The principle of suspicion • http://www.misq.org/archivist/bestpaper/misq99.pdf

12. Some other methodological concepts you may encounter: • Quantitative and/or positivist research: • Dependent (’outcome’ variable, what you measure) and independent variables (’factors’, what you manipulate or change) • External validity: related to generalizations (to which degree would your conclusions hold for other people in other places, at other times?) • Internal validity (for studies of causal relationships). How well have you managed to ’separate out’ and distinguish the effects among the variables? Are there other possible causes (explanations) for your observations that you have forgotten or neglected?

13. The handouts from last week • Examples and templates that you may use (modify if needed) • Template for field notes • A description of how to create analytic memos • Template for research proposal • Master thesis structure • Fieldwork report (5-10 pages): • Content: • ’Facts’: where did you go, what time, whom did you see? • What happened? Describe what you saw in your observations, what was being discussed in the interviews etc. (the findings) • Sum up: What did you learn (that you did not know before)? • Reflect on what implications may this have for further work? (does it help you towards formulating a more focused research question? Do you know more about what you do NOT want to do?) • Not analysis, and/or use of theory • Due by March 1st (or negotiable).

14. Other issues • How to get help while writing your thesis (e.g. to locate relevant literature) • Use readings from previous courses, and additional literature • Ask faculty to suggest relevant literature (both supervisor and others) • How to plan in order to succeed: • Serious thinking and literature survey now • Serious work with proposal during the spring • Start serious writing by Dec 1st)

15. Topic for discussion: • What does it mean to be scientific? • One answer: adopting methods of study that are appropriate to the subject matter • From Petter’s email: • Theory as maps? • Is sociological/anthropological knowledge useful in IS? • About hypotheses