Ensuring Validity in Qualitative Data Studies

1. Validity of Qualitative Data Carolyn Seaman UMBC Baltimore, USA Presented to the International Software Engineering Network (ISERN) 21 August 2001 Glasgow, Scotland

2. What is Qualitative Data? • Data in the form of words and pictures, not numbers • Data is richer and carries more information, but is harder to analyze than quantitative data • Can be either objective or subjective • Methods designed to elicit perceptions, feelings, and opinions

3. Advantages • Richer, more explanatory results • Terminology of results closer to “customer” • More participatory research process • Avoid errors in interpretation • Opportunity to clarify and explain findings • Many important questions simply can’t be answered quantitatively

4. Data Collection Prior Ethnography Participant Observation Interviewing Surveys Document Analysis Data Analysis Coding Constant Comparison Method Cross-case analysis Member checking Auditing Overview of Techniques

5. Are Qualitative Studies Valid? • Stupid question • Validity concerns are no more (and no less) present in qualitative studies, but they are different • It’s not harder to do a valid qualitative study, it’s just harder to show validity convincingly

6. Ensuring Validity • Validity of data • Is the data accurate? • Triangulation • Recording (audio, video, scribing) • Instrument bias • Political motivations

7. Ensuring Validity (cont.) • Validity of analysis • “Weight of evidence” - how much is enough? • Variety as well as quantity of evidence • Contradictory evidence • emerging theory must be modified in the presence of evidence that contradicts the current version of the theory • all contradictory evidence must be accounted for • Outliers, extreme cases, and surprises are good!

8. Showing Validity • Documentation • all steps in process must be documented - keep a log • auditing • transparency • keep track of sources for each data item • document exploration of all rival explanations

9. Showing Validity (cont.) • Dissemination (papers, talks, etc.) • make validity concerns explicit • spell out all procedures to ensure validity (triangulation, recording, etc.) • use quotes from data to illustrate conclusions • explain how contradictory evidence was handled • quantify where possible and appropriate • rival explanations

10. Miles & Huberman:Standards for the Quality of Conclusions • Objectivity - treatment of contradictory evidence and rival explanations; documentation of methods • Reliability - competence of researchers; quality of methods • Internal validity - logical sense, triangulation, rival explanations • External validity - representativeness; documentation of context • Utilization - relevance

11. Appropriateness When are qualitative methods appropriate? • Subject of study involves human behavior • No concrete or easily stated hypotheses • Variables hard to define or quantify • Little previous work • Quantitative results may be hard to interpret • When there’s probably not a yes/no answer

12. Conclusions • Qualitative data is highly useful in many types of studies • The validity of qualitative results is as variable as that of quantitative results • One can take steps to ensure the validity of qualitative data and analysis methods • Harder to show that one’s qualitative work is valid