Analysing and interpreting cognitive interview data: a qualitative approach

1. Analysing and interpreting cognitive interview data: a qualitative approach

2. Presentation structure • background • aims and objective of QT • design of cognitive interviews • review of methods of analysis • NatCen approach • issues for discussion

3. Background • aims and objectives of QT • does test question meet measurement objectives • if not, what problems arise • implications of problems for survey • design of cognitive interviews • probe sheet • interviewing techniques • sampling strategy

4. Review of methods of analysis • little written on analysis of cognitive interview data • two main methods cited: • standardised coding scheme • qualitative analysis • standardised coding scheme approaches are documented (usually) • qualitative analysis descriptions are often scant on detail

5. Standardised coding schemes • can be used to as stand alone question appraisal tool (e.g. QAS) • can incorporate some elements of behaviour coding • interviewer has a problem reading the question or recording the answer • focus on cognitive Q & A model • comprehension/communication • recall/computation • bias/sensitivity (judgement issues) • response category • plus ‘logical issues’

6. PROS lend themselves to presenting data quantitatively (x Rs had this problem) perceived (by some) to be more robust process is replicable useful in cross-national/ cross cultural settings, where standardisation important CONS need a lot of detailed codes under each main heading (particularly comprehension) time consuming loose context: why did R interpret Q in that way? lend themselves to presenting data quantitatively (x Rs had this problem) Issues with standardised schemes

7. Qualitative approach • “Just naming and classify what is out there is usually not enough. We need to understand the patterns, the recurrences, the whys. As Kaplan (1964) remarks, the bedrock of inquiry is the researcher’s quest for ‘repeatable regularities’.” • Miles & Huberman (1994)

8. Approaches to qualitative analysis • Ethnographic accounts • detailed ‘thick’ descriptions of cultures or organisations • Life histories • analysed as individual cases or mined for common themes • Content analysis • identifies content and context of documents, often involves counting (not strictly qual) • Grounded theory • generates analytical categories and the links between them through an iterative process of collecting and analysing data

9. Approaches to qualitative analysis • Narrative analysis • examines how a story is told and the intention of the teller • Conversation analysis • examines the structure of (usually) naturally occurring conversations • Discourse analysis • focuses on how knowledge is produced through the use of language • Interpretative analysis • attempts to present and re-present the world of those studied, by identifying and describing substantive themes, and searching for patterns between them

10. Key stages of the analytical process • Generation of findings • describing • classifying • finding linkages and patterns • identifying explanations • Data management • identifying themes • sorting and reducing data • Characteristicsof ‘good’ analysis system • Remains grounded in the data • Transparent data reduction process • Facilitates and displays ordering • Permits within and between case analysis

11. The analytical hierarchy in qualitative research Explanatory analysis Seeking wider applications Developing explanations Detecting patterns of association Establishing typologies Descriptive analysis Identifying elements & dimensions ? Summarising / synthesising data Data management Sorting data Tagging data Identifying initial concepts / themes Data collection Primary data

12. Thematic analysis - purposes and principles • Structured display of data by theme (Q) across all cases • Creating categories and classifying data within them* • Demonstrating range and diversity • Using examples to illustrate and amplify • Must be comprehensive • Labelling and categorising must be valid

13. Carrying out thematic analysis - NatCen approach • Familiarisation with data • Identification of factors • highlight, summarise, provisionally label • Categorisation • is this a different manifestation of that • is this a subset of that • is this of the same order as that • Iterative process of refinement • Start close to the data - become more abstract and interpretative • Must be comprehensive • Aim is analytical coherence

14. Looking for explanations • Informed by: • hunches and hypotheses • reflections during fieldwork and analysis • other research or theories • Process involves: • detailed within case analysis • comparison between cases • repeated interrogation of the data • milking data • moving back and forth between cases • searching for rival explanations • Expect multiplicity • Must be comprehensive

15. Summary of NatCen approach • Detailed notes made on interviews • Notes reviewed • Chart set up • Notes charted (chart revised) • Charts reviewed • Data interpreted • Findings emerge • Recommendations made • Report written

16. Example chart

17. Issues to consider • better documentation of qualitative analysis approach • integration of code frames within thematic approach • development of best practice for analysis of cognitive data • cognitive interview data as one component of testing strategy • collaborate findings using other data sources (e.g. split ballot experiments)