Content

1. Content 1. The research context 2. Cognitive interviewing – definition, features, techniques, assumptions 3. Case study: our approach – processes used, examples of probing 4. Outcomes and drawbacks 5. Conclusions

2. 2. Cognitive interviewing “a process used to study the manner in which targeting audiences understand, mentally process and respond to the materials that we present’ (Willis 2005, p. 3)

3. Assumptions • That the way in which a particular question is designed will have an impact on the cognitive processes used by a respondent to develop an answer • That we are able to examine the processing undertaken by a respondent to answer a particular question as a way of tapping sub-processes such as comprehension, retrieval of information • That information we receive from the respondent is useful in that it provides a basis for judging whether the question has been processed as intended

4. Features • Focus - detect problems with questions • Timing - after drafting, before piloting • Conduct - by trained interviewers • Volunteers - representative • Processes - ‘verbal probing’ , ‘think aloud’ • Environment - usually volunteer’s choice • Sample size - small numbers

5. Techniques Verbal probing: … [asking] target questions… the subject answers it, but the interviewer then follows up (either immediately or at the end of the interview) by probing for other specific information relevant to the question or to the specific answers given (Willis 2005, p.47) ‘Think aloud’: interviewer reading aloud each question and then recording verbatim the subject’s verbalisation of their thinking as they respond (Willis 2005, p.43)

6. 3. Case study: our approach • NVETRE project on private providers • Telephone interviewing a national sample • Initial drafts – letter + instrument • Feedback from significant others • C/I protocol developed – verbal probes • C/I conducted, 3 providers, just over hour, focus on cognitive processes used rather than results received

7. Examples of probing (1) • understanding of key terms Q2: ‘What does the term “non-government sources” mean to you?’ Q7: ‘What do you understand by the term “higher education institution”?’ • how easy or difficult they found particular questions Q3: ‘Is it hard to think of the main reason?’ followed by ‘How much have you thought about the reasons why your organisation is registered to provide nationally accredited training?’ Q5 & 6: ‘How difficult would it be for you to answer these questions?’

8. Examples of probing (2) • certainty of their answers Q8: ‘In your own words, what is this question asking?’ followed by ‘How sure would you be of your answer?’ • ways in which they came to their answer Q16: ‘How would you get the answer to this question?’ followed by ‘How sure would you be of your answer?’

9. 4. Outcomes and drawbacks Outcomes: • Was an invaluable process – questions: 4 discarded, 9 revised, 5 had no modifications • Gave insights into ways words/concepts can be understood by respondents • Provided researchers with different ways of expressing questions (‘ to step into their shoes’) • Required close consideration of many different aspects of each question • Invited careful scrutiny of the exact nature of problems

10. Outcomes and drawbacks Drawbacks: • Time-consuming • Potential to add to development costs • Significant efforts in training interviewers • Finding volunteers to take part in this process • Not always easy/possible to reconcile differing perspectives • Issue re number of interviews to conduct

11. 5. Conclusions • Time taken to complete interviews in main study reduced as far as possible (thus containing costs) • Few reported mis-understandings • Adding definitions/clarifications provided interviewers with help if needed • Evidence for SCH: permission to continue with pilot study, then main study