Developing a framework for evaluating qualitative research

1. Developing a framework for evaluating qualitative research Liz Spencer, Jane Ritchie, Jane Lewis, Lucy Dillon NatCen Team 24 June 2004

2. The context • Within the Cabinet Office • responsibility for encouraging ‘excellence in government research and evaluation’ • commitment to the ‘contribution of research and research synthesis to evidence-based policy and practice’ • concern about lack of agreed standards for ‘what constitutes high quality qualitative methods of policy evaluation’ Liz Spencer, New perspectives, 2004 (member of the NatCen team)

3. The brief: aims and objectives • identify a set of standards / produce a provisional set of criteria against which qualitative policy evaluationstudies can be critically appraised • to help determine whether particular qualitative studies / studies reported in the literature meet the agreed standards of validity, reliability, relevance, and robustness to be included in the evidence base for effective policy making • produce guidance on how the standards / criteria can be applied in practice Liz Spencer, New perspectives, 2004 (member of the NatCen team)

4. The brief: suggested approach • literature review (debates about qualitative methods and criteria, other frameworks / sets of guidelines, research reports) • interviews with academics, research practitioners, funders, commissioners and users / policy makers • NatCen proposed • a workshop • trial application Liz Spencer, New perspectives, 2004 (member of the NatCen team)

5. Issues arising from the brief • the meaning / scope of ‘policy evaluation studies’ • the notion of standards / criteria in qualitative research • the call for application guidelines • the assumption that reliability, validity, relevance and robustness are key considerations / components • the relationship between evidence and policy making • epistemological assumptions and implications Liz Spencer, New perspectives, 2004 (member of the NatCen team)

6. ‘Qualitative policy evaluation’ • evaluation versus research? • different aims, timescales, and ways of assessing • particular contribution of qualitative approaches to evaluation • adopted inclusive interpretation of policy evaluation • policy research, studies of practice, assessment of interventions/ initiatives/programmes • background/context setting, development, implementation, outcomes • broadened scope to include ‘empirical’ qualitative research Liz Spencer, New perspectives, 2004 (member of the NatCen team)

7. Evaluation traditions • how to address different aims of evaluation? • generation of information to aid decision making • participation • enlightenment • reform • emancipation • focussed on evaluation which utilises qualitative research methods, where aim is to produce defensible knowledge claims and quality of research still matters Liz Spencer, New perspectives, 2004 (member of the NatCen team)

8. The relationship between evidence and policy • tension between evidence-based practice/policy and qualitative research based on questionable assumptions that • there is a hierarchy of methods • qualitative research is a last resort / soft option • explicit procedures are superior to informed judgements • aggregation and synthesis are superior to mapping • policy can be forged from ‘brute facts’ / evidence gives the answer • reinforced decision to to concentrate on quality of qualitative findings (issues of timeliness, feasibility, and political will left to policy makers) Liz Spencer, New perspectives, 2004 (member of the NatCen team)

9. The whole idea of qualitative standards or criteria • Many different ‘positions’ • rejection of criteria for philosophical or methodological reasons • proposal of ‘alternative’ criteria (unrelated to notions of rigour or credibility) • proposal of ‘parallel’ criteria (addressing notions of rigour or credibility) • adoption of traditional ‘scientific’ criteria (to be applied rather differently) Liz Spencer, New perspectives, 2004 (member of the NatCen team)

10. The idea of criteria (contd.) • concern about rigid checklists • concern about ‘tick box’ mentality • avoided the term ‘criteria’ • adopted series of flexible open-ended questions around guiding principles and quality issues • retained centrality of experience and judgement Liz Spencer, New perspectives, 2004 (member of the NatCen team)

11. Addressing philosophical debates • philosophical assumptions crucial to acceptance/rejection of ‘criteria’ and to bases of quality assessment • confusing/confrontational labelling of positions • adopted ‘elemental’ approach for scope of framework • ‘reality’ mediated through human constructions • shared meanings • neutrality as guiding ideal • reflexive practice • no methodological hierarchy • flexible but rigorous conduct Liz Spencer, New perspectives, 2004 (member of the NatCen team)

12. Addressing the ‘holy trinity’ • no escape from ‘validity’, ‘reliability’ and ‘objectivity’ • identified underlying themes: • internal validity (procedural/methodological;interpretive / accuracy or credibility of findings; relational / outcomes in relation to participants) • external validity (relevance; generalisability; auditability; contextual detail) • reliability (replication; consistency; auditability) • objectivity (neutral/value free; auditability; reflexivity) • soundness / well-foundedness vs goodness / worthwhileness Liz Spencer, New perspectives, 2004 (member of the NatCen team)

13. Identification of some underlying central concerns and principles • defensibility of approach • rigour of conduct (research practice and relationship to those being researched / evaluated) • credibility of claims • contribution and wider impact Liz Spencer, New perspectives, 2004 (member of the NatCen team)

14. Determining content • series of readings on central concerns and principles • relating to all stages of inquiry • reflecting recurrent themes within literature and amongst interviewees • reflecting context of the commission • reflecting locus, experience and skills of the research team Liz Spencer, New perspectives, 2004 (member of the NatCen team)

15. The structure of the framework • Three tiers: • 4 central principles • 18 appraisal questions (indicative, discretionary, and avoiding yes/no answers, no scoring) • series of quality indicators (illustrative rather than exhaustive or prescriptive, no scoring) Liz Spencer, New perspectives, 2004 (member of the NatCen team)

16. Guidance on application • framework intended to be used flexibly • discretion and judgement remain central • importance of further trials / assessment • contribution to an evolving process Liz Spencer, New perspectives, 2004 (member of the NatCen team)

17. Interpretation of the brief • flexible response achieved • by stressing Cabinet Office objectives for framework to have wider credibility / utility • by analysing recurrent themes in the literature • by reviewing other sets of guidelines / frameworks • by reflecting views of participants (during interviews and at workshop) • because of the National Centre’s role and position Liz Spencer, New perspectives, 2004 (member of the NatCen team)

18. Full copy of the framework and accompanying report can be found at:www.policyhub.gov.uk Liz Spencer, New perspectives, 2004 (member of the NatCen team)