Can Qualitative Research Have Impact? Is It Even Evidence-Based?

1. Can Qualitative Research Have Impact? Is It Even Evidence-Based? Martyn Hammersley The Open University Seminar, London Medical Sociology Group, Kings College, London, October 2011

2. The Role of Sociology Sociologists have long been concerned with the social value of their work, though they have taken different views about their role. We can distinguish the following broad options, the sociologist as: • Technician; • Professional; • Critic; • Engaged activist.

3. Impact and Evidence-based Policymaking and Practice • Recent pressure for social science to maximise its impact: institutionalised in ESRC requirements, and in government demands that universities contribute directly to the national and local economies. • The notion of impact employed here is close to that championed by the evidence-based policymaking and practice movement. • From this point of view, the primary task of research is to demonstrate ‘what works’, in other words to evaluate the effectiveness of policies and practices, or to generate new, more effective ones.

4. The Concept of ‘Impact’ • One problem with this notion of impact, from the point of view of qualitative researchers, is that it privileges quantitative work designed to control variables and measure ‘effect size’. • In its classical form, the gold standard for research serving evidence-based medicine, and evidence-based practice more generally, was the randomised controlled trial. • Such research was seen as demonstrating ‘what works’, what does not, and what causes unacceptable side effects.

5. Liberalisation • In many quarters, over time, there has been a liberalisation of this classical model, as regards: • The relationship between research and practice; • The sorts of research that can inform policymaking and practice. • However, this shift has been untheorised and amounts to little more than a defensive strategy

6. What Does ‘Impact’ Mean? We need to think carefully about the term ‘impact’. It is a metaphor whose prototype is: one object hitting another and causing it to change position or trajectory; whether a mere ‘nudge’, to use a currently influential term, or a more dramatic transformation.

7. The Problems With Impact • There are problems with this metaphor: • The influence of research findings on policymakers and practitioners is not direct, it is mediated by many factors; • It is always an interpretative not a mechanical process; • We should not assume that high impact is always good. To do so is to adopt a naïve Enlightenment perspective.

8. Engineering Model The notion of impact corresponds with what has been called the ‘engineering model’ of the relationship between research and policy/practice (Janowitz, 1972; Bulmer 1982; Finch, 1986; Weiss 1977; Hammersley 2002). This assumes that: Social science can make the same dramatic contribution to practice as natural science. It can do this through producing theoretical or instrumental knowledge that is then 'applied to' or 'translated into’ policies or practices; and/or through social science methods being applied within policymaking and practice, rendering these more rational and effective.

9. The Enlightenment Model The usual contrast to the engineering model is the enlightenment model. There are different versions: What we might call strong enlightenment sees social science as providing a comprehensive theoretical perspective that facilitates the transformation of policymaking and practice; Weak enlightenment assumes that research shapes policymaking and practice in a range of indirect and piecemeal ways, percolating through social networks: it may throw doubt on current assumptions, suggest alternative perspectives, point to unknown consequences, etc.

10. Problems With ‘Strong Enlightenment’ • It implies that somebody is in the dark, and that only researchers have a light. But: • A comprehensive theoretical perspective is of questionable practical value. • There are other valuable sources of knowledge besides research. • There are serious doubts about the capacity of social science to produce all of the kinds of knowledge required to make good policy and practice.

11. Is Qualitative Research Evidence-based? • There are two ways in which qualitative research is faulted as regards its capacity to serve evidence-based policymaking and practice: • It frequently does not address the question of ‘what works?’ • It is alleged that it cannot provide the quality or reliability of knowledge required.

12. The Evidence-based Model of Research • What is involved here is a crude positivist conception of research in which findings are derived by means of determinate inference (logical or calculative) from observational givens. In short, research is to be carried out on the basis of explicit procedures. • However: • Neither medical research nor natural science approximates closely to this ideal in practice. • The philosophical rationale for this ideal has been comprehensively demolished over the past 50 years within the philosophy of science.

13. Problems With the Procedural Model • There are no observational givens, in the sense required: all observation is assumption-laden and disposition-relative. • Relying upon procedural rules may correct some errors but can also generate new error as well. • There are no logical or calculative means by which we can infer conclusions from data. The latter always have to be interpreted, and competing interpretations evaluated through informed judgment.

14. A Wider Concept of Impact There has been an interesting ambivalence about the demand for impact among qualitative researchers. Indeed, many wish their research to have impact, in the sense of challenging and changing policy/practice. As I noted at the beginning, they may even see themselves as social critics or activists; for instance, recent discussion of ‘public sociology’ (Burawoy 2005; Holmwood 2007; Brewer 2011) Of course, a positivist conception of research practice is rejected here; and perhaps also the notion of strong enlightenment, in favour some model of the ‘specific’ intellectual.

15. Problematic Assumptions About Social Science What is the relationship between research and policymaking/practice assumed here, if it is not either that of the engineering or strong enlightenment model? Is there not retained the same excessive estimate of the capabilities of social research that is characteristic of those two models? There are two aspects to this. It seems to be assumed that: • Social science can answer evaluative or value questions. • It can provide reliable factual evidence about all the questions that are relevant to key policy or practice issues.

16. Problematic Assumptions about the Wider Society 1 • These arguments about how sociology can have impact seem to assume that a deliberative democracy is in operation. But: • This is not the case (see Weiss 1983; Hammersley 2006); • There are major tensions between experts providing knowledge and many conceptions of deliberative democracy.

17. Problematic Assumptions about the Wider Society 2 • Where impact is conceptualised as being upon the individual consumer or client, two problematic assumptions are involved: • That this consumer is sufficiently informed and rational (in the appropriate sense) to make good use of any knowledge provided. • That sociological knowledge can be preserved in the relatively brief and simple messages that are required if impact of this kind is to take place (Hammersley 2006).

18. Conclusion • The notion of impact involves misconceptions about the nature of social research, and what it can offer policymaking/practice, and about the wider society. • This is true not just in relation to notions of research for evidence-based practice, but also as regards the idea of a public sociology. • The weak enlightenment model forces us to adopt a more modest role, and to try to defend this in the face of excessive external and internal demands for ‘impact’.

19. References • Burawoy, M. (2005) ‘For public sociology’, American Sociological Review, 70, pp4-28. • Brewer, J. ‘The new public university, 19th January 2011, ‘A new kind of public university and a new kind of social science’, 11th February 2011’, and ‘The New Public Social Sciences 2’ 25 March 2011. Available at (accessed 6.10.11): http://sociologyandthecuts.wordpress.com/category/john-brewer/ • Bulmer, M. (1982) The Uses of Social Research, London, Allen and Unwin. • Finch, J. (1986) Research and Policy: the uses of qualitative methods in social and educational research, Lewes, Falmer. Hammersley, M. (2002) Educational Research, Policymaking and Practice, Paul Chapman/Sage. • Hammersley, M. (2006) Media Bias in Reporting Social Research? The case of reviewing ethnic inequalities in education, London, Routledge, 2006. • Hammersley, M. (2011) Methodology, who needs it?, London, Sage. • Holmwood, J. (2007) ‘Sociology as Public Discourse and Professional Practice: A Critique of Michael Burawoy’, Sociological Theory, 25, 1, pp46-66 • Janowitz, M. (1972) Sociological Models and Social Policy, Morristown NJ, General Learning Systems. • Weiss, C. (1977) ‘Research for policy’s sake: the enlightenment function of social science research’, Policy Analysis, 3. • Weiss, C. (1983) ‘Ideology, interests, and information’, in D. Callahan and B. Jennings (eds.) Ethics, The Social Sciences, and Policy Analysis, New York, Plenum Press.