Dr. Dave O’Brien, Leeds Metropolitan University and ESRC/AHRC Placement Fellow at DCMS

1. Dr. Dave O’Brien, Leeds Metropolitan University and ESRC/AHRC Placement Fellow at DCMS Cultural policy and measurement: context, challenge and change

2. Overview • Evaluation in the cultural sector • How central government thinks about evaluation • The Green Book • Essential questions for today

3. General problems in cultural policy • Data collection, statistics and evidence are a longstanding issue in the cultural sector (Selwood 2002, 2010) • Reflects the problems of defining culture (e.g. Gray 2006, Miles and Sullivan forthcoming) • And defining what cultural policy is for

4. What is cultural investment for? 'Mozart is Mozart because of his music and not because he created a tourist industry in Salzburg..... Picasso is important because he taught a century new ways of looking at objects and not because his painting in the Bilbao Guggenheim Museum are regenerating an otherwise derelict northern Spanish port...... Absolute quality is paramount in attempting a valuation of the arts; all other factors are interesting, useful but secondary.‘ Tusa 1999 cited in Reeves 2002:36

5. .....And can cultural programmes be evaluated? ‘The concept of the arts itself is indefinable, and any attempt to measure it cannot begin to represent its essential quality’ (Missel 1983, cited in Hewison 2002:85)

6. Understanding impact • Narratives of the value of DCMS’ sectors e.g. Arts Council England’s Public Value Programme • Economic impact of DCMS’ sectors e.g. Impacts08’s work • Social Impact e.g. MLA’s Generic Social Outcome indicators • Educational Impact e.g. MLA’s Generic Learning Outcome indicators • Intrinsic Impact work e.g. Nesta (2010), Brown & Novak (2007)

7. However.... ‘the sector is hindered by its failure to clearly articulate its value in a cohesive and meaningful way, as well as by its neglect of the compelling need to establish a system for collecting evidence around a set of agreed indicators that substantiate value claims’ Scott (2009:198)

8. What is policy evaluation? • Policy evaluation uses a range of research methods to systematically investigate the effectiveness of policy interventions, implementation and processes, and to determine their merit, worth, or value in terms of improving the social and economic conditions of different stakeholders. Policy evaluation uses quantitative and qualitative methods, experimental and non-experimental designs, descriptive and experiential methods, theory based approaches, research synthesis methods, and economic evaluation methods. Policy evaluation for government privileges no single method of inquiry and acknowledges the complementary potential of different research methods. (Magenta Book http://www.nationalschool.gov.uk/policyhub/evaluating_policy/magenta_book/chapter1.asp)

9. And why are we interested in it? • “To be effective policy making must be a learning process which involves finding out from experience what works and what does not and making sure that others can learn from it too. This means that new policies must have evaluation of their effectiveness built into them from the start; established policies must be reviewed regularly to ensure that they are still delivering the desired outcome; and the lessons learned from evaluation must be available and accessible to other policy makers.” (Cabinet Office 2000)

10. Evaluation should...... • Be systematic • Be analytical • Study actual effects • Judge success (Cabinet Office 2000)

11. Evaluation should aim to....... • Improve decision making • Help resource allocation • Enhance accountability • Bring organisational learning (Cabinet Office 2000)

12. Judging success, studying actual effects and helping resource allocation? ‘It is difficult to decide whether Shakespeare’s Hamlet is better than his King Lear and even harder to persuade others of our decision or define what such a ranking would mean. How many Gershwin songs sum up to a Shostakovich symphony? Is a Haydn string quartet better than a Hemmingway short story? How does a Blake poem compare to a modern ballet performance?’ Cowen (2006:6)

13. HMT’s understanding of value • Green Book • Cost Benefit Analysis • Net Present Value (NPV): value of costs of a policy minus value of benefits, discounted over time • Recommends economic techniques to value ‘non-market’ goods • Works well for some forms of government policy, but not for others • We know the costs of culture- how do we value the benefits?

14. Concluding Questions • How to we close the gap between the Green Book and the Cultural Sector? • At a time of funding cuts, how do DCMS’ sectors best demonstrate: ‘their merit, worth, or value in terms of improving the social and economic conditions of different stakeholders?’