The British Experience of “evidence based policy making” 1997-2004

1. The British Experience of “evidence based policy making” 1997-2004 Ben Page Director MORI Social Research Institute info@mori.ie

2. What has changed • Serious investment in research and performance management capacity to gather and analyse hard data of all sorts • mandatory local and national survey research in local government and health services, 100s of PIs…… • Establishment of stronger centre to shift from policy to delivery…. • Office of Public Service Reform • Strategy Unit • Prime Minister’s Delivery Unit • Among others! • Major programmes of reform and investment in learning and skills, regeneration, education, NHS, reform and modernisation in local government • Evaluation of these programmes varies dramatically

3. What has changed • Some programmes have large scale and very costly evaluation programmes – more around investment in local communities eg • Sure Start • New Deal for Communities • Connexxions • LSPs etc • But others have none at all – simply implementation

4. Sure Start - A Case Study The Aim of Sure Start To work with parents-to-be, parents and children to promote the physical, intellectual and social development of babies and young children - particularly those who are disadvantaged - so that they can flourish at home and when they get to school, and thereby break the cycle of disadvantage for the current generation of young children.

5. By 2004, 524 Sure Start programmes will be helping up to 400,000 children living in disadvantaged areas, including a 1/3 of under 4s living in poverty. • Sure Start programmes will work with parents and parents-to-be to improve children's life chances through better access to: • Family support • Advice on nurturing • Health services • Early learning

6. LOCAL CONTEXT ANALYSISAIMS • Describe Sure Start Programme Areas – what and where are they? • Compare Programme Areas - with each other and nationally • Relate area characteristics to Implementation of Sure Start • Look at change over time in communities

7. The Digitisation of Sure Start Programme Boundaries • Step 1 - Paper maps are received from the Sure Start programmes.

8. The Digitisation of Sure Start Programme Boundaries • Step 2 - The boundaries are captured as a series of points on the National Grid. • The shape of the Sure Start programme area can then be analysed in GIS (Geographic Information Systems) software. A Sure Start programme area captured by GIS

9. The Digitisation of Sure Start Programme Boundaries • Step 4 - The postcodes become the common link to data from other sources, to profile each Sure Start programme area. Child Benefit Data Child Protection Register Data Birth Rates Data School Exclusions Data The Common Link

10. LOCAL CONTEXT ANALYSISWhat we are studying • Demographics and Poverty • Economic Profile and Labour Market • Crime, Danger and Disorder • Adult Health and Disability • Child Health and Development • School readiness and achievement • Parenting • Neighbourhood Networks • Access to services for 0-3 year olds and their families

11. IMPLEMENTATIONThe study incorporates quantitative and qualitative methods, and has three components: • A National Survey of 260 Sure Start Programmes • An In-Depth Study of 26 programmes • A Series of Themed Evaluations

12. IMPACTStage 1: 2001\2002 STAGE 1 100%OF SURE START COMMUNITIES • local context analysis information • proposed approach to Sure Start • implementation data

13. IMPACTStage 2: 2003/2004 Control groups allows us to identify effects of programme Stage 1 STAGE 2 • 150 SURE START COMMUNITIES • 50 CONTROL COMMUNITIES • cross-sectional study of 9 month, and 3 year olds and their families

14. IMPACTStage 3: 2005 and beyond Stage 1 Stage 2 STAGE 3 • 100 SURE START COMMUNITIES • 80 families per community • longitudinal study of 3 and 5 year olds

15. Interim conclusions so far!! • “Limited evidence that SSLPs may influence some aspects of parenting, i.e. acceptance” • “Stronger evidence that SOME specific SSLPs rather than SSLPs generally may be effective on a composite of child/parenting outcomes” • “The distinctive features of “effective” SSLPs may be more about community characteristics rather than programme characteristics but this needs further investigation…”

16. The rise and rise of performance management and inspection…….. • Huge numbers of inspections – but becoming clear that peer review and self assessment are as important to improvement as inspection/external review and targets • But unlikely that improvements would have happened based solely on self assessment – inspection and measurement has become accepted over the last seven years…… • There is a huge thirst for information • Increasing expectations of the “evidence base”, devolution of delivery to regional, local authority and sub-local authority level, having datafit for purpose

17. How much detail can you have, and how much can you cope with?

18. Other challenges in England….. • Practical limits for micro-management? • Changing agendas? • Some evaluations were set up for regimes that have now changed so fundamentally that it is very hard to isolate the impact of a particular initiative….. • So many initiatives, unclear what you are evaluating

19. More and more detailed understanding of multiple factors impacting on measurement • Similar investment and delivery standards across a service do not lead to uniform outcomes in perception or delivery • Impact of place, impact of nature of community, impact of media coverage, political belief and values are becoming more and more apparent…….

20. Satisfaction with Area v Deprivation Net satisfaction (+) Average Deprivation Score (IMD 2000) Base: MORI Norm Database

21. GCSE Results versus Deprivation Index GCSE Results - 5+ grade A-C Sutton Camden Hackney Portsmouth Deprivation Index

22. Satisfaction with council versus deprivation score Net satisfaction with Council (+%) Vale of White Horse Correlation -0.65 Sunderland Gateshead Camden Mid Beds Harlow Birmingham Brent Oldham Manchester Deprivation Score (IMD 2000) Base: All

23. MORI Frontiers Analysis Likely frontier of satisfaction Weak Average

24. Ethnic Diversity and overall PCT ratings Mid Devon PCT Airedale PCT Preston PCT South Birmingham PCT Lambeth PCT Overall PCT rating Thurrock PCT Barking & Dagenham PCT Greenwich PCT Newham PCT Haringey PCT Basildon PCT Bradford City PCT Ethnic diversity Source: PCT patient surveys 2003/MORI

25. Ethnic diversity and ratings of inpatient care Taunton & Somerset NHS Trust Airedale NHS Trust University Hospitals of Leicester Ratings of overall care Sandwell & West Birmingham Hospitals Medway NHS Trust West Middlesex University Hospital North Middlesex University Hospital Newham Healthcare Ethnic diversity Source: NHS Acute Trust inpatient surveys 2001-2/MORI

26. The Perception Gap in delivery: hard numbers alone or “belief” in progress?

27. Optimism about future improvements to public services is not clear despite billions of extra investment

28. Will the Government deliver on Public Services? Q Do you agree or disagree… “In the long term, this government’s policies will improve the state of England’s Public Services” Agree Disagree Mar 2004 June2004 June 2001 Oct 2001 Mar 2002 Nov 2001 May 2002 Sept 2002 Dec 2002 Mar 2003 Jun 2003 Sept 2003 Jul 2003 Dec 2003 Base: 966 British adults, 18-20 June 2004 Source: MORI Social Research Institute Delivery Index

29. But they have great expectations - often raised by politicians

30. Choose carefully what you promise....Huge cynicism to overcome

31. Priorities for improvement Q In your view, which TWO, if any, of the following changes would best indicate to you personally that the NHS was improving? Spring 2003 Increased numbers of doctors and nurses Waiting less than 4 hours in A&E Reduced waiting times for inpatient admissions Being able to see a local doctor/GP or practice nurse within 48 hours Reduced waiting times for outpatient appointments Being able to choose time and date and place of treatment Don’t know/refused Base: All - Spring 2003 (1,041)

32. England has lots more nurses….

33. Number of NHS doctors (GPs & consultants) in England, 1997-2003 65,000 61,400 60,000 58,252 56,467 54,653 55,000 53,308 Number of doctors 52,021 50,863 50,000 45,000 40,000 1997 1998 1999 2000 2001 2002 2003 Source : Department of Health … and doctors

34. Waiting lists ARE falling Number of inpatient waiters 300,000 2001/02 2002/03 2003/04 2004/05 2005 250,000 6+ month 200,000 150,000 100,000 9+ month 50,000 12+ month 0 A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D Source : Department of Health

35. Waiting lists ARE falling Number of inpatient waiters 300,000 2001/02 2002/03 2003/04 2004/05 2005 250,000 6+ month March 04 milestone 200,000 150,000 100,000 9+ month 50,000 12+ month 0 A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D Source : Department of Health

36. Respected independent organisations confirm that progress is being made “Waiting times are generally shorter than at any time in the history of the NHS.” King’s Fund Briefing Paper March 2004

37. But the public disagree...... % who think waiting times are getting... Longer Same Shorter Base: All -Spring 2003 (1,000) Winter 2002 (1,002), Spring 2002 (1,041) Winter 2001 (1,021) and Spring 2000 (1,046)

38. The Right Waiting Time Targets? Positive Negative Base: All respondents (2,037)

39. Respect & dignity +37% Clean wards/toilets +15% Pain control +13% Purpose of medicines explained +13% Ratings of overall inpatient care Communicated side effects +8% R2 = 0.94 +7% A&E organisation Privacy to discuss treatment +6% Key Drivers of Perceptions of Hospital Care – things to focus on Waiting is less important to patients than it is to the public

40. Most important drivers of satisfaction with the running of the NHS at the national level Greatest drivers ofoverall satisfaction Political belief/Support for the government Age Poor experiences of A&E services Staff bad-mouthing Weaker driversof overall satisfaction Media coverage Direct NHS communication Positive experiences of inpatient services

41. What are the issues the public see the NHS confronting? 1. Under-funded 33% 2. Bureaucracy 31% 3. Not enough doctors/nurses 27% 4. Waiting 19% 5. Cleanliness/superbugs/MRSA 16% 6. Overworked staff 12% 7. Pay 7% 8. Over-stretched/ageing population 5% 9. Staff training 5% 10. Staff retention 5% 11. Foreigners/immigrants 5% 12. Shortage of beds 4% 13. Quality of treatment/care 4% 14. Shortage of NHS dentists 3% Source: MORI Public omnibus (12 - 16 August 04)

42. Crime...

43. The aim is to reduce domestic burglary by 25% ... Domestic Burglary - 25% reduction from 1998-99 to 2005 British Crime Survey 1,400,000 1,200,000 1,261,364 1,000,000 946,023 800,000 Number of offences 600,000 400,000 200,000 - 2000 BCS 2001/02 2002/03 2003/04 2004/05 2005/06 (crime in 1999) interviews interviews interviews interviews interviews Source: Home Office

44. … and performance is ahead of trajectory Domestic Burglary - 25% reduction from 1998-99 to 2005 British Crime Survey 1,400,000 1,200,000 1,261,364 1,000,000 973,522 965,502 946,023 800,000 Number of offences 600,000 400,000 200,000 - 2000 BCS 2001/02 2002/03 2003/04 2004/05 2005/06 (crime in 1999) interviews interviews interviews interviews interviews Source: Home Office

45. The aim is to reduce vehicle crime by 30% ... Vehicle Crime - 30% reduction from 1998-99 to 2004 British Crime Survey 3,500,000 3,000,000 2,941,927 2,500,000 2,000,000 Number of offences 2,059,349 1,500,000 1,000,000 500,000 - 2000 BCS 2001/02 2002/03 2003/04 2004/05 (crime in 1999) interviews interviews interviews interviews Source: Home Office

46. … and performance is ahead of trajectory Vehicle Crime - 30% reduction from 1998-99 to 2004 British Crime Survey 3,500,000 3,000,000 2,941,927 2,500,000 2,486,314 2,366,070 2,000,000 Number of offences 2,059,349 1,500,000 1,000,000 500,000 - 2000 BCS 2001/02 2002/03 2003/04 2004/05 (crime in 1999) interviews interviews interviews interviews Source: Home Office

47. Total violent crime has fallen since 1997 Number of Violent Crimes - British Crime Survey 4,000,000 3,665,000 3,423,000 3,500,000 2,784,000 2,781,000 3,000,000 2,500,000 Number of Offences 2,000,000 1,500,000 1,000,000 500,000 - 1998 BCS 2000 BCS 2001/2 2002/3 Interviews Interviews Source: Home Office (Crime in 1997) (Crime in 1999)

48. But most people believe crime is rising Beliefs about the change in the national crime rate 1996 to 2002/03 80% 70% 29% 60% 35% 50% 34% 34% Percentage saying crime has increased 29% 31% 40% 30% 46% 20% 38% 33% 30% 30% 25% 10% 0% 1996 1998 2000 2001 2001/02 2002/03 Lot more Little more Source: Home Office

49. Gun crime has increased - as has media coverage Total Recorded Crimes Involving Firearms (other than air weapons) 12,000 10,000 8,000 Number of Crimes 6,000 4,000 2,000 - 1997/98 1998/99 1999/00 2000/01 2001/02 Source: Home Office

50. Not focussing on the right crime? Fear of anti social-behaviour Vandalism and graffiti 40% 35% 30% 25% Teenagers hanging around on the streets Percentage that perceive disorder to be a 'very' or 'fairly' big problem 20% 15% People using / dealing drugs 10% 5% 0% 1992 1994 1996 1998 2000 2001 2001/02 2002/03 Source: Home Office