Why collect data? Current issues for service monitoring and evaluation

1. Why collect data? Current issues for service monitoring and evaluation Linda Bauld

2. Context The policy context for smoking cessation is going through a process of significant change. • Smoking cessation update in Scotland (May 2007) • New Service and Monitoring guidance in England (updated February 2008) • NICE guidance in England (February 2008) • Update to the Thorax guidelines (2008) • Revised MDS in Scotland (2009?) • Results of DH consultation (100,000 responses) (2009) • National Centre for Smoking Cessation Training (2009) Given these developments, what should we be measuring?

3. All smokers Motivated smokers First contact withservice Quit date Treatment Short-term quit Longer-term quit Relapse New quit attempt

4. All Smokers Motivated Smokers First contact withservice Quit Date MDS + most clinical research Treatment Short-term Quit Longer-term quit Relapse New quit attempt

5. Measuring Success Robust monitoring of: • the number of clients setting a quit date • the number who have quit at 4 weeks • confirmed by CO validation • submitted centrally+ any local analysis is crucial, and remains the key mechanism for assessing performance. However ….

6. All smokers Motivated smokers Who from amongst these groups are we reaching? First contact withservice Areas and activities we increasingly need to measure and assess Quit date Treatment Short-term quit Longer-term quit Relapse New quit attempt

7. All smokers Motivated smokers Which smokers should we be reaching? How do we proactively identify smokers? How many smokers should we be reaching?

8. Reaching Smokers • Although we have effective services, they reach few smokers. • Data from Robert West’s smoking toolkit study suggests around 3-5% of smokers attempt to quit each year with support from NHS services in England. This proportion increased during the ‘surge’ resulting from smokefree legislation. • Throughput figures from services in both England and Scotland suggest some services may be reaching a higher proportion of smokers than that, in specific parts of the country.

9. The importance of reach In the past, programs were usually evaluated by their abstinence rates. A program resulting in a 30% abstinence rate was judged twice as effective as one producing 15% abstinence. But a program producing 30% abstinence and 3% participation has only a 0.9% impact. A program producing 15% abstinence with 60% participation has 9.0% impact, which is 1000% greater. What the field needs are interventions that can maximise participation without sacrificing abstinence rates.” Prochaska et al 2001, Addictive Behaviours, 26, 584-5.

10. What constitutes ‘success’ in reach? • NICE guidance in England recommends that services treat 5% of their smoking population each year. • There is no explicit ‘reach’ target in Scotland although HEAT targets now require (as in England) certain numbers of four week quitters.

11. Evidence of Reach • 4 evaluations of NHS SSS have looked specifically at effectiveness in accessing disadvantaged groups: • Lowey et al 2002 • Chesterman et al 2005 • NEPHO 2005 • Baker et al 2006 Bell et al, 2007 NICE review

12. “Mapping study” Marks et al, 2007 Cost effectiveness reviews “The effectiveness of smoking cessation interventions to reduce the rates of premature death in disadvantaged areas through proactive case finding, retention and access to services” Bauld et al, 2007 NICE interventions guidance – in development

13. Bauld, McNeill, Hackshaw & Murray 2007

14. What can we do to improve reach? NICE guidance in development • Proactive identification through: primary care* (case records) QOF, combining smoking with other interventions eg screening, cold calling, direct mail • Social marketing approaches & tailoring • Flexibility of delivery eg drop in or rolling clinics, out of hours services • Flexible location (workplace) • Pharmacy based smoking cessation services • Dental services • Lay people as stop smoking advisors • Incentives

15. Does your service try to attract smokers from economically disadvantaged backgrounds? McEwen A, unpublished data from 2008 survey, n=497 smoking cessation practitioners

16. Methods used to attract deprived smokers McEwen A, unpublished data from 2008 survey, n=497 smoking cessation practitioners

17. Other populations? • Prisoners • Smokers with serious mental health disease • Smokers with drug and alcohol problems • Pregnant smokers • Unemployed smokers • Smokers from black & minority ethnic groups

18. Success and Reach • Monitoring who you reach and how many smokers are reached is something services can do locally. • If new approaches are trialled, they should be carefully piloted and reviewed using local monitoring data or partnerships with researchers. • Robust evidence of effective strategies to improve reach, particularly with disadvantaged groups, is desperately needed.

19. Outcomes and Reach • Once smokers have been reached, assessment of quit rates is essential. • This is already done via national monitoring. • However, local analysis of outcomes is also important • and assessment of outcomes for specific sub-groups of smokers (via national and local monitoring and also research) is currently lacking and needed.

20. How do we measure what client’s think of the service? How can we build on the success of clients who do quit? How can we encourage relapsers to return? Short-term quit Longer-term quit Relapse New quit attempt word of mouth referrals

21. Client Views • Most services (71% in a recent survey of all coordinators in England) seek the views of their clients in some way • This is important because it provides information that can be used to improve services

22. Client Views • However, very few services have the time or resources to conduct surveys or qualitative research with clients routinely or robustly • There is a need for a simple tool that could be used to assess client satisfaction • One has recently been developed and validated through the SCSRN (www.scsrn.org) and is available to download • This could complement any more detailed work services choose to do with particular client groups.

23. Building on Success • Work with successful quitters also provides opportunities to assess the benefits of your service. • Satisfied clients are the best ambassadors. • Among other things, they can generate word of mouth referrals and community ownership/recognition of your service.

24. Building on Success • Collecting ‘source of referral’ data, as many services already do, can help you assess the extent of ‘word of mouth’ (family/friend) referrals. • Some services have seen this proportion rise following changes to improve reach, such as the introduction of drop in clinics. • Local innovation is also needed by should be evaluated – i.e. work around a word of mouth referral card in the South West.

25. Relapse • We know that around 75% of 4 week quitters will relapse by one year. • These smokers should be encouraged to return to use services again, maximising their chances of a successful quit attempt. • Barriers to returning (ie the old 6 month ‘rule’) are increasingly being dismantled by services. • Measuring who returns and what the outcomes are is an important area for local monitoring and research.

26. New Opportunities • The National Centre for Smoking Cessation Training in England, once established, will support services to improve data collection and reporting systems • It will also link with the latest research evidence to inform practice • The NCSCT should commence work in the second half of 2009.

27. New Opportunities • Research is also underway to examine methods used for CO verification of abstinence by services, with a view to developing guidelines for best practice. • See www.scsrn.org for more details.

28. Conclusion • We will always need to robustly measure the ‘core’ elements of success (quit rates) However, we also need to assess: • Reach (particularly amongst disadvantaged groups) • Outcomes for specific groups • Client views • Referral sources • Returns following relapse All these issues need more research, but monitoring can help assess what you are achieving, and point to where more needs to be done.

29. Thank You L.Bauld@bath.ac.uk www.ukctcs.org