1 / 18

Methods for assessing fidelity and quality of delivery of smoking cessation behavioural support

Methods for assessing fidelity and quality of delivery of smoking cessation behavioural support. Fabiana Lorencatto, Robert West, Carla Bruguera, & Susan Michie Centre for Outcomes Research and Effectiveness, University College London

sage
Download Presentation

Methods for assessing fidelity and quality of delivery of smoking cessation behavioural support

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Methods for assessing fidelity and quality of delivery of smoking cessation behavioural support Fabiana Lorencatto, Robert West, Carla Bruguera, & Susan Michie Centre for Outcomes Research and Effectiveness, University College London Health Behaviour Research Centre, University College London 3National Centre for Smoking Cessation and Training

  2. Why is there substantial variability in quit outcomes across services? • Is it the content of support delivered? • Many services have structured behavioural support treatment manuals • Specify recommended content and format of sessions • Help guide and standardise practitioner behaviour • Practitioners from same NHS SSSs, operating under same manual have widely varying success rates (Brose et al. 2012) Actually following the treatment manual?

  3. Two aspects of delivery need to be examined: • What manual specified content is actually delivered? • Fidelityof delivery • extent to which interventions are delivered as intended (i.e. recommended practice  actual practice) 2. How is this content actually delivered? • Qualityof delivery • Appropriate + comprehensive delivery of intervention content *** Does this affect outcomes?

  4. Aims Two linked studies Study 1: • Develop a reliable method for assessing fidelity of delivery of behavioural support in practice. Study 2: • Develop a reliable method for assessing quality of delivery of key technique, setting a quit date (i.e. goal setting) • Examine the association between quality of goal setting and likelihood of clients making a quit attempt.

  5. Study 1 assessing fidelity of delivery ‘WHAT is delivered’

  6. METHODS: Sample and materials • National telephone behavioural support service (i.e. quit line) • Six practitioners • All trained to NCSCT standard • All aware of treatment manual • Service treatment manual (i.e. recommended practice) • Outlines recommended content/format of pre-quit, quit day, and post quit sessions • Transcripts of 64 audio-recorded, consecutively delivered behavioural support sessions (i.e. actual practice). • Pre-quit sessions (n=27) • Quit-day sessions (n=16) • Post-quit sessions (n=21).

  7. Procedure • Two coders identified + categorised BCTs in treatment manuals and session transcripts using taxonomy of 43 smoking cessation BCTs. • Inter-rater reliability assessed using % agreement • Examined what % of manual-specified BCTs were present in transcripts of practice sessions (recommended vs. actual practice)

  8. RESULTS Reliability: average percentage agreement was high (87.1%; range: 78.4% - 95.6%) Fidelity: Note: <50% = ‘low fidelity;’ 51-79%= ‘moderate;’ 80-100% =‘high’

  9. Study 2 ‘HOW’ assessing quality of delivery of ‘setting a quit date’ (i.e. goal setting) association between quality of goal setting and quit attempts.

  10. METHODS • Sample & Materials: • Same quit line service and practitioners • New set of transcripts: 85 pre-quit behavioural support sessions • 85 clients • Pre-quit sessions as this when quit date likely to be set • Outcomes: whether client made a quit attempt as planned by following session (yes/no)

  11. PROCEDURE 1) Development of quality rating scale: • Quality= appropriate + comprehensive delivery of technique • Two researchers independently examined guidance documents to identify components of high quality ‘setting a quit date’ • NCSCT training standard, standard treatment programme, training module, quit line service treatment manual • Extracted components featured across all guidance documents  description of ‘gold standard’ goal setting • Basis for content of rating scale

  12. Points allocated for appropriate delivery Points deducted for inappropriate delivery Score range: -2 to 6 Higher score implies more comprehensive + appropriate delivery 0 Absence of goal setting +1 Prompts goal setting (i.e. encourage client set quit date) +1 Agrees clear quit date (dd/mm/yy) +1 Agreed quit date within appropriate time frame (1-2 weeks) +1 Quit date allows appropriate time to obtain medication +1 Emphasise goal is to not smoke a single cigarette/puff +1 Provide relevant normative information and examples -1 Inappropriate goal setting (i.e. unclear date, incorrect time frame) -1 Undermines client commitment to quit date (i.e. imply flexibility)

  13. Scored sections of transcripts in which ‘setting a quit date’ occurred using scale • Inter-rater reliability assessed using Cohen’s Kappa (weighted) • Examined association between quality scores and whether client made a quit attempt as planned • Multilevel logistic regression • both using total quality score as predictor • And individual scale component as predictor

  14. RESULTS Inter-rater reliability: Weighted kappa = 0.68  ‘substantial agreement’ (Landis & Koch,1977) Average quality score: - overall: 1.6 (SD 1.2)  low quality (scale range: -2 to 6) - clients madeattempt (n=18; 21.2%): 2.2(SD.70) - clients did notmake attempt (n=67; 78.8%):1.4(SD 1.27)

  15. Association with quit attempts: Higher quality goal setting in pre-quit sessions significantly increased likelihood of clients making quit attempt as planned [p<.001 OR 2.60 (95% CI: 1.54 to 4.40)] Only 1 scale component independently associated significantly with quit attempts: • Setting a clear quit date (dd/mm/yy): p<.001, OR 36.9, 95% CI: 4.52 to 302.11) • Opposite of this: Setting inappropriate quit date (i.e. unclear date): p<.001, OR .03, 95% CI: .24 to 2.88)

  16. DISCUSSION • It is possible to reliably assess fidelity and quality of delivery of smoking cessation behavioural support in practice • Preliminary analysis of national telephone quit line service: • Less than half of what is specified in treatment manuals is not being routinely delivered in practice • Quality of ‘setting a quit date’ is low, but higher quality significantly increases likelihood of initiating quit attempts • Particularly if a clear date is set. • Deconstruction of technique into individual components: identify specific aspect driving outcome (i.e. ‘active ingredient’)

  17. Limitations: • Limited generalisability- single service examined; other modalities • Service manual extremely comprehensive (22-29 BCTs per session!)  feasible? Relevant? • Implications & Future work: • Identification of training needs + aspects of service delivery to be targeted for improvement • Expand quality scale to other behaviour change techniques • Need to examine how practitioners actually use manuals in practice • And whether manuals are fit for purpose • Conclusions: Stop smoking services should establish routine procedures for monitoring fidelity and quality of behavioural support delivered.

  18. Thank you for listening. Fabi.lorencatto@ncsct.co.uk The team: Fabiana Lorencatto, Robert West, Carla Bruguera, & Susan Michie Acknowledgements: Collaborating services Funding: National Centre for Smoking Cessation and Training; Department of Health

More Related