What’s New in Outcome Measures?

1. UKABIF Annual Conference Wednesday 11th November 2009 What’s New in Outcome Measures? Professor Nick Alderman Consultant Clinical Neuropsychologist St Andrew’s Healthcare Northampton UK nalderman@standrew.co.uk

2. Special Issue of Neuropsychological Rehabilitation ‘Evaluation of Outcomes in Brain Injury Rehabilitation’ (1999) Concluded there is “...no single solution to a complex problem” (Fleminger & Powell)

4. A Compendium of Tests, Scales and Questionnaires The Practitioner’s Guide to Measuring Outcomes after Acquired Brain Impairment By Prof Robyn Tate (University of Sydney) A comprehensive reference manual providing a detailed review of approximately 150 specialist instruments for the measurement of signs and symptoms commonly encountered in neurological conditions, both progressive and non-progressive. Published by Psychology Press 6th January 2010

5. Structure of Presentation • What outcomes to measure? • At what level? • The Outcome ‘Basket’ • Challenges to measuring service level outcomes • Some ongoing examples of developments in the measurement of neurobehavioural outcomes • Some final thoughts

6. Service Users Outcome ? Clinicians Families Purchasers

7. What Outcomes? Depends on who is asking about what… Turner-Stokes (1999) suggests: • Medics – impairment • Therapists – disability and independence • Patients & Families – reduction in handicap and quality of life Rehabilitation – regaining functional independence Most outcome tools therefore measure: • Increase in independence • Reduction of disability

8. Lynne Turner-Stokes (1999) Outcome measures should be: • Valid • Reliable • Relevant to stage of recovery • Sensitive to change • Integrated seamlessly within routine clinical practice

9. At What Level? The individual service user • Everybody concerned with the care and wellbeing of the person with ABI interested in this level of outcome measurement ABI Rehabilitation Services • Commissioners and purchasers of ABI services (‘efficacy snapshots’)

10. ‘tactical’ ‘strategic’ vs. Evidence of Homer’s individual rehab outcomes Evidence of outcomes reflecting the efficacy of Springfield General Hospital as a service

11. Lynne Turner-Stokes (1999) Outcome measures are either: ‘Focal’ vs. ‘Global’ Discuss in the context of individual vs. service level outcome measurement

12. ‘Focal’ Measures Distinct behaviours or functions • 10 Metre Walk • Nin-hole Peg Test • Verbal aggression Often used to measure outcome of specific interventions • Can be directly observed and consistently recorded • Very flexible, designed to meet individual needs (e.g. Goal Attainment Scaling) • Contributes to evidence based knowledge and practice

13. Use of methodologies borrowed from single-case experimental design can help answer questions asked by different people by objectively and validly demonstrating quality outcomes arising as a consequence of treatment. For example, for the clinician: “Is my treatment the cause of this person’s change?” (Wilson, 1991)

14. This can incorporate principles from single-case experimental design methodology… baseline treatment baseline treatment

15. ‘A-B-A-B-C’ Design Showing Reduction in Frequency of Spitting after ABI A baseline B DRL A baseline B DRL C DRI 1 5 10 15 20 25 30 35 40 45

16. Advantages of Single Case Experimental Design Methodologies at Level of Individual Outcome Measurement • person is their own control • tailored to meet individual needs • flexible • few ethical concerns

17. Standardised Behaviour Rating Scales for People with Acquired & Progressive Neurological Conditions • The Overt Aggression Scale - Modified for Neurorehabilitation (OAS-MNR: Alderman, Knight and Morgan, 1997) • The St Andrew’s Sexual Behaviour Assessment (SASBA: Knight, Johnson, Alderman, Green, Birkett-Swan & Yorston, in preparation)

18. Overt Aggression Scale - Modified for Neurorehabilitation (OAS-MNR) • A standardised behaviour rating scale with good psychometric properties suitable for the measurement and assessment of aggression for people with ABI within in- patient settings • Each incident of aggression is captured to determine possible relationships between the environment and behaviour (objectively inform formulation) • Although evolved primarily for use with ABI, OAS-MNR has seen widespread use with other clinical populations

21. B’line Intervention

23. ‘Global’ Measures Provide a holistic view of disability (rating scales) • Barthel Index • FIM • FIM+FAM • Self-report measures Advantages • Standardised measures with known psychometric properties (validity, reliability) • Broad overview of strengths and weaknesses • Highlight targets for rehab • Common language providing continuity within and between services

24. Example FAM-splat 26-30 Cognitive Function 1-7 Self Care 8-9 Sphincter Control 22-25 Psychosocial Adjustment 10-13 Mobility/ Transfers 17-21 Communication 14-16 Locomotion Adm Cur Goal

25. What To Use Multiple Consequences of ABI: • physical • functional • cognitive • emotional • psychosocial • behavioural Type of service (generalised, specialised)

26. No single outcome measure is suitable for all brain injury rehabilitation Pick and chose a ‘basket’ of outcome measures

27. Most Frequently Used Outcome Measures in UK Rehab Centres (Turner-Stokes & Turner-Stokes, 1997) 82% neurorehab services, 123/180 used global measure (Barthel, FIM, FIM+FAM)

28. FIM+FAM HoNOS ABI HoNOS Secure

29. Service Level Outcomes • Global measures of disability typically used • Data from individual SU’s pooled (average) • Provides ‘snapshot’ to commissioners regarding effectiveness of service

30. Not all outcome baskets – or what is put into them – are the same

31. Not all service users are equivalent

32. Multiple Consequences of ABI • physical • functional • cognitive • emotional • psychosocial • behavioural

33. Particular Difficulties in Using Group Methodologies to Determine Quality Outcomes with ABI Non-homogenous group Mateer & Ruff (1990) “...no two head injuries are alike” Wilson (1991) people present with combination of problems, rarely identical • lack of homogeneity • between & within sub-populations • needs & strengths • different stages of recovery

34. Lack of homogeneity Integrity of measures when data are pooled • group averages obscure individual outcome • data distributions can be abnormal • variability within data can be extreme

35. Effectiveness of a Behavioural Approach in Reducing the Frequency of Physical Aggression in a Group of 40 Clients (from Alderman, Bentley & Dawson 1999) 22.9 7.1

36. Effectiveness of a Behavioural Approach in Reducing the Frequency of Physical Aggression in a Group of 40 Clients (from Alderman, Bentley & Dawson 1999) 72.4 17.3

37. Effectiveness of a Behavioural Approach in Reducing the Frequency of Physical Aggression in a Group of 40 Clients (from Alderman, Bentley & Dawson 1999) 451 33

38. 451 82 13 33 Effectiveness of a Behavioural Approach in Reducing the Frequency of Physical Aggression in a Group of 40 Clients (from Alderman, Bentley & Dawson 1999)

39. Some Ongoing Developments in Measuring Neurobehavioural Outcome Response to these challenges • Generation of a ‘snapshot’ of service level outcomes from an observational behaviour rating scale designed for use with individual SU’s. • ‘focal’ measure (aggression) • Individual to service level outcome • A new rating scale to measure neurobehavioural disability and social handicap. • ‘global’ measure • Individual and service level outcome

40. Creating Service Level Performance Indicators from Routinely Administered Clinical Tools Can a clinical measure employed to collect data for the purpose of delivering individual care can be utilised as an ongoing indicator of service level outcome? Advantages of: • specificity • known reliability/validity • savings in clinicians time for training and administration

41. OAS-MNR data recorded for 79 service users: • ABI acute neurobehavioural wards • PNC wards AAS = total frequency of aggression x mean weekly weighted severity Aggregate Aggression Score calculated for the first and most recent three months of admission

42. Comparison of Frequency and AAS Scores ABI PNC

43. Comparison of Median Frequency and AAS Scores Percentage Median AAS Improvement between First and Last 3 Months of Admission

44. Creating Service Level Performance Indicators from Routinely Administered Clinical Tools • Pooling data still creates problems with abnormal distributions – use median • Incorporating a range of information into a rolling PI gives a more balanced index of change • Can be drilled down, e.g. to ward level • Crude but valuable ‘snapshot’ to help inform commissioners and other stakeholders re efficacy of service

45. Kolitz et al (2003) argued that there was still a need to develop a valid and comprehensive instrument for the measurement of NBD.

46. Neurobehavioural Disability • executive dysfunction • attention deficits • diminished insight • poor social judgement • labile mood • problems with impulse control • personality change Wood (2001) Complex, subtle, pervasive constellation of cognitive-behavioural changes that characterise post-acute ABI social handicap

47. Social Handicap Undermines a Person’s Capacity for Independent Social Behaviour

48. Kolitz et al (2003) argued that there was still a need to develop a valid and comprehensive instrument for the measurement of NBD. Wood, Alderman & Williams (2008) Assessment of neurobehavioural disability: a review of existing measures and recommendations for a comprehensive assessment tool Brain Injury, 22, 905–918 • Undertook review of the psychometric properties of 8 widely used measures of NBD • Made proposals regarding development of future measures of NBD

49. Wood, Alderman & Williams (2008) Some of the proposals made regarding development of future measures of NBD included: • Measures have clear theoretical underpinning/conceptual framework that drives item selection • Information is obtained from an informant • Sufficient pool of items to capture the diverse range of NBD signs and symptoms • Robust, well known statistical properties • (reliability data sparsely reported)

50. Wood et al (2008) review did not contradict claim by Kolitz et al (2003) regarding the need to develop a valid and comprehensive instrument for the measurement of NBD