Benchmarking Adult Presentation

1. Benchmarking Adult Presentation The highlights of our trip. Fiona Whitecross on behalf of the travelling party!

2. Itinerary overview • Get to know the travel participants. (9 MHS across Australia) • Determining best route- the best indicators for the group. • Rigorous review of the itinerary- indicators technical specs, iron out the bumps. Determining best routes. • Adding to the itinerary- supplementary indicators. • Customs check required to determine if any prohibited data entering.

3. Adult forum phases of development-Phase 1 • About definitions-What constitutes episode of care? What is definition of new? • Focused on differences. We are so different we cant benchmark going through the same……. • Developing trust & building relationships

4. Phase 2 • Data becoming cleaner. • QI built in increased motivation. • Went off on tangents- gender 28 day bought back to task. • Group trust and relationships matured. • Identified supplementary indicators • Looked at 28 day KPI in more detail

5. Phase 3 • Developed a better sense of KPI usefulness • QI progress reports identified improvements being made through benchmarking • Developed a better sense of best practice i.e. 28 day readmission. • Developed a confidence in the data and benchmarking process.

6. What we learned…The indicator set BM Forums asked of each KPI: • What we learned? • Is it relevant? • Does it measure what it intended? • Are national data specs right? • Can uniform targets be set? • Can it be interpreted and understood? • Is it feasible to collect and report on at a local level? • Recommendations?

7. KPI # 1 28 day readmission rate • says ‘something’ about services, it is not diagnostic of a particular problem but rather tells of a potential pathology within the service. • A range of factors influence the indicator, including: bed availability; experience and skill mix of staff (inpatient and community); • Analysis and identification of appropriate allied indicators (such as average length of stay and post-discharge community care) and contextual factors is essential to accurately interpret the output. • It is important to note that not all readmissions are a failure of care. • Specific action or inaction can be linked to higher or lower readmission. • Although not all factors influencing readmission rates are in the control of service organisations, there is work that can be undertaken locally to impact on readmission rates. • The time period of reporting and analysis is important as the indicator fluctuates on a month-to-month basis.

8. KPI # 2 National Standards Compliance • The indicator has a tendency to produce a ‘Yes’ or ‘No’ output and as such does not provide information about incremental improvement by an organisation. • There are additional standards (such as the National Mental Health Practice Standards) that are relevant to mental health services that are not measured by this indicator. • The use of expenditure to distribute compliance across the service complicates understanding of the indicator and the increasing trend for services to be accredited as a ‘whole’ rather than as individual units or settings further diminishes the utility of the indicator at the service level. • It is good to acknowledge an external review of processes, however it may be more useful to be able to distinguish between where organisations are at within the accreditation and continuous quality improvement cycle. • Further work is needed to refine the definition and specifications of this indicator.

9. KPI # 3 Av LOS • The mean is impacted on by extreme outliers (e.g. consumer needing extended treatment care receiving care within acute unit as no beds available). • It needs to be interpreted within the context of the service and other indicators as the indicator is susceptible to changes in medical and nursing leadership and practice, discharge practices, bed occupancy, community resources. • Median LOS provides additional contextual information that enables a more accurate picture of the typical length of stay of most consumers.

10. CONT…… • The indicator should remain within the national indicator set as currently defined and specified. • A preliminary good practice target of 12 days or less should be considered for use with general adult mental health services. • This indicator should be considered in the context of the median length of stay.

11. KPI # 3 Inpatient episode cost • Inpatient episode costs are largely driven by length of stay, therefore the influences on length of stay also impact on the costs. This is an issue when an extreme outlier distorts the average. • Inpatient episode cost is utilised within general health. • At the organisational level there is a need to unpack costs and identify associated issues (such as staff hours per day) to enable understanding of efficiency. • Annual average cost per bed is also a useful supplementary indicator. • Can be a misleading indicator of efficiency due to a range of reasons (e.g. impact of outliers, poor expenditure data). • Reliability of indicator is dependent upon good quality, accurate and consistent financial reporting (especially regarding organisational overheads).

12. KPI # 4 Treatment days per 3 month community care • The indicator needs to be interpreted within the service context as it is influenced by the model of service adopted. • Treatment days can be influenced by a range of factors outside the control of the local services, such as staff experience, service models, rurality, access to inpatient services, access to NGO services. • The average can be impacted on by extreme outliers, particularly in smaller services. • The indicator is not a measure of FTE productivity and is not intended to account for how clinicians spend there time. • It was noted that utilisation of days is a method to account for reporting variation associated with occasions of service or service contacts, both within and between jurisdictions. • Under-reporting of ambulatory contacts is a significant issue that impacts on the reliability of the output.

13. KPI # 5 Cost per three month community care period • The indicator is highly susceptible to poor compliance by clinicians with local information reporting, particularly contact reporting, requirements (i.e. low reporting rates increases costs). • The impact of outliers and other factors can make this a misleading indicator of efficiency due to a range of reasons. • There are significant concerns regarding the quality of expenditure information. • Average cost per treatment day may be more useful at the MHSO level.

14. KPI # 7 Population under care • The indicator is informative as defined for the national project (that is, split between the three settings). • Access is impacted on by a range of issues (structural, population and service) that may not be within the control of the service. • Access to mental health services is an ongoing issue for most services and capacity to monitor and improve access (where necessary) is relevant. • Need to be clear that it is not about the percentage of the catchment population receiving mental health care, but rather the percentage of catchment population receiving mental health care from local services, although for some services this may equate to the same thing.

15. KPI # 8 Local access to inpatient care • The concept of ‘local’ is difficult to define, therefore the indicator looks at local as being within the defined catchment area of the service, which from the perspective of the consumer, carer and/or clinician may not be ‘local’. • For services whose inpatient catchment stretches a large geographic region it makes the concept of ‘local’ less meaningful. • Access to mental health services, particularly scarce inpatient resources, is an ongoing issue for most services and capacity to monitor and improve access (where necessary) is relevant.

16. KPI # 9 New Client Index • Access (or lack thereof) to mental health services is an ongoing issue for most services and capacity to monitor and improve access (where necessary) is relevant. • This is a conceptually complex indicator, primarily because defining ‘new’ has many interpretations and definitional approaches, such as new to service versus new to setting versus new to program versus new to diagnostic group and so on. • The indicator looks at who is new to an organisation, regardless of setting or program (i.e. if come from other program not considered ‘new’). There is a difference between ‘new’ to service / program and ‘new’ to mental health care that cannot be determined by the current definition. • The indicator does not specify that the client needs to be an ‘active’ or ongoing client of the service (i.e. includes assessment only) as the indicator is about access and getting an assessment is about accessing the service. • The forum expressed the view that ‘new’ should refer to consumers those consumers having their first contact with any mental health service.

17. KPI # 10 Comparative area resources • This is not an indicator of service performance as funding allocation is not completely within the control of individual mental health service organisations. However, it has the potential to provide: (i) significant leverage for influencing policy and funding decisions; and, (ii) information to service managers to assist in the interpretation of other indicators. • The indicator is informative as defined for the national project (that is, split between the three settings). • Access is impacted on by a range of issues (structural, population and service) that may not be within the control of the service. • The reliability of output is dependent upon good quality, accurate and consistent financial reporting (especially regarding organisational overheads).

18. KPI # 11 Pre admission community care • This indicator is based on the concept that pre-admission community care can potentially (i) ease transition into acute care, (ii) reduce the length of stay (limited evidence-base for this argument), (iii) that the inpatient setting is not the ‘front-door’ or entry point to a mental health service organisation. • The indicator is not about identifying proportion of admissions that could have been prevented or averted and does not assume that a high percentage pre-admission community care is an indication of failure of community care. • The indicator attempts to identify those consumers who are not seen – i.e. those who are not receiving a service or are falling through ‘the gaps’ in community care prior to admission. • Sensitive to demographic factors. • Vulnerable to poor amb collection adherence

19. KPI # 12 Post D/C community care • The indicator is clinically relevant and generally understood by most clinicians and service managers. • The indicator measures good practice directly. It has meaning at the individual clinician level and can drive practice improvement and change. • Public mental health services cannot be expected to see everyone discharged from public inpatient units as some consumers are appropriately followed up by GPs, private psychiatrists or other services. • As the indicator is currently specified there is no differentiation between people who are not contacted versus those where contact is attempted by service but refused or failed (due to movement from jurisdiction). • The seven day parameter was chosen due to substantial literature indicating increased risk of suicide within the first seven days following discharge from acute care. However, there is less evidence that follow-up within seven days makes a difference for the consumer in regards to community tenure. • Indicator is vulnerable to poor ambulatory data collection compliance.

20. KPI # 13 KPI Outcomes readiness • This is predominately a context indicator that facilitates interpretation of outcomes information. • The indicator is overly generous in its calculation of participation, which causes some difficulty in interpretation and face validity (eg when have 150% participation). • The current construction is skewed in the favour of residential or long-stay services. • Compliance with data collection protocols is not an indication of data quality.

21. Trip highlights • Developed positive collegial relationships. • Developed insights into own service performance as compared to others. • Learned about best practice in other organisations. • Developed a comprehensive understanding of the 13 KPI’s • Luxury to meet over 2 years commitment of time beneficial not about networking but establishing relationships. • Senior staff there consistently gave process credibility.