A FOCUS ON QUALITY OUTCOMES

1. A FOCUS ON QUALITY OUTCOMES DAO and WANADA Session 1st May 2012 ANITA GHOSE B. App Sci; MBA; MAICD DIRECTOR Anita Ghose and Associates

2. AGENDA 1. An overview of outcomes 2. Developing outcomes 3. Measuring outcomes

3. 1. An Overview of Outcomes

4. REFRESHER – KEY CONCEPTS INPUTS OUTPUTS OUTCOMES Activities Products Participation Short Medium Long-term Resources Used What we produce What are the results? What changed? What we invest What we do Who we reach Eg: Number of hours of service delivered Number of counselling sessions provided Number of people in residential rehab services Eg: Provide counselling Provide diversion services Provide residential rehab services Eg: People who secured stable housing Improved attitudes towards cannabis use Increased knowledge of the effects of alcohol and drug use Reduction in harmful use of alcohol and drugs Eg: Money Staff time Facilities Equipment

5. PROCESSES – eg waiting lists, response times, client/staff ratios INPUTS OUTPUTS OUTCOMES Activities Products Participation Short Medium Long-term Resources Used What we produce What are the results? What changed? What we invest What we do Who we reach Based on Productivity Commission Report 2010; United Way of America (1996)

6. INPUTS OUTPUTS OUTCOMES Activities Products Participation Short Medium Long-term Resources Used What we produce What are the results? What changed? What we invest What we do Who we reach Based on Productivity Commission Report 2010; United Way of America (1996) Population

7. Inputs—the resources used by the organisation to achieve the objectives of an activity – “what we invest” eg money, staff time, facilities, equipment Processes – the related tasks and systems of the organisation to fulfil its mission (eg. response times, staff trained, worker safety, client/staff ratio, length of wait lists) Outputs— the direct products of the organisation’s activities or intervention (“what we produce”) and the participation (“who did we reach”). Eg number of service hours delivered, number of counselling sessions provided, number of clients in residential rehabilitation services Outcomes—the effects or change on an individual or group as a result of the activities and outputs (“what was the result?) Eg people who secured stable housing, reduction in harmful use of alcohol and drugs, increased knowledge of the effects of cannabis Impacts—the broader effects of an activity reflected in community wellbeing domains (what changed for the population?) Eg. reduced unemployment, increased mental health, stronger families

8. The effects or change as a result of the activities or outputs INPUTS OUTPUTS OUTCOMES Activities Products Participation Short Medium Long-term Resources Used What we produce What are the results? What changed? What we invest What we do Who we reach

9. Government Agencies are moving here in OBP INPUTS OUTPUTS OUTCOMES Activities Products Participation Short Medium Long-term Resources Used What we produce What are the results? What changed? What we invest What we do Who we reach Eg: Number of hours of service delivered Number of counselling sessions provided Number of people in residential rehab services Eg: Provide counselling Provide diversion services Provide residential rehab services Eg: People who secured stable housing Improved attitudes towards cannabis use Increased knowledge of the effects of alcohol and drug use Reduction in harmful use of alcohol and drugs Eg: Money Staff time Facilities Equipment

10. This isn’t being prescribed by Govt – BUT organisations do need to understand their business here INPUTS OUTPUTS OUTCOMES Activities Products Participation Short Medium Long-term Resources Used What we produce What are the results? What changed? What we invest What we do Who we reach Eg: Number of hours of service delivered Number of counselling sessions provided Number of people in residential rehab services Eg: Provide counselling Provide diversion services Provide residential rehab services Eg: People who secured stable housing Improved attitudes towards cannabis use Increased knowledge of the effects of alcohol and drug use Reduction in harmful use of alcohol and drugs Eg: Money Staff time Facilities Equipment

11. OUTCOMES There will generally be effects, results, changes in one or more of these core areas: KNOWLEDGE OR SKILLS ATTITUDE OR OPINION VALUES BEHAVIOUR CIRCUMSTANCE CONDITION OR STATUS

12. OUTCOMES KNOWLEDGE OR SKILLS - increased knowledge of the effects of alcohol and drug use on the body ATTITUDE OR OPINION – improved attitudes or opinion about the dangers of cannabis use VALUES - enhanced values on the impact of drug use on others around the person BEHAVIOUR – reduction in the harmful use of drugs and alcohol CIRCUMSTANCE – increased ability to secure and maintain stable housing CONDITION OR STATUS – improved general health and wellbeing

13. OUTCOMES – KEY CONSIDERATIONSProximal/DistalAttribution/ContributionPathways approachLink to outputsReasonable numberPerverseMeasurement versus achievementPositive or negativeIntended or unintended

14. EXAMPLECommunity Drug Service TeamCounselling modelProximal outcome:Amphetamine users have reduced their drug and alcohol useAmphetamine users have an improved level of mental health and wellbeingDistal outcome:Amphetamine users have reduced levels of offending

15. EXAMPLEMother and Child ProgramSpecialist supportProximal outcome:Mothers reduce their alcohol and drug useMothers demonstrate an increased awareness of the harmful effects of alcohol and drug use on their childMothers remain the main caregiver of their childDistal outcome:Mothers participate more in their children’s schooling program

16. How efficient is the organisation?How effectiveis the organisation?

17. Efficient and effective INPUTS OUTPUTS OUTCOMES Activities Products Participation Short Medium Long-term Resources Used Efficiency = Inputs Outputs Effectiveness = Inputs Outcome Example: the average cost per completed as planned episode Example: the average cost per session of counselling delivered

18. Outcomes Frameworks

19. Is a framework for describing the relationships between investments, activities, and results. Depicts a program showing what the program will do and what it is to accomplish (reasoning) Provides a series of “if-then” relationships that, if implemented as intended, lead to the desired outcomes Provides a common approach for integrating planning, implementation, evaluation and reporting. Applied at a program, process, multi component program, organisational level A Logic Model Source: University of Wisconsin – Extension, Cooperative Extension, Program Development and Evaluation

20. Fully detailed logic model Source: University of Wisconsin – Extension, Cooperative Extension, Program Development and Evaluation

21. Logic model and reporting Source: University of Wisconsin – Extension, Cooperative Extension, Program Development and Evaluation

22. RESULTS BASED ACCOUNTABILITY

23. RESULT or OUTCOME A condition of well-being for children, adults, families or communities. Population Population INDICATOR or BENCHMARK A measure which helps quantify the achievement of a result. PERFORMANCE MEASURE A measure of how well a program, agency or service system is working.Three types: Performance Performance Source: Mark Friedman 2005 Strong and stable families, positive family relationships Rate of family breakdown due to alcohol and drug use Rate of domestic violence 1. How much did we do? 2. How well did we do it? 3. Is anyone better off? = Customer Results = Customer Results

24. % Common measures e.g. client/staff ratio, unit cost, clients from a target group, staff turnover, staff morale, staff fully trained, worker safety, client satisfaction (were they treated well), wait time, wait lists/turnaways, meeting specific industry standards % Skills / Knowledge (e.g. parenting skills) # % Attitude / Opinion (e.g. toward drugs) # % Behaviour (eg drug and alcohol use) # % Circumstance (e.g. working, in stable housing) # QUANTITY QUALITY How well did we do it? How much did we do? # Clients/customers served EFFORT % Activity-specific measures # Activities (by type of activity) e.g. timely, clients completing activity, correct and complete processes, meeting standards Is anyone better off? EFFECT Source: Mark Friedman 2005; Anglicare 2011

25. How Population &Performance AccountabilityFIT TOGETHER

26. THE LINKAGE Between POPULATION and PERFORMANCE POPULATION ACCOUNTABILITY General Physical HealthPrevalence of liver diseaseStable Families and RelationsRate of alcohol and drug useMental HealthRate of co-occurrence with mental health issues PERFORMANCE ACCOUNTABILITY POPULATIONRESULTS Contributionrelationship Alignmentof measures Residential Rehab Services % receiving assessment withinagreed timeframe # ofbed days Appropriateresponsibility % increased awareness of the impact of alcohol and drug use on physical health # increased awareness of the impact of alcohol and drug use on physical health Source: Mark Friedman 2005

27. 2. Developing Outcomes

28. Key Assumptions Partnership and collaboration between agency, organisation and individuals, families to identify community outcomes Appropriate consultation framework and engagement strategies with key stakeholders Appropriate mechanisms in place (formal and informal) to capture the information required to get the outcomes and time frames right The Request This is the stage of developing and identifying outcomes which is outside of scope for this workshop but is the important precursor to defining and writing outcomes

29. Outcomes - different levels and purposes Individual outcomes for people with alcohol and drug issues Family outcomes Program or Service level outcomes Organisational outcomes Corporate – Financial; Human resource; Risk Community outcomes Population outcomes procurement

30. Types of Levels

31. Key considerations: • Statement of the desired change or result (what does it look like?) • Interpretation – get clarity and agreed understanding across stakeholders • Be clear about target group (eg child, mother, parent) • Outcomes – at what level (eg individual, service level, population) • What are you trying to assess? • Measurable and meaningful; reasonable number Outcomes – Responding to a Request

32. Key considerations: • Attribution – circle of control and influence; proximal and distal • System – how does this fit within the whole? Communities of practice • Think about time frame – short term to long term; longitudinal change • Consider context eg chronic relapsing condition • Outcomes in the AOD sector are generally about results/effects or change in knowledge or skills; attitude or opinion; values; behaviour; circumstance; condition or status Outcomes – Responding to a Request

33. SMART objectives – specific, measurable, achievable, results oriented, time frame A useful approach to writing an outcome statement

34. 3. Measuring Outcomes

35. Indicator - is a measure for which data is available which provides a specific and reliable means to quantify the achievement of the outcome. These results indicators are what we will see if change occurs. Ask the questions: What will we see if change occurs? What will be the evidence? Each outcome can have a set of indicators, which, if taken together gives a reasonable approximation of the achievement of the outcome or result

36. Indicator – can be expressed as a # or % (quantifies the achievement of the outcome) Eg: The percentage of participants reporting increased awareness about alcohol and drugs The number of people who have secured stable housing since the programme’s commencement The percentage of mothers who have retained primary care of their children since the program’s completion

37. Sometimes, one indicator may be sufficient to evidence the outcome’s achievement. Often, several indicators are required. Select carefully to be meaningful and not over burdensome. Outcome: SOBER UP CENTRE – Clients are kept safe overnight Reduction in clients attending A&E depts in hospital Reduction in complaints about public drunkeness Reduction in incidence of domestic violence Reduction in street arrests of clients

38. About measurement tools There is no magical measurement tool! There are a range of tools (eg. clinical profiling tools, instruments, surveys and questionnaires etc) that can be used but they all have limitations Develop an “evidencing the outcome” approach that is context specific, in collaboration with the sector organisations, individuals and families which can give a reasonable approximation of the achievement of the outcome There is always a story, assumptions and a context which needs to be explicitly stated

39. THIS IS A JOURNEY – CONSIDER THESE AND WORK TOWARDS Reliability refers to the extent to which a measure produces accurate information. Validityrefers to the extent to which the measure actually measures what it is intended to measure. Suitability refers to the criteria for selecting measures that meet the target group or intent – and are appropriate for those administering it Relevancy refers to measures clearly related to the outcome being evaluated, providing useful information and allowing meaningful conclusions to be drawn

40. We need a baseline (pre-post)

41. Outcomes, Indicators, Data Linkages and relationship, alignment, what is important, reducing administrative burden Evidence based practice

42. SUMMARY OF THE DAY 1. An overview of outcomes 2. Developing outcomes 3. Measuring outcomes

43. QUESTIONS

44. Consultants to Government and Community