1 / 31

Performance measurement: Finding our way from outputs to outcomes

Performance measurement: Finding our way from outputs to outcomes. Finding our way from outputs to outcomes. How do we know we are having an impact on children, youth and their families? How can we come together to ensure that the services we offer are appropriate and effective?. Outline.

hayes-nguyen
Download Presentation

Performance measurement: Finding our way from outputs to outcomes

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Performance measurement: Finding our way from outputs to outcomes

  2. Finding our way from outputs to outcomes • How do we know we are having an impact on children, youth and their families? • How can we come together to ensure that the services we offer are appropriate and effective?

  3. Outline • Overview • Discussion: Knowing our audience • Example: Using CAFAS to track program and organizational outcomes • Discussion: Building capacity to do performance measurement

  4. Overview: Performance measurement • WHAT is it? • WHY is it necessary? • WHO benefits? • HOW is it done?

  5. What is performance measurement? “The regular collection of information for monitoring how a policy, program or initiative is doing at any point in time. It can be used to report on the level of attainment of planned results and on performance trends over time.” - Treasury Board Secretariat of Canada

  6. Why is it necessary? • “Serves as a descriptive tool” on how a project, policy or program is doing • Serves as an early warning if the direction of a program, policy or project is not going as planned

  7. Who is the audience? • End users: clients, families caregivers • Service providers, educators, program staff • Organization or network • Health system • Public at large

  8. PM: provides regular snapshots of how a program or policy is doing; focuses on what the outcome is Client satisfaction surveys are often a key indicator of PM Evaluation: can provide insight into how and why an outcome is occurring Client satisfaction is part of the process evaluation and can influence outcomes How is it different from evaluation?

  9. OUTCOMES ACTIVITIES SITUATION INPUTS OUTPUTS Quantity of work, products or participants Resources of a program Change in target audience Performance measurement and evaluation

  10. How is it done? • Program logic models • Balanced score cards • Strategy maps

  11. Balanced scorecard approach on health promotion Source: ICES, 2004

  12. Improve client focus of addiction services Improve access to appropriate addiction treatment Further develop & increase equitable resources and capacity Improve healthy behaviours, health promotion and disease prevention Reducing Risk through influencing the broader determinants of health Ensure the continuum of interventions includes prevention, health promotion, early intervention, harm reduction and treatment services. Ensure evidence informed practices are developed, implemented and maintained across province Increase productive use & appropriate allocation of resources across system Increase linkages, transition & integration within addiction services Improve health outcomes at the individual and population level Ensure quality assurance within the addiction system Increase availability and retention of the qualified human resources Improve linkages and transitions between addiction, mental health, health, education, social and justice systems Increase sustainability and equity of the addiction & health systems Addiction System Strategy Map Ontario Federation of Community Mental Health & Addictions Programs

  13. Child welfare performance measurement Source: OACAS QA Framework, 2004

  14. Types of performance measures • Outcome measures • Intermediate outcome measures • Process measures • Output measures • Input measures

  15. Strategic priorities by the Select Committee

  16. MCYS Strategic Goals, 2008-2012

  17. In small groups • Divide into 5 groups • In each group, discuss • What types of information your group/constituency are interested in (focus on two or three) • How frequently you want the information • How you want the information presented • What concerns, issues or challenges exist on these types of information • What solutions, ideas or strengths we can build on

  18. Challenges and opportunities • Selecting measures • Valid and reliable • Relevant, feasible, sensitive to changes • Developmentally and culturally appropriate • “Measures” database under development • Searchable database on measures used for evaluating CYMH programs

  19. Challenges and opportunities • Information management capacity within agencies and in the government • Collaboration and staff buy-in • Organizational culture that fosters learning

  20. Example: Use of CAFAS • The Child and Adolescent Functional Assessment Scale (Hodges, 2002) measures outcomes among 6-17 year old children and youth receiving mental health services in Ontario • CAFAS identifies level of functioning in 7 subscales: • Home, School, Community, Behavior towards others, Moods and emotions, Substance use, Thinking • Led by Dr. Melanie Barwick, Hospital for Sick Kids

  21. Implementation Supports Source: Barwick, 2009 22 (c) Barwick

  22. Severe Impairment on CAFAS Subscales at Entry to Treatment – years 2005 to 2008 (c) Barwick Source: Barwick, 2009

  23. Severity at Entry to Treatment for Ontario and Regions(2005: N=9,065; N= 2006: N=18,255; 2007: N=23,566; 2008: N=26,974) Source: Barwick, 2009 (c) Barwick

  24. Using CAFAS data for a local agency • Pull key variables & performance indicators from the CAFAS Quarterly Report • Enable comparison over time • Present data in a simple spreadsheet • Append charts to aid interpretation Source: Roger Rolfe, 2009. Utilizing CAFAS exports and reports at CTYS.

  25. Solution:Agency CAFAS Performance since 2006.xls Source: Rolfe, 2009.

  26. Sample Chart Source: Rolfe, 2009.

  27. Program-level Chart:GW Male/Female Improvement Scores Source: Rolfe, 2009.

  28. In small groups • What practices or strategies are successful in your organization/community: • For getting buy-in from all staff and/or senior management? • For getting consistent data and reporting? • For doing evaluation?

  29. Summary • Performance measurement is a useful tool for measuring improvements in mental health outcomes for children and youth • The usefulness of performance measures vary by audience needs and quality of the information • Strategies that enhance organizational learning are needed to build our capacity to do performance measurement

  30. Evangeline Danseco, PhD Director, Evaluation and Research 613.737.7600 Ext 3319 edanseco@cheo.on.ca Susan Kasprzak, MA Research Associate 613.737.7600 Ext 3320 skasprzak@cheo.on.ca Questions? www.onthepoint.ca

More Related