Measuring Evaluating The Progress of the Patients for Patient Safety Initiatives: Nationally and Locally

1. Measuring & EvaluatingThe Progress of the Patients for Patient Safety Initiatives: Nationally and Locally FOR TOOLKIT FOR MEASUREMENT & EVALUATION Developed and Submitted by Katharina Kovacs Burns, MSc, MHSA, PhD First Presented at Patients for Patient Safety Workshop October 18, 2006 in Vancouver, B.C. As the title indicates, this presentation is about measuring and evaluating a specific initiative, which in this case happens to be the Patients for Patient Safety Canada initiative. This initiative has national and local aspects to consider which actually stem out international efforts. The main purpose of this presentation slide deck is to discuss the components of program or initiative planning which builds in or includes measurement and evaluation. Although the initiative selected here is specific, the presentation can be provided for other initiatives as well. The concepts are generic and transferable across many programs, initiatives or interventions. More detailed notes accompany this presentation � Toolkit for Measurement & Evaluation with Applied Example form Patients for Patient Safety Initiative. As the title indicates, this presentation is about measuring and evaluating a specific initiative, which in this case happens to be the Patients for Patient Safety Canada initiative. This initiative has national and local aspects to consider which actually stem out international efforts. The main purpose of this presentation slide deck is to discuss the components of program or initiative planning which builds in or includes measurement and evaluation. Although the initiative selected here is specific, the presentation can be provided for other initiatives as well. The concepts are generic and transferable across many programs, initiatives or interventions. More detailed notes accompany this presentation � Toolkit for Measurement & Evaluation with Applied Example form Patients for Patient Safety Initiative.

2. Katharina Kovacs Burns, PhD 2 OVERVIEW Describing the components of program/initiative planning, �measurement� and �evaluation� Discussion of: Importance of measurement & evaluation Components that need to be in place or developed Framing a plan/�logic model� for measuring & evaluating the progress of Patients for Patient Safety Initiatives This presentation will cover the key areas as bulleted. This presentation will cover the key areas as bulleted.

3. Katharina Kovacs Burns, PhD 3 DESCRIBING THE COMPONENTS OF PROGRAM PLANNING, �MEASUREMENT� & �EVALUATION� Section I of the presentation is defining and/or describing the components of program planning, �measurement�, and �evaluation�. Section I of the presentation is defining and/or describing the components of program planning, �measurement�, and �evaluation�.

4. Katharina Kovacs Burns, PhD 4 PROGRAM PLANNING Elements of a program/initiative plan: Purpose & goals Resources (inputs) Strategies/activities to achieve goals Products/services as a result of activities (deliverables or outputs) Targeted audience(s)/ recipients of outputs Anticipated outcomes or end results There are standard or recognized components or elements identified with a program plan. The key element of a program plan is always going to be the purpose and goals of the program, without which it is difficult to impossible to build the rest of the plan elements as noted in the slide. Each will be described in more detail in this presentation.There are standard or recognized components or elements identified with a program plan. The key element of a program plan is always going to be the purpose and goals of the program, without which it is difficult to impossible to build the rest of the plan elements as noted in the slide. Each will be described in more detail in this presentation.

5. Katharina Kovacs Burns, PhD 5 CLEARLY DEFININGELEMENTS Before a Program/Initiative Plan can be built, the elements need to be clearly defined or well understood i.e. What are the �goals�, �inputs�, �outputs�, and �outcomes�? What planning model template will be used to frame these elements? e.g. Logic Model Will the selected model facilitate �measurement� and �evaluation� of the program/initiative? Goals: Overall purpose of initiative/ project/program; goals help to steer a clear direction for the initiative/project/program, and reflect priorities of the organization or group. E.g. from Patients for Patient Safety Canada initiative: �communicate our expectations for disclosure on medical errors�, or �incorporate the patient experience into patient safety research�. Inputs: Resources that are needed in order to ensure that activities for goals can be implemented. What do we need to provide or ensure is in place in order for the activities to be accomplished and for our goals to be achieved? E.g. financing/funding, grants, time of volunteers, patient stories Outputs: The direct and measurable deliverables (services or products) of the initiative/project�s strategies or activities; often outputs are singular or a specified number e.g. one website or #workshops or #presentations; includes not only activities but also target audience for whom deliverables are intended or would benefit Activities: Tasks or strategies or actions to implement initiative and to achieve goals and outcomes; are intended to produce deliverables to a target audience; reasonable tasks to achieve. What do we have to do to ensure our goals are met? E.g. establish website�.; set up conference/workshop�Target Audience: to whom are the activities and deliverables, or products and services, for? Who benefits from activities? Who needs to participate, be involved or reached? E.g. patient and consumer groups; health care professionals, decision makers Outcomes: Formula for outcomes is �who will do what�; can be individual/ patient focused, family or community focused, organizational/institution focused, or system focused. Good outcomes are achievable, realistic, directly related to service, based on the needs/desire of the population served, consistent with the mission, linked to research, & measurable. Short-term outcomes � results that can be achieved in a relatively short period of time (6-12 months) and link logically to intermediate and long-term outcomes; easier to accomplish and easier to measure because they are learning or knowledge changes. E.g. Patients and consumers publicly talk about disclosure statement and key messages. .Intermediate Outcomes � results that will lead to the long-term outcome, but cannot be accomplished until initial outcomes have set the stage for attainment. These outcomes are usually behavioral changes. E.g. Health professional and patient education materials incorporate disclosure statement. Long-term Outcomes � The ultimate results, that are positive and beneficial and that an initiative, project or program desires for its participants; are more difficult to accomplish and more difficult to measure. These outcomes are changes in conditions as a result of sustained actions and behavior or attitude changes. E.g. Health professionals and institutions approve the inclusion of disclosure in their policies.Goals: Overall purpose of initiative/ project/program; goals help to steer a clear direction for the initiative/project/program, and reflect priorities of the organization or group. E.g. from Patients for Patient Safety Canada initiative: �communicate our expectations for disclosure on medical errors�, or �incorporate the patient experience into patient safety research�. Inputs: Resources that are needed in order to ensure that activities for goals can be implemented. What do we need to provide or ensure is in place in order for the activities to be accomplished and for our goals to be achieved? E.g. financing/funding, grants, time of volunteers, patient stories Outputs: The direct and measurable deliverables (services or products) of the initiative/project�s strategies or activities; often outputs are singular or a specified number e.g. one website or #workshops or #presentations; includes not only activities but also target audience for whom deliverables are intended or would benefit Activities: Tasks or strategies or actions to implement initiative and to achieve goals and outcomes; are intended to produce deliverables to a target audience; reasonable tasks to achieve. What do we have to do to ensure our goals are met? E.g. establish website�.; set up conference/workshop�Target Audience: to whom are the activities and deliverables, or products and services, for? Who benefits from activities? Who needs to participate, be involved or reached? E.g. patient and consumer groups; health care professionals, decision makers Outcomes: Formula for outcomes is �who will do what�; can be individual/ patient focused, family or community focused, organizational/institution focused, or system focused. Good outcomes are achievable, realistic, directly related to service, based on the needs/desire of the population served, consistent with the mission, linked to research, & measurable. Short-term outcomes � results that can be achieved in a relatively short period of time (6-12 months) and link logically to intermediate and long-term outcomes; easier to accomplish and easier to measure because they are learning or knowledge changes. E.g. Patients and consumers publicly talk about disclosure statement and key messages. .Intermediate Outcomes � results that will lead to the long-term outcome, but cannot be accomplished until initial outcomes have set the stage for attainment. These outcomes are usually behavioral changes. E.g. Health professional and patient education materials incorporate disclosure statement. Long-term Outcomes � The ultimate results, that are positive and beneficial and that an initiative, project or program desires for its participants; are more difficult to accomplish and more difficult to measure. These outcomes are changes in conditions as a result of sustained actions and behavior or attitude changes. E.g. Health professionals and institutions approve the inclusion of disclosure in their policies.

6. Katharina Kovacs Burns, PhD 6 MEASUREMENT Definition: Measurement is the identification of the expected quality, performance or practice, characteristics, quantity, or other attributes of a program/service/ initiative; depends on indicators. Types of Measurement: Objective/Quantitative Subjective/Qualitative Section 2 of this presentation covers measurement and evaluation in more detail, including their importance in program/initiative planning and implementation. The definition of �measurement� as condensed from several sources, is as indicated on slide. Type of Measurement: There are different types of measurement which can be conducted regarding the program plan components. These include: Objective or Quantitative measurement which is actual statistics or quantified data as in numbers of�. (or # of �), or percentages of items or people (%). Numbers or percentages are the hard facts as identified and published. e.g. 100% of patients agree with�..; or 25 of 40 patients agree with� Subjective/Qualitative: identification of personal stories, experiences, perceptions, comments, or other non-quantified information. Numbers and percentages are not part of this discussion. Often, there can be some common themes identified if a group of individuals are interviewed or a discussion of program aspects is captured.Section 2 of this presentation covers measurement and evaluation in more detail, including their importance in program/initiative planning and implementation. The definition of �measurement� as condensed from several sources, is as indicated on slide. Type of Measurement: There are different types of measurement which can be conducted regarding the program plan components. These include: Objective or Quantitative measurement which is actual statistics or quantified data as in numbers of�. (or # of �), or percentages of items or people (%). Numbers or percentages are the hard facts as identified and published. e.g. 100% of patients agree with�..; or 25 of 40 patients agree with� Subjective/Qualitative: identification of personal stories, experiences, perceptions, comments, or other non-quantified information. Numbers and percentages are not part of this discussion. Often, there can be some common themes identified if a group of individuals are interviewed or a discussion of program aspects is captured.

7. Katharina Kovacs Burns, PhD 7 What�s needed for measurement? Identifying Goals of the program/initiative e.g. Patient for Patient Safety Initiative Identifying Indicators/measures that reflect the outputs and outcomes of goals What does literature or research provide for indicators? Survey Analysis Group Identification of reasonable indicators - formal vs. informal approach Measurement of programs/initiatives can only be done if certain things are in place such as the goals of the initiative and other relevant outcomes for which indicators can be identified and measured. The indicators are identified in several ways � from the literature or research studies on the topic area, surveys to experts who will identify what they perceive to be the best indicators of successful achievement of each goal, and focus or other group discussions in which the indicators are identified. Focus groups are a more formal process than are informal dialogue or discussions amongst expert individuals. It is often a challenge to understand the relationship between the outputs and the outcomes. At times it takes more than a vivid imagination to see this relationship or linkage. Sometimes � a miracle! Measurement of programs/initiatives can only be done if certain things are in place such as the goals of the initiative and other relevant outcomes for which indicators can be identified and measured. The indicators are identified in several ways � from the literature or research studies on the topic area, surveys to experts who will identify what they perceive to be the best indicators of successful achievement of each goal, and focus or other group discussions in which the indicators are identified. Focus groups are a more formal process than are informal dialogue or discussions amongst expert individuals. It is often a challenge to understand the relationship between the outputs and the outcomes. At times it takes more than a vivid imagination to see this relationship or linkage. Sometimes � a miracle!

8. Katharina Kovacs Burns, PhD 8 E= mc�