Evaluation is a professional and ethical responsibility and is a core part of PHN professional practice

2. Evaluation Process Evaluation • Evaluation is a professional and ethical responsibility and is a core part of PHN professional practice • Commitment to evaluation helps build the PHN intelligence about • what interventions work • in what context • why and how • Evaluation findings helps improve practice to be more effective and efficient Process Evaluation

3. Process Evaluation

4. Evaluation • Evaluation determines the extent to which an intervention has achieved the desired health outcomes and assesses the contribution of strategies used • Key reasons for evaluation PHN interventions is to assess and improve intervention: • Effectiveness – has the intervention worked • Efficiency – relative effectiveness compared to other interventions • Efficacy – effectiveness under ideal circumstances • Economic Impact – cost-effectiveness and use of resources • Intelligence – inform future planning and theory building • Accountability – justify resource allocation and use Process Evaluation

5. Action Statements and Evaluation Levels Process Evaluation

6. Levels of Evaluation • There are several different levels of evaluation in PHN practice: • Formative Evaluation - data collected prior to intervention implementation which is used to inform intervention design and assess capacity • Process Evaluation – assessed the intervention strategies and capacity building strategies • Impact Evaluation – measures whether the intervention objectives have been met • Outcome Evaluation – measures whether the intervention goal has been met • Economic Evaluation – measures cost-effectiveness of the intervention or intervention strategies Process Evaluation

7. Qualitative V’s Quantitative methods • The two main forms of data gathering used in evaluation include: qualitative and quantitative • Quantitative methods focus on numeric data that can be statistically analysed and can test the extent to which an intervention causes change in health status, health behaviour, knowledge, attitude etc • Qualitative methods attempt to determine the meaning and experience of the intervention for the target group and other participants • Good quality evaluation usually has components of both qualitative and quantitative methods Process Evaluation

8. Process Evaluation • Process evaluation assesses intervention implementation and is concerned with • Intervention exposure – extent target group are engaged or aware of PHN problem • Reach – proportion of target group who participate • Participant satisfaction – whether participants are happy and like the intervention activities • Delivery – whether activities are implemented as intended • Fidelity – assessing performance of intervention materials and components • Contextual aspects – aspects of the environments that influence the intervention implementation Process Evaluation

9. Process Evaluation • Both quantitative and qualitative methods are used in process evaluation: • Quantitative methods measure reach, delivery and exposure aspects of the intervention • Qualitative methods assess participant satisfaction, fidelity and context elements of intervention delivery. • Process evaluation provides rapid feedback on the quality and integrity of the intervention – useful management tool • Process evaluation is relatively low cost and is a useful quality assurance tool Process Evaluation

10. Evaluating education materials • Several tools exist for evaluating education materials: • Standard protocol for leaflets and audiovisual materials - considers: attraction, comprehension, acceptability, personal involvement, persuasion • SMOG test - Formula for readability by calculating the number of polysyllabic words • Group leader performance - true/false questionnaire completed by participants Process Evaluation

11. Key methodological components to consider in process evaluation Process Evaluation

12. Process indicators • Evaluation indicators are the criteria against with data or observations are assessed for judgement of intervention success or failure • Evaluation indicators may come from : • Historical comparisons with similar efforts in the past • Comparisons with contemporary activities • Professional consensus – using the above and professional judgement • Finding comparison data may be difficult due to a lack of published results hence collective professional judgement should be applied Process Evaluation