Proposal and Program Evaluation

1. Proposal and Program Evaluation Ivan Yau

2. Critical Reflections • Different Purposes / Targetslead to different Strategies and Demands on rigorousness of program evaluation – to public, visual documentation and stories can be used effectively (have a sense that these are representative), to academic and funder, generally, statistical analysis is used (but should have meaningful no.!!)

3. Critical Reflections • Poor in Program Evaluation in general so far!! – these can be reflected in program evaluation published with poor objectives, evaluation methods and its interpretation. So what are your roles as social workers?

4. Critical Reflections • Evaluation on Program Effectiveness are linked to ‘accountability’ and ‘monitoring’ both. If impact / effectiveness can be demonstrated (accountability), our service can only be monitored by output. • Program Evaluation is a progressive and goal-driven process, not a repetitive routine measure, e.g. after a rigorous evaluation on effectiveness, only certain kind of QC or a more in-depth evaluation is required.

5. Further development

6. Some examples • Hong Kong Family Welfare Society (1989) A research report on cognitive-behavioral Group Therapy. - further developed into more refined program on clients with more severe M.I. symptoms, parent group and other with more skill deficits • Chronic Disease Self Management Program (CDSMP) – more vigorous screening is required to ensure the commitment in the program and reduce drop-out rate

7. Critical Reflections • Evidence – biased preference • Numbers – even converting subjective experience into Numbers!! • Counting and Rating • Statistical Analysis (T-test, sign-test,…logic of significant level and hypothesis testing)

8. Program Logic One • Linear (common, but oversimplified) Problem Intervention Objectives Evaluation

9. Undesirable Consequences Theory Application Direct Observation Problem Analysis Target Mechanism of Change Interviews and Hearsay Literature Review Practice Wisdom Program Logic Two • Holistic View: Analysis of Target and Problem, Mechanism of Change (Intervention), Objectives, Evidence and Expected changes

10. 目標 目標 需要 需要 需要 個人/ 環境 技巧/ 策略 技巧/ 策略 技巧/ 策略 個人 問題 Program Logic Three • From SSLD perspective:

11. Program Evaluation • Objectives: • Change • Important • From SW concern/ perspective!! • Effective? - Evidence (changes observed) is consistent with the objective.

12. Illustration – Program Logic • 跨越鴻溝 – 促進親子溝通計劃（2001） • 針對大埔區青少年不斷離家出走的現象 • 合作伙伴：警方（轉介） • 潛伏危機：受誘惑而去吸毒、販毒、賣淫 • 服務目標所受的影響：減少再次離家出走的情況 • 社會控制的議程：加派警力巡查、對離家出走的青少年/家長提出警告、向多次離家出走青少年的家庭提出保護令的申請、……

13. Illustration – Program Logic • 現象分析：親子缺乏溝通是一般離家出走少年的起源？ • 理論：青少年成長理論、家庭發展歷程、…… • 進一步探索： • 青少年離家後在甚麼地方流連？有何活動？跟甚麼人連繫？（滿足甚麼需要？） • 這些青少年的家庭狀況？他們對家庭的觀感？家庭關係？（家庭無法滿足他們甚麼需要？）

14. Illustration – Program Logic • 一些可能的需要： • 進一步探索： • 社群連繫（歸屬感、關顧、認同…) • 刺激、歡樂、成功感、… • 滿足需要的其他途徑……?

15. Program Plan • What are common problems in program / session plan?

16. Program Plan • Common Problems • Linear without contingent thinking • Not aware of mechanism of change

17. Program Plan

18. Mechanism of Change • Some Illustrations • Skill training–Parenting skill training as an example  structured exercise, brief lecture, demonstration, role-play and feedback, experimentation in real life • Cognitive restructuring – getting familiar with ABC (differentiating DT /DT, aware of undesirable impact of DT), identifying own DT, disputing these through empirical enquiry, self-restructuring process

19. Mechanism of Change • Some Illustrations • Self Understanding / acceptance – self-exploration on critical incidents in trustful environment, feedback from others after intensive interaction, connecting with others deeply and comparing the difference • 生死教育 – reflection on the edge of life / death experience or stimulated experience

20. Program Plan

21. Program Evaluation Mechanism of Change Objectives Expected Changes Intervention

22. Some Clarifications • Intervention is different from Objective / Effectiveness. • Process review is useful for improving the practice. • Client’s satisfaction is Not the program effectiveness.

23. Objective • Usually viewed as ‘Prevention of Undesirable Consequence’ / ‘Alleviation of the (social) Problem’ • Important and Desirable – from who’s perspective? Client’s? • Influence from stake-holder? Theory-driven? Social Control?

24. Effectiveness and Change • Extent: Magnitude in relation to all participants (sometimes Magnitude alone is sufficient!) • Confidence: Change is not the result of chance • Causal relationship: can hardly be established

25. Intervention • Intervention must link to mechanism of changes (theory-driven and common sense!!) and desirable outcome. • Group proposal – brief session outline first, and further detailed session plan as the group evolves gradually.

26. Effectiveness I Participants’ change (pre-post, post only) • Issues of measurement: validity (face validity), reliability (items-correlation) and sensitivity • Changes reach statistically significant level • Single-item comparison • Scale construction: literature review, local validation, feedback from participants,…

27. Effectiveness II Outcome Indicator • Criteria for success: how come? • Baseline makes comparison feasible (indicating the change) • Meaningful (and significant) to reflect the accumulated effort, or indirect evidence to achieve the ultimate goal (community level intervention)

28. Different Levels – G & P • Group and Programs • Specific target group • Focused, limited intervention • Operationalization of Objectives – pre-post measures, post-measures • Use of statistical analysis to assess significant changes

29. Different Levels - Project • Project • Specific target group with rather long-term intervention • Focus on several key outcome indicators • Pre-post measures are also feasible

30. Different Levels - Service • Service • Diversified target groups with long-term intervention and general / vague objectives (IFSC, DEC,…) • Aggregated data is always useful • Focused on major groups, or use of randomizations if necessary

31. Political Agenda • Making use of client’s views in advocating for changes, e.g. IFSC,CSSA,SWfield work supervision, 自力更新服務 • Connecting impact of the program with advocacy / policy change,e.g. 類風濕性關節炎自我管理課程、中學義工服務、小學學校社工服務、兒童之家服務、…

32. Rounding Up • Aggregate data (client’s satisfaction, objective reached, target complaints, etc) is useful for service evaluation, key outcome indicator (decrease in A & E of child injury due to child neglect, decrease in suicide) for project evaluation, while accumulated impact (make use of community resources by NA, support network, no. of high risk case receiving service) for both.

33. Rounding Up • Evaluation should be better linked to continuous improvement (don’t repeat the evaluation every year), service development (promotion of a better service model) or social movement (advocacy) (e.g. changing concepts of chronically ill patient – ownership of health, instead of replying on medical treatment).

34. Rounding Up • Evaluation can be and should be Political – particularly help marginalized group to voice out their views / needs. • Social workers should: • Be able to link up the objective, intervention and outcome. • Be quipped with themselves of different ways / strategies for program evaluation.

35. Reference • 陳錦棠（2006） 社會服務成效評估：程序邏輯模式之應用. 香港：香港基督教女青年會 • 羅致光等編（1999）靑少年成長與活動案例. 香港：香港基督敎服務處。 • 羅致光主編（1992）靑少年中心活動成效評估 : 方法與案例. 香港：香港基督敎服務處 • United Way of America (2006) Measuring Program Outcomes: A Practical Approach