Challenges in evaluating child & family welfare services

1. Challenges in evaluating child & family welfare services Robyn Parker, Myfanwy McDonald, & Jenny Higgins Association of Children’s Welfare Agencies Conference, 2-4 August 2010, Sydney

2. Context • Evaluation of programs becoming more of a reality • Clients’ right to receive high quality services • Growing need to justify expenditure • Only way to know is through evaluation • Not new – we do it every day, just not in a formal & systematic way AUSTRALIAN INSTITUTE OF FAMILY STUDIES

3. Context • This session not about actually doing evaluation; more about principles and avoiding derailment • Sticking to the rules • Easing the way • Looking to the future AUSTRALIAN INSTITUTE OF FAMILY STUDIES

4. What is evaluation? • A systematic process to determine whether objectives met – did the client change or improve in the way we wanted them to, and was the program responsible for that change? • Involves making a judgment – is this program worth pursuing/funding, is the investment justified, to what extent is it doing what it was intended to do? AUSTRALIAN INSTITUTE OF FAMILY STUDIES

5. Why evaluate? • Did participants change significantly in the way the program logic suggested? i.e. did parenting confidence improve, is the child making progress, are newbie parents better equipped to cope with parenthood, do parents of toddlers employ positive parenting skills, does the child exhibit fewer challenging behaviours, etc... AND • To what extent was the program responsible for the change? AUSTRALIAN INSTITUTE OF FAMILY STUDIES

6. General process • Describe program in detail – objectives, methods, etc • Develop evaluation questions • Decide what to measure, when, where, how, who will be ‘measured’, & who will do the ‘measuring’ • Put procedures in place • Collect & analyse data • Interpret results, report & act on findings AUSTRALIAN INSTITUTE OF FAMILY STUDIES

7. Challenges (I) • Ruling out other explanations • Integrity of the evaluation – sticking to the rules • Evaluation design (pre/post test; comparison/control groups) • Program objectives – evaluation questions • Break it down – what is the pathway to results for participants? • What does the outcome “look like”? • “If program effective, client will ...” AUSTRALIAN INSTITUTE OF FAMILY STUDIES

8. Challenges (II) • Avoiding “wouldn’t it be nice to know ...” items (i.e. research) • Collect only what you need, use all that you collect • Integrating data collection into day-to-day practice • Synthesising data from different sources AUSTRALIAN INSTITUTE OF FAMILY STUDIES

9. Challenges (III) • Practice-based evidence requires input of practice-based professionals • Building the capacity of practice-based professionals to conduct evaluation • Building capacity requires investment – time, support &/or funding AUSTRALIAN INSTITUTE OF FAMILY STUDIES

10. Challenges (IV) • The data currently being collected may not produce the outcomes data you need (to say “it works!”) • Avoiding the influence of other agendas • Be prepared to be flexible AUSTRALIAN INSTITUTE OF FAMILY STUDIES

11. Strategies • Getting to “how we do things here”: • Planning; gradual introduction • Participation, skill development • Briefings & training in protocols • Reinforce through ongoing, active management support • Provides updates, interim reports • Acknowledge & accommodate impact on workload • Try starting small AUSTRALIAN INSTITUTE OF FAMILY STUDIES

12. Ethics – what an ethics committee will want to know • 4 basic principles: • Merit & integrity; beneficence; justice; respect. • Harm to participants • Minimising, dealing with distress • Obtaining informed consent • Parent, child, both? • Privacy & confidentiality • Limits (can’t always guarantee) • Disconnect in assessment of above - derailment AUSTRALIAN INSTITUTE OF FAMILY STUDIES

13. Further resources: • www.aifs.gov.au/AFRC [look for new resource published later this year] • www.aifs.gov.au/CAFCA [Evaluation Resource sheet] • www.nhmrc.gov.au [National Statement ...] AUSTRALIAN INSTITUTE OF FAMILY STUDIES

14. FINAL TIPS • PLAN • PLAN • REVIEW PLAN • PLAN • REVIEW PLAN • PLAN ... • RINSE AND REPEAT . . . . . . . . . AUSTRALIAN INSTITUTE OF FAMILY STUDIES