Self-Evaluation Toolkit

1. Self-Evaluation Toolkit Rosslyn Monro NACLC Conference August 2018

2. This workshop • CLCQ developed a Self-Evaluation Toolkit • Practical step through the toolkit • Toolkit is on-line – www.communitylegalimpact.org Self-Evaluation Toolkit |2

3. Theory of Change Theory of Change sets out the link between CLC activities and the achievement of medium and long-term outcomes. Self-Evaluation Toolkit|3

4. Theory of Change Pairs: Think of a client that stands out for you that no longer has contact with your service. Based on what you know about the client, paint a picture for your partner of the client’s best life. Talk about their: • health, • relationships • sense of safety and security • engagement in community life Self-Evaluation Toolkit |4

5. Theory of Change The story of your client’s best life are the outcomes or impacts. An evaluation seeks to trace our activities (inputs) with the client (stakeholder) to understand the link between our work and the client’s life (stakeholder outcomes) The evaluation toolkit tracks outcomes for 4 stakeholder groups: • Clients • Community partners • The justice system • Volunteers Self-Evaluation Toolkit |5

6. Theory of Change • “Without our service the Police would have more work to do” Outcome – savings to the justice system • “Our clients say they feel a weight has been lifted from their shoulders because they know their options” Outcome – reduced stress, better health, reduced reliance on health care system Self-Evaluation Toolkit |6

7. Theory of Change • Toolkit developed a Theory of Change with intermediate and long term outcomes Self-Evaluation Toolkit

8. Theory of Change – Intermediate Outcomes • The toolkit methodology predominately focusses on intermediate outcomes because they are: • More immediately identifiable • Measurable • Based on a program logic that if we achieve intermediate outcomes they will contribute to the longer-term outcomes which are harder to measure • As we develop our evaluation methodology we will work on developing ways to measure the long term outcomes Self-Evaluation Toolkit |8

9. Theory of Change – Intermediate Outcomes • Your turn: Workshop Activity 1 - Develop a summary of intermediate outcomes • Summarise the list of outcomes with no more than 2 phrases or sentence eg. Clients get the services they need eg. Volunteers have enhanced career prospects Self-Evaluation Toolkit |9

10. Theory of Change – Intermediate Outcomes for clients Self-Evaluation Toolkit |10

11. Theory of Change – Intermediate Outcomes for the community Self-Evaluation Toolkit |11

12. Theory of Change – Intermediate Outcomes for Justice System Self-Evaluation Toolkit |12

13. Theory of Change – Intermediate Outcomes for Volunteers Self-Evaluation Toolkit |13

14. Gathering Data Effective outcomes measurement is dependent on collecting the right data at the right frequency so that change can be tracked over time. Self-Evaluation Toolkit |14

15. Gathering Data – Toolkit overview • Toolkit has specific guidance about the data to collect and how to link it to the intermediate outcomes/domains • 3 main sources of data: • Hard Data – CLASS and other CLC statistical records • Survey Data – pro forma surveys for each stakeholder group and how to implement the survey process • Case studies – resources to guide writing case studies Self-Evaluation Toolkit |15

16. Gathering Data – Hard Data • Toolkit will update information about CLASS reporting as it becomes available which includes: • client demographics • activity types • description of services etc. • Many CLCs already produce this type of data for annual reports Self-Evaluation Toolkit |16

17. Gathering Data - Surveys • Each stakeholder group has a survey that tracks to the intermediate outcomes • Trial process • Email and/or telephone process on contacting and responding • Collation through Survey Monkey • Generally a good response rate • Smaller sample size compared to clients Self-Evaluation Toolkit |17

18. Gathering Data – Case studies • An effective case study describes how the work of the CLC illustrates the intermediate outcomes/domains for the stakeholder group. DOs • Choose individual stories that suggest that they are symptomatic of a bigger problem so that the individual is not the problem • Describe the impact of the problem on people’s lives and communities • Provide an overview of the activities that summariseswhy the CLC was best placed to help solve the problem, rather than another organisation • Include a testimonial • Describe the change from the stakeholder's perspective  • Refer to the major area of impact (intermediate outcome)and how it was specifically achieved in this situation Self-Evaluation Toolkit |18

19. Gathering Data – Case studies DON'Ts • Provide so much background information it dilutes the impact of the CLC work on the problem • Provide blow by blow description of the activities undertaken by the CLC • Use a testimonial as the outcome e.g. 'the client was happy that their legal problem was solved' is not an outcome that describes the change created.​ Self-Evaluation Toolkit |19

20. Gathering Data – Case studies • Example case study: A client presented at Local CLC. He had been to see a number of government and non-government agencies as he thought he was entitled to a disability support payment but was struggling with the paperwork involved. At Local CLC our solicitor took his instructions and provided him with advice about his entitlements. The solicitor worked with the client to find and collate all the necessary paper work and contacted Centrelink to negotiate on the client’s behalf, including providing Centrelink with the client’s medical records. The client was very happy when his disability support payment was confirmed by Centrelink. Self-Evaluation Toolkit|20

21. Gathering Data – Case studies • 3 elements of a case study: PROBLEM | SOLUTION | RESULT/OUTCOME PROBLEM | SOLUTION | RESULT/OUTCOME • Describe the problem within the broader social context and how the problem was a threat to commonly held values e.g. access to justice, voice, security, fairness, opportunity • Situate the problem as part of a bigger social problemeg. homelessness, violence against women, police targeting marginalised groups, mortgage stress • Describe whether and/or how a community partner was part of identifying the problem. Self-Evaluation Toolkit |21

22. Gathering Data – Case studies • 3 elements of a case study: PROBLEM | SOLUTION | RESULT/OUTCOME PROBLEM | SOLUTION | RESULT/OUTCOME • Describe the problem within the broader social context and how the problem was a threat to commonly held values e.g. access to justice, voice, security, fairness, opportunity • Situate the problem as part of a bigger social problemeg. homelessness, violence against women, police targeting marginalised groups, mortgage stress • Describe whether and/or how a community partner was part of identifying the problem. Self-Evaluation Toolkit |22

23. Gathering Data – Case Studies PROBLEM – ​SOLUTION – RESULT/OUTCOME Problem: Access to social security benefits ensures that people who have a disability can live with dignity and have their basic needs met. For people with a disability it can be challenging to negotiate a complex social security system to obtain social security benefits to which they are entitled. After approaching a number of government and non-government agencies for assistance to obtain disability support payment, Michael sought advice from Local CLC. He was experiencing high levels of stress about how to pay for his daily needs, as he did not have regular income. Self-Evaluation Toolkit |23

24. Gathering Data – Case studies PROBLEM | SOLUTION | RESULT/OUTCOME • Describe the activities that the CLC undertook and how they added value in addressing the problem • Highlight the quality and the relevance of the activities and how the activities were best suited to solving the problem Local CLC assisted Michael by contacting Centrelink and negotiating with them about Michael’s entitlement to income support. This involved providing documents to Centrelink that showed evidence of Michael’s situation. Self-Evaluation Toolkit |24

25. Gathering Data – Case studies PROBLEM | SOLUTION | RESULT/OUTCOME • Describe the changes or improvements that were made as a result of the CLC activities • The end of the case study should be able finish the sentence, 'Now that the problem is solved…‘ • Be able to answer the “so what” question from the stakeholder’s perspective As a result of Local CLC’s work, Michael is now in receipt of disability income support. Michael has been able to start paying off a back log of bills, and has said that: ‘Thanks to Local CLC, I feel less stressed and more in control.’ Presentation Title and Footer Information (change on Master page) |25

26. Gathering Data – Case studies PROBLEM – ​SOLUTION – RESULT/OUTCOME ​Access to social security benefits ensures that people who have a disability can live with dignity and have their basic needs met. For people with a disability it can be challenging to negotiate a complex social security system to obtain social security benefits to which they are entitled. After approaching a number of government and non-government agencies for assistance to obtain disability support payment, Michael sought advice from Local CLC. He was experiencing high levels of stress about how to pay for his daily needs, as he did not have regular income. Local CLC assisted Michael by contacting Centrelink and negotiating with them about Michael’s entitlement to income support. This involved providing documents to Centrelink that showed evidence of Michael’s situation. As a result of Local CLC’s work, Michael is now in receipt of disability income support. Michael has been able to start paying off a back log of bills, and has said that: ‘Thanks to Local CLC, I feel less stressed and more in control.’ Self-Evaluation Toolkit |26

27. Gathering Data – Case studies • Your Turn: Workshop Activity 2 • Problem • Solution • Outcome Presentation Title and Footer Information (change on Master page) |27

28. Reporting and Review Self-Evaluation Toolkit |28

29. Reporting and Review – Impact Reports • Impact Reports put all the data together and makes conclusions about the extent to which intermediate outcomes (domains) have been met or contributed to • Your turn – Workshop Activity 3 Using the pre-collated data for your stakeholder group write a few dot points on the issues that the data raises using the domains as your criteria. Self-Evaluation Toolkit |29

30. Reporting and Review – Impact Reports Our Impact on our clients Clients of Local CLC agree that we provide appropriate services that can assist with them resolving their legal problems and as a result they feel better equipped with how to seek help in the future. While some clients identified that they felt their legal problem was not resolved to their satisfaction, they did not attribute this to a problem with the services provided by Local CLC. The case studies illustrate that when clients have their legal problems resolved this can lead to greater impacts upon their lives such as improved family relationships, freedom from debt, and access to an income. In the coming 12 months we look forward to working towards further increasing our access to the community through the development of a Reconciliation Action Plan. Self-Evaluation Toolkit |30

31. Reporting and Review – Impact Reports Our impact in our community Local CLC works closely with community organisation partners to ensure that members of Local community get the legal services they need when they need them. Referral to and from our community partners are a vital part being responsive to the community’s legal needs and ensuring that the people who most need our help have priority access to our services. There is some future work in enhancing Local CLC’s warm referral process to community partners to ensure clients get the extra support they need. Local CLC provides community provides community legal education activities and resources to the Local community and community partners. Our legal education program is designed to improve the legal education and knowledge of the community and our partners, so that our community can avoid legal problems in the first place, or get the legal help they need. The evidence indicates that this is best done with community partners that work most closely with the clients that need our services and in the next 12 months we look forward to developing our CLE program in consultation with them. Self-Evaluation Toolkit |31

32. Reporting and Review – Impact Reports Our impact on the justice system This year, Local CLC has contributed the efficiency and effectiveness of the justice system by delivering services to our community that help people navigate the justice system or avoid entering the justice system in the first place. We do this through targeted service delivery that has direct impact on the effective operation of the Magistrates and Family Courts and by conducting community legal education that helps people avoid legal problems. A majority of our justice colleagues in courts and tribunal have said that our work assists them to ensure that the justice system works more effectively and efficiently. This year Local CLC has been effective in influencing public and legal policy in areas that affect our community. Half of our justice system colleagues are of the view that we help them to better understand the issues faced by our client and how the justice system addresses them. Through strategic litigation Local CLC changed the interpretation of workplace discrimination law which will assist many more future claimants facing illegal treatment. As part of the Fund Equal Justice campaign, Local CLC contributed to the reversal of the funding cuts to CLCs across the country. In the coming year we will continue to work with Queensland Consumer Alliance on pay day lending and work with decision makers about making tenancy law fairer for tenants. Self-Evaluation Toolkit |32

33. Reporting and Review – Impact Reports Our impact on volunteers Overall, Local CLC is able to leverage a substantial amount of resources for the community through harnessing the energy, enthusiasm and commitment of our 80 volunteers. Many of the volunteers, through the survey, have indicated that their volunteering experience at Local CLC provided positive benefits to their professional development, particularly for students and professional early in their career. The challenge for Local CLC in the future is to ensure that we continue to engage experienced volunteers and that their volunteering experience at Local CLC is professionally fulfilling and productive. Self-Evaluation Toolkit |33

34. Reporting and Review – Service Review • During the process of looking at the data from the last exercise and summarising the outcomes there were probably some questions or gaps that came up. Your turn – Workshop Activity 4 Look at the data again and write a list of: • questions or areas for more information that would better understand the impact of the service or • areas for more work in the future as the data is showing some gaps Self-Evaluation Toolkit |34

35. Reporting and Review – Service Review The toolkit has a number of service review questions for each stakeholder group The service review questions can help with: • gathering better information • strategic and operational planning • developing new services or • changing existing services Self-Evaluation Toolkit |35

36. Reporting and Review – Service Review Self-Evaluation Toolkit |36

37. Reporting and Review – Service Review Self-Evaluation Toolkit |37

38. Next Steps • We need you! • Evaluation work helps CLC sector tell the positive stories of CLC to funders, politicians, and the community: • NPA Review • More funding! • Share your impact reports with peaks and the community – they can form the basis for your annual report Self-Evaluation Toolkit) |38

39. Presentation Title and Footer Information (change on Master page) |39