EVALUATION PLANNING Legal Services Corporation May 19, 2004

1. EVALUATION PLANNING Legal Services Corporation May 19, 2004 Please standby while others connect

2. Goals for the Session • Review the LSC evaluation plan template • Learn how to formulate strong evaluation questions • Introduce LSC’s evaluation questions • Give you the skills to understand which data collection techniques are best suited to answering what kinds of evaluation questions • Gain a better understanding of how program assessment can help legal services organizations improve services

3. Objectives • At the end of the training grantees will be able to: • Complete the LSC evaluation template • Integrate LSC’s required evaluation questions with questions you develop on your own • Deploy a data gathering strategy that helps you answer the questions you created

4. Ground Rules • Using Isoph Blue’s features • We ask that you not use a speaker phone unless you have a mute button to block external sounds. • Please do not use the “Hold” function on your phone, as it can induce a loud beeping noise for the rest of us on the call.

5. Questions During the Session • If you can, please hold your questions till we get to a question break • If you can’t wait, please click the question mark and type your question into the window, we’ll answer as soon as we can. • If we do not get a chance to answer your question, please email it to Ehren Reed at ereed@innonet.org.

6. What is Evaluation? The systematic collection of information about a program in order to better understand it, improve its effectiveness, replicate it and/or make decisions about future programming.

7. Benefits of Evaluation • Effectively communicates successes • Builds common understanding about the program • Helps you improve what you do • Turns shortcomings into opportunities • Helps staff overcome obstacles • Positions your organization as a learning organization

8. 10 Steps to Evaluation • Develop program goals • Identify specific goal targets • Outline strategies and activities • Develop evaluation questions • Create data collection instruments • Collect data • Analyze and interpret information • Present findings • Reflect on findings • Revise program & institute changes

9. Evaluation Plan

10. Project Goals • Clear statements identifying the ultimate results you want to achieve

11. Goals & Objectives • Develop objectives or more specific targets for your goals. • Consider the major steps will you need to accomplish in order to achieve each of them. • Make your objectives specific and achievable. • Some goal words: • Enhance • Establish • Enable • Tip: Review your grant proposal to see what you said you would accomplish

12. Some Intake Examples Intake • Goal: • Enable Asian-American clients in northern Virginia to seamlessly access legal services through a single, integrated intake system • Objectives: • Design integrated system that allows partner organizations to access critical client information

13. Some Infrastructure Examples Infrastructure • Goal: • Enhance the delivery of legal services to low-income families in rural Mississippi via the website through high quality web applications • Objectives: • Develop partnerships with providers and case managers to identify critical needs • Increase access to services by integrating communication systems

14. Some Pro Se Examples Pro Se • Goal: • Enhance the ability of poor people in Ohio to access quality legal services to assert their rights and interests through online and in-person training and technical assistance • Objectives: • Identify patterns and causes of fundamental legal programs affecting our client community • Develop and test solutions to recurring legal problems faced by our client community • Ensure appropriate and sustainable web-based access to legal services

15. Strategies & Activities • The major actions a program takes to achieve desired outcomes • Use action words like: • Conduct • Train • Mentor • Provide • Support

16. Some Intake Examples Intake • Develop database program that gathers case information and assesses eligibility for legal aid • Develop process for integrating systems and databases in various offices that conduct intake, provide legal advice, and represent clients • Train case workers to use the system • Identify interpreters for mono-lingual clients

17. Some Infrastructure Examples Infrastructure • Identify potential partners • Hold regular meetings with partners and document community needs • Develop web-based services • Train providers in how to use web-based services

18. Some Pro Se Examples Pro Se • Prepare training materials for template developers • Participate in national template developers training • Prepare logistics for Ohio template developers training • Conduct Ohio template developers training • Organize multi-state template developer work group

19. Questions

20. Work on Strategies and Activities • Take a look at your grant proposal or work from memory and complete the first two columns of the LSC Evaluation Plan grid. • List your activities and organized by category or strategy. Remember to use appropriate action words. • When you’re done, click in the check box next to you name

21. Check in / Debrief / Discussion

22. Evaluation Questions • Identify broad questions that will guide your evaluation report. • Process questions: What did you do and what were the tangible results of your work? • Outcome questions: What change occurred among your constituents as a result of your activities? • Try to keep the questions open-ended

23. Some Intake Examples Intake • How has the intake system assisted clients in accessing seamless services? • To what extent has the new system increased efficiency among intake staff?

24. Some Infrastructure Examples Infrastructure • To what extent has the website helped strengthen client education? • How have the partnerships supported the development of quality content for the website? • How has the website increased access to legal services for rural clients? • In what ways has this project been supported by LSC and the organizations partners?

25. Some Pro Se Examples Pro Se • To what extent did the training provide necessary skills and knowledge to achieve intended outcomes? • Was the training curriculum and delivery offered of high quality? • How well have the online services met the needs of the community and addressed the current trends in necessary legal assistance? • Has our online process and materials adequately prepared clients to accurately complete their legal forms?

26. What are Your Questions? • What information do you want to know about your program? • Think about the assumptions that you have made about how your program will be planned and implemented. • Draft questions that you want to answer about your program. • Tip: Avoid Yes/No questions. Instead, ask when, how, how much or to what extent

27. Check in / Debrief / Discussion

28. Evaluation Data • Talk to people • Interview, focus group, facilitated discussion • Gather written responses • Surveys, questionnaires, paper tests • Observe • Individuals, communities, group activities • Review documents • Attendance records, progress reports, journals

29. ….and the source • Where will you find the information? • What target group or documents will you need to gather the necessary information?

30. Some Intake Examples Intake • Survey of intake staff • Focus Group with mono-lingual clients • Exit survey of mono-lingual clients

31. Some Infrastructure Examples Infrastructure • Monthly update form completed by project staff • Survey of rural clients/users • Survey of case workers • Content review of website

32. Some Pro Se Examples Pro Se • Survey clients • Document review/Content review of online process

33. Finish Your LSC Evaluation Plan • List the data collection methods and source of information for each one of your evaluation questions • Remember to keep in mind the following questions: • Who or what has the answer to each evaluation question (partners, clients, records, etc • What is the best way to access it (survey, focus group, document review, etc) • Tip: Think about what information you are already gathering and try to integrate those tools into this data collection effort.

34. Evaluation Plan

35. Check in / Debrief / Discussion

36. Questions

37. Next Steps • Let us know where you are in the process • Have you completed your plan? • Do you want us to review it? • Are you ready to create data collection instruments?

38. We can provide more assistance • Asynchronous training • Individual technical assistance • Synchronous training • Potential topics include: • Survey • Document review

39. For More Information Innovation Network, Inc. 1001 Connecticut Ave., NW, Suite 900 Washington, DC 20036 (202) 728-0727 www.innonet.org