1 / 31

Evaluating Your Program: Developing Successful and Sustainable Programs

Evaluating Your Program: Developing Successful and Sustainable Programs. OJJDP New Grantee Orientation April 6–7, 2010 Washington, DC CSR, Incorporated. Outline. What is Program Evaluation? The Logic Model Evaluation Questions Types of Program Evaluation Data Collection and Analysis

hiero
Download Presentation

Evaluating Your Program: Developing Successful and Sustainable Programs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluating Your Program: Developing Successful and Sustainable Programs OJJDP New Grantee Orientation April 6–7, 2010 Washington, DC CSR, Incorporated

  2. Outline • What is Program Evaluation? • The Logic Model • Evaluation Questions • Types of Program Evaluation • Data Collection and Analysis • Evidence-Based Programs

  3. Program Evaluation • Objective measurement and systematic analysis to determine the manner and extent to which a program achieves its intended goal. Definitions adapted from the OMB Circular no. A-11 Part 6, Section 200 http://www.ojjdp.ncjrs.gov/grantees/pm/glossary.html

  4. What Is Program Evaluation? • There are many ways to define program evaluation • Evaluation can be defined as the systematic process of collecting and analyzing data in order to: • Determine whether program objectives have been achieved; • Make a decision about the program; • Improve the delivery of services. Boulmetis, J., & Dutwin, P. (2002). The ABC’s of Evaluation. San Francisco: Jossey Bass.

  5. Why Evaluate Your Program? • Satisfy funding agency’s grant requirements • Document program accomplishments • Justify program • Improve program activities and services

  6. Why Evaluate a Program? • Evaluation results provide useful feedback to: • Sponsors and Funders • Program Staff • Clients • Donors • The Larger Community

  7. First Steps in Program Evaluation • Construct a logic model • Identify performance measures/outputs • Identify outcomes

  8. Logic Model: Definition • “Tool used to visually describe the linkages between program goals, activities, and expected outcomes.” Definition from: http://www.jrsainfo.org/jjec/resources/definitions.html

  9. Logic Model Activities Outputs Outcomes Conditions (Needs or Problems) What services will the project offer?(e.g., tutoring, after school sports, mentoring) How will you measure project services? (e.g., number ofyouth served) How will you know thatthe community need has decreased? (e.g., youth have improved attitudes, increased self-esteem; youth delinquency or substance use rates have decreased; youth school attendance improved) What community need will the project fill?(e.g., youth need for positive social outlets)

  10. Developing a Framework Activities Outputs Outcomes Conditions (Needs or Problems) What services will the project offer?(e.g., tutoring, after school sports, mentoring) How will you measure project services? (e.g., number ofyouth served) How will you know thatthe community need has decreased? (e.g., youth have improved attitudes, increased self-esteem; youth delinquency or substance use rates have decreased; youth school attendance improved) What community need will the project fill?(e.g., youth need for positive social outlets) High rates of truancy among elementary students (K to 5th grade) Hours of tutoring Number of youth tutored Number of support sessions provided Improved attitudes toward school Improved self-esteem Improved school attendance After-school tutoring Services of a familysupport coordinator

  11. Developing a Framework ADVANCED MODEL ADVANCED MODEL ASSUMPTIONS: Principles, evidence-based approaches, and theories that guide the program INPUT ACTIVITIES OUTPUTS OUTCOME IMPACTS Advance Model will go here Situation Analysis PrioritySetting Problem Identification Mission Goals Resources and Contributions. Tasks conducted by grantee’s staff, subcontractor, or volunteers. Activities are directly linked to outputs. Products and services delivered. Changes in individuals, agencies, systems, and communities. Outcomes may be intended or unintended. Initial Learning Awareness Knowledge Attitude Skills Opinions Aspirations Motivations Intermediate Action Behavior Practice Policies Social Action Decision-Making Long-Term Conditions Social Economic Civic Environmental ENVIRONMENT: External and contextual factors that influence the program Sources: GAO-02-923 –Strategies for Assessing How information Dissemination Contributes to Agency Goals. GAO-03-9 –Efforts to strengthen the link between resources and results at the administration of children and families. GAO/GGD-00-10 Managing for Results: Strengthening Regulatory Agencies' Performance Management. Ellen Taylor-Powell, 2000. “A logic model: A program performance framework.“ University of Wisconsin-Cooperative Extension Program Evaluation Conference.

  12. Why Develop a Program Logic Model? • Clearly identifies program elements — goals, inputs, activities, outputs, and outcomes • Shows how program elements fit together • Illustrates the logic of how your program works • Helps specify what to measure in an evaluation

  13. Why Develop a Program Logic Model? • Builds a common and shared understanding of the program’s goals, structure, need for resources, and expected results among program stakeholders; • Provides a valuable and concise tool when presenting on the program to other groups; • Provides a record of your program.

  14. How to Construct a Program Logic Model There are several ways to do so: • Convene a meeting of your program’s stakeholders • Develop the model as a group. (Building the model helps build consensus and support for your program.) • Program staff develop the logic model • Meet separately with various stakeholder groups to elicit their perceptions and expectations. Different stakeholder groups may hold different views about the program’s activities and expected results.

  15. Where to Begin in Constructing a Logic Model • A program logic model is a type of flowchart • One may begin constructing the model at either end of the chart. • Try to construct a program logic model as early as possible during program planning. • A program may be up and running before a logic model is developed. • A logic model is a living document • Revisions may be necessary as the program changes.

  16. Next Steps in Program Evaluation Once you have the logic model: • Decide on your evaluation questions (They are related to the goals of your program) • How effective is your program?

  17. Moving From Goals to Evaluation Questions Project: After-school recreation program designed to promote self-esteem and reduce delinquency Program Goals: Increase youth protective factor of self-esteem. Lower arrest rates among program youth. Evaluation Goal: Determine whether youth served increase their self-esteem and decrease delinquency Evaluation Questions: Do youth who complete the program show increased self-esteem and decreased likelihood of delinquency?

  18. Types of Program Evaluation • Process Evaluation • How does the program work? • What services are provided? • Is the program implemented as planned? • What are the outputs? What works or does not work in the program?

  19. Types of Program Evaluation • Outcome Evaluation • Assess program effectiveness • What are the results? • Short term • Long term

  20. Next Steps in Program Evaluation Choose the type of evaluation to conduct: • Decide on the data to be collected to monitor performance measures/outputs and outcomes • Create instruments needed to collect data • Decide how to analyze and use data

  21. Data Collection • Need up-front planning • Need a sense of what you are trying to accomplish • What data will you collect and why? • What data sources are available and which will you use?

  22. www.evaluationtools.org/tools_main.asp Types of Data Collection Tools

  23. Create Instruments • Construct your own instruments • Attendance sheets • Activity logs • Surveys for clients • Use existing instruments • Templates available online • Modify to suit your particular program

  24. Ways to Collect Data • Paper ̶ Internal forms/checklists • Electronic ̶ Excel, Access, SPSS • Online ̶ Web-based data collection tools ̶ OJJDP’s Data Collection & Technical Assistance Tool (DCTAT)

  25. Analyze and Use Data • Analyze Data • How will the data be analyzed? • Who will be responsible for data analysis? • Use Data • How will the data be used? • Will the data be made available in reports? • Who will receive the reports?

  26. Evidence-Based Programs • A program is evidence-based if: • evaluation research shows that it produces the expected positive results; • the results can be attributed to the program itself, not to other factors or events; • the evaluation is peer-reviewed by experts in the field; • the program is “endorsed” by a federal agency or respected research organization and included in its list of effective programs. • From: Evidence-based Programs: An Overview • http://www.uwex.edu/ces/flp/families/whatworks_06.pdf

  27. The Importance of Evidence-Based Practices • Government-wide move toward accountability • Government Performance and Results Act • Shift from accountability for process to accountability for results • Programs must show effectiveness to justify funding • President’s Management Agenda • Evidence-based practices are cost effective • It is important to provide programs that work

  28. Program Evaluation: Success and Sustainability • Use data to create interest and support for program continuation • From funders • From the community • Program evaluation helps identify the most successful components of your program • Maintain effective staff, procedures, or activities

  29. Conclusion • Program evaluation is an ongoing process • Start with a logic model to identify the main elements of the program • Choose the type of evaluation needed • Plan data collection and analysis activities in advance • Use program evaluation results to justify and market the program

  30. Resources The Program Manager’s Guide to Evaluation Office of Planning, Research and Evaluation Administration for Children and Families U.S. Department of Health and Human Services http://www.acf.hhs.gov/programs/opre/other_resrch/pm_guide_eval Planning a Program Evaluation University of Wisconsin Cooperative Extension http://learningstore.uwex.edu/assets/pdfs/G3658-1.pdf The Global Social Change Research Project http://gsociology.icaap.org/methods/evaluationbeginnersguide.pdf • Boulmetis, J., & Dutwin, P. (2002). The ABC’s of Evaluation. • San Francisco: Jossey Bass.

  31. Contact Information Monica Robbers, Ph.D. Patricia San Antonio, Ph.D. CSR, Incorporated 703.312.5220 mrobbers@csrincorporated.com psanantonio@csrincorporated.com

More Related