Using the Evaluation System to Answer Key Questions About Your Initiative

1. Using the Evaluation System to Answer Key Questions About Your Initiative

2. What do we mean by answeringevaluation questions? Answering five key questions can help you learn whether any changes in the community or system you’re concerned with are actually taking place, what they are, whether and how your work has influenced them (and vice versa), and whether and how they interact.

3. Is your effort serving as a catalyst for community change related to its mission? What factors or processes are associated with the rate of community or system change? How are community or system changes contributing to efforts to promote community health and development or other mission-related goals? Are community or system changes related to improvements in population-level outcomes that reflect the objectives of your effort? Did the effort lead to improvements in population-level outcomes?

4. Why answer the key questions? • To improve your work. • To understand what affects the work in what ways. • To see how to accelerate results. • To understand specifically how “peripheral” actions, events, or conditions – the attitudes of line staff, the atmosphere of the space, the location, the way people feel they’re treated, etc. – affect the work. • To understand what impact your actions have in the community, and adjust accordingly. • To become a better catalyst. • To learn what kind of leadership works best.

5. To understand what works to bring about community change, and adjust accordingly. • To understand how to address specific events and changes within your organization or effort so they will have the most positive or least negative effect on your work and on the process of community change. • To understand the relationship between your specific work and wider community change that may result from it. • To understand how the timing of events and actions affects your effort. • To see the context whole. • To show the community the value of your work. • To use as an argument for funding. • To pass on the knowledge to other organizationsand communities.

6. Who should be involved in answeringthese questions? • This type of evaluation works best as a participatory effort. Those who might be involved include: • Participants in or beneficiaries of the effort. • Residents of a geographic area you’re focused on. • Professionals and volunteers carrying out the work. • Those whose jobs or relationships bring them into contact and involvement with the population and/or issue you’re concerned with.

7. Who should be involved in answeringthese questions? • Other community organizations and appropriate branches of local or state government agencies involved in community health and development. • Local officials. • Funders. • Community organizers and other activists involved in the issue you’re targeting or related issues. • The business community.

8. When should you set up and use an evaluation system to answer key questions about the effort? Start at the very beginning of the effort, so you can record the whole of your process – outreach, planning, implementation, and evaluation. Continue gathering, recording, analyzing, and using data throughout the course of the effort.

9. How do you use an evaluation system to answer key questions about the effort? • Decide what information you need to answer each question. • Decide how to gather that information. • Devise a method for recording and setting up your data that makes it easy to analyze. • Graphing is particularly good because it allows you to easily compare different sets of data. • Consider each question separately:

10. Is your effort serving as a catalyst for community change related to its mission? • Changes to look for include: • Programs or events initiated. • Shifts in formal policy. • Shifts in informal policy. • Media reports or requests for information. • Recognition of the issue by officials. • Community reports of or events that indicateinformal changes.

11. Is your effort serving as a catalyst for community change related to its mission? • Changes to look for include: • Invitations for your organization or its staff members to join boards, committees, coalitions, planning bodies, etc. • Keep track of the timing of changes. • Look for patterns in community or system changes. • Consider whether community or system changes are great enough to have an effect. • Consider how you can obtain more accurate or more nearly complete data.

12. What factors or processes are associated with the existence and/or the rate of community or system change (positive or negative)? • The process by which you conduct the outreach, assessment, planning, implementation, and evaluation aspects of the effort. • People served or benefited. • Significant program events. These might include: • Changes in leadership. • Staffing changes • Changes in method or direction, • Milestones of the effort. • Increases or cuts in funding. • Unforeseen circumstances. • Acts of God.

13. What factors or processes are associated with the existence and/or the rate of community or system change (positive or negative)? • Community events, such as: • A change of local political administration • Racial or ethnic conflict • An economic windfall • An economic disaster • A community celebration • Community and system processes. • Combine graphs and compare the timelines of organizational and community and system events and processes with those of community.

14. How are community and/or system changes contributing to efforts to promote community health and development? • Examine changes in relation to various aspects of your work: • Your goals or aims. • The strategies of intervention you’re using. • Risk and protective factors. • The expected duration of change. • The populations benefited. • The sectors addressed. • The ecological level addressed. • Place.

15. Are community or system changes related to improvements in population-level outcomes that reflect the objectives of your effort? • Community-level indicators of an issue are the statistical indications of trends, events, or conditions in the community as a whole, rather than indicators of individual behavior. • Population-level outcomes can be found or inferred by consulting publicly available statistics and records, including: • Census data • Public health statistics • Records of health and human service organizations. Although most of these organizations shield individual participants with confidentiality, their general records – number of people served, units of service, general outcomes, etc. – are often open to scrutiny.

16. Statewide data and data from other communities (for comparison purposes) • Police and court files • Bank records • Corporate filings • Chamber of Commerce information • Educational data – standardized test averages, truancy, dropouts, high school completion rates, incidence of school violence • Environmental statistics – pollution rates, bad air days, amount of open space, water quality, etc. • You can also collect qualitative data, using interviews, observation, and similar methods. • Graphing and comparing sets of data on community-level indicators can show connections among them.

17. Did the effort lead to improvements in population-level outcomes? • The effort may have led to improvements by influencing changes in the community that brought them about. • The effort may have led directly to improvements in population-level outcomes. • Comparing data on population-level outcomes, community changes, and the timing of various phases and events of your effort can help you understand the connections among them. • Continue to gather, record, analyze, and use data indefinitely to understand how to adjust your effort for greatest effectiveness.