How Do You Know You Are Achieving Excellence and Making an Impact?

1. How Do You Know You Are Achieving Excellenceand Making an Impact? Presented by: IT Resource Center and TCC GroupMarch 25, 2005

2. Introductions

3. IT Resource Center • Nonprofit technology assistance provider • Technology consulting, training, and problem-solving for nonprofits • Objective source of information • Advocacy on the importance of technology to nonprofit effectiveness

4. Motivations for Evaluation • Interested and supportive funder in The Chicago Community Trust. • 20th Anniversary of ITRC – mature organizations should set good examples • Desire for outside confirmation of impact in the community • Need for long-term evaluation strategy • Develop a logic model appropriate to field

5. Big Question: Impact of ITRC • On staff behavior of individuals • On nonprofit organizations in making them more effective • As a management support organization assisting a community of nonprofits

6. TCC Group’s History • In operation for over 26 years • Formerly The Conservation Company • Started in Philadelphia, now with offices in New York City, Chicago and affiliates in other geographic locations • 25 professionals with diverse education and professional experiences across the nonprofit sector • Management consulting, board development, organization assessment, grants management, evaluation

7. TCC Values • Stakeholder involvement (community of learners) • Capacity building among stakeholders (evaluative learning) • Rigorous data collection • Utilization focused reporting • Recommendations for strategic direction

8. Evaluation Overview: Getting Started

9. What is Evaluation? • The key sense of the term “evaluation” refers to the process of determining the merit, worth or value of something (including the product of a process) through the identification of standards, an investigation of performance, and an integration of results. -Michael Scriven’s Evaluation Thesaurus, 4th Edition • Evaluative learning is an ongoing and a collaboratively designed process that has the primary purpose of serving organizational learning and planning. –Peter York , Learning as We Go

10. Evaluation What it is…What it is not… Analytical yet broad, general Empirical research Multidisciplinary (a science & an art) Purely hypothesis testing Focused primarily on utility Focused solely on theory Results in evaluative conclusions Results focused in proofs

11. Readiness for Collaborative Relationship:Evaluative Learning/Capacity Building • Have receptive staff • Conduct periodic organizational assessments • Conduct periodic needs assessments • Have strong organizational leadership • Have Board buy-in/familiarity • Conduct periodic environmental scans • Have some form of MIS/data management • Have strong administrative support • Engage clients in planning efforts • Build and maintain external relationships • Acquire diversified and sustainable funding • Have knowledge of how to use research • Have evaluation history and knowledge • Have staff job security and low staff turnover Extremely important Very important

12. Consulting Relationship • Collaborative in nature • Evaluation team governed • Unique roles and responsibilities • TCC Staff • ITRC Staff • ITRC Board • Significant time in planning phase to develop relationship, define roles and responsibilities, and develop the evaluation design framework

13. Advantages of Collaborative Approach Allowed for: • Incorporation of internal expertise related to the field, the organization, and the context for evaluation • External objective evaluation framework and data collection • Evaluation capacity building among IT Resource Center staff • IT Resource Center investment and ownership in the evaluation • IT Resource Center readiness for use of evaluation findings

14. Evaluation For Capacity Building Definition: an ongoing, collaborative, and stakeholder-led evaluation process that has the primary purpose of informing program and organizational planning and development. Traditional evaluation Capacity building evaluation Designed and implemented by a few Problem centered Report card External accountability Low stakeholder involvement Imposition on those being evaluated Passive learning Focused on the past Periodic Consequences Designed collaboratively Focused on strengths and/or solutions Organizing tool Internal planning and decision making High stakeholder involvement Owned by all stakeholders Active learning Future oriented Ongoing Low risk

15. Evaluation Framework, Design and Tools

16. Logic Model A systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan to do, and the changes or results you hope to achieve. -(WKKF)

17. Logic Model Terminology • Inputs: all the resources necessary for supporting a program; e.g., money, time, expertise, experience, leverage, facilities, technology, etc. • Program Strategies: the specific activities, interventions, services and/or programs that serve a particular target audience; e.g., case management services for homeless families, advocacy efforts for particular legislation and/or policy changes, etc. • Outputs: a short-term measure of program strategy implementation; e.g., number of clients served, number of services provided, actual dollar expenditures per client or unit of service, etc. • Outcomes: the short- and longer-term effects of program strategies on client behaviors, attitudes, knowledge, and/or perceptions; e.g., improved student performance in math coursework, homeless clients obtain and maintain affordable housing, etc. • Impact: the long-term and aggregate effect of a sustained program, service or intervention on the overall target population.

18. If your strategies are fully implemented and of high quality, then the amount of service/product will be produced If these benefits to clients are achieved, then certain changes in organizations, communities or systems will occur Resources needed to operate your program effectively and efficiently If you have the necessary resources, then you can use them to implement the strategies If you accomplish your planned strategies, then clients will benefit in certain ways Resources/Inputs Strategies Outcomes Impact Typical Logic Model Outputs ExternalInfluences and Related Programs

20. Evaluation Workplan

21. Data Collection • Key informant interviews with experts in the field of technical assistance • Interviews with partners, strategic relationships, funders • Focus groups with members • Survey with members and nonmembers • Site visits to member organizations

22. TCC Group Primary tool design Key informant interviews Interviews Survey administration Data analysis Reporting IT Resource Center Some tool design Volunteers conducted some key informant interviews Board member conducted focus groups University experts conducted site visits Roles and Responsibilities

23. Findings

24. Findings:Related to Services at the Individual Level ■Finding #1: Among both members and nonmembers, the IT Resource Center is best known for, and in demand for, its training services. ■Finding #2: Among members, consulting/customized services is the second most frequently needed (in demand) service. ■Finding #3: Members and nonmembers identified a need for advanced level training and services.

25. Findings (continued):Related Services at the Organizational Level ■Finding #4: The IT Resource Center is perceived as making technology readily applicable and accessible to a broad range of clients (at an organizational level). ■Finding #5: Members were more likely than nonmembers to indicate that their staff has the technology training needed to do their job. ■Finding #6: It appears that a greater proportion of members than nonmembers have functioning technology plans. ■Finding #7: Members participating in IT Resource Center Technology Planning value it. ■Finding #8: Members view the IT Resource Center as a resource for information and support related to software and other tools.

26. Findings (continued):Related Services at the Organizational Level ■Finding #9: Members are more likely to routinely collect data (than nonmembers) and to do so electronically. ■Finding #10: Members value the support provided by the IT Resource Center around improving the quality of their data collection, reducing data redundancy and increasing data accessibility. ■Finding #11: Members attribute the improvement of their internal and external communications to the assistance of the IT Resource Center in building their (the nonprofit’s) technology infrastructure. ■Finding #12: Members report that the IT Resource Center’s role in equipping them with technology and training has improved their organization’s efficiency and effectiveness. ■Finding #13: Members appear to use the IT Resource Center for specific immediate issues (e.g., crises such as a virus, troubleshooting, etc.) and in supporting these nonprofits through their crises the IT Resource Center builds capacity.

27. Findings (continued):Related Field Level Outcomes ■Finding #14: Overall, funders vary in their understanding and support of capacity building including the role of technology as a capacity building tool. ■Finding #15: Technology plans are effective tools for nonprofits approaching potential funders. ■Finding #18: Members and nonmembers recognize technology as significant and/or critical to fulfilling their mission. ■Finding #19: Members’ and nonmembers’ responses revealed a growing trend toward using permanent staff to handle organizational technology needs. ■Finding #20: Among both members and nonmembers, organizational leaders are regarded as supportive of the appropriate use of technology, and it is integrated into requests for funding. ■Finding #21: A majority of members and nonmembers identified improved data collection, data entry strategies and database analysis as high priority needs.

28. Contact Information TCC Group • www.tccgrp.com • 888-222-0474 Peter York Director of Evaluation One Penn Center Suite 1500 Philadelphia, PA 19103 215.568.0399 pyork@tccgrp.com Chantell Johnson Senior Consultant 875 N. Michigan Avenue Suite 3930 Chicago, IL 60611 312.642.2249 cjohnson@tccgrp.com Jen Avers Consultant 875 N. Michigan Avenue Suite 3930 Chicago, IL 60611 312.642.2249 javers@tccgrp.com

29. Contact Information IT Resource Center • www.itresourcecenter.org • 312-372-4872 Tim Mills-Groninger Associate Executive Director IT Resource Center 29 E. Madison Street Suite 1005 Chicago, Il 60602 312-372-4872 timmg@itresourcecenter.org Betsy Murphy Program Associate IT Resource Center 29 E. Madison Street Suite 1005 Chicago, Il 60602 312-372-4872 bmurphy@itresourcecenter.org