Beginning the Evaluation Journey with FSG

1. Beginning the Evaluation Journey with FSG Prepared for: KCIC Boot Camp March 24, 2010

2. Today’s Agenda Today’s Discussion Things You Need to Do to Evaluate Your Project’s Progress and Outcomes • Articulate your Theory of Change and develop a project Logic Model • Develop an evaluation plan • Execute your evaluation plan • Participate in Knight’s evaluation process

3. Evaluation Overview The KCIC Evaluation Has Three Overarching Objectives Evaluate Progress • Knight’s 2 Goals: • Foundations are engaged in helping meet community information needs • Communities are more informed and engaged Inform and Improve Program Strategy • Inform the Knight Foundation’s program strategy for the Community Information Challenge and identifying opportunities for refinements over time • Createdata that promotes shared learning within the KCIC grantee communityand the field of place-based foundations to increase understanding of how community information needs can be addressed effectively Promote Learning

4. The systematic collection of information based on questions of critical importance to the organization or community: What do you want to know? Why do you want to know it? Where will you find what you need to know? What will it look like? How will you use what you learn? Who are the intended users of the information? To make judgments, inform decisions, and take action Is an ongoing process; should not be an “event” Is best when implemented as a collaborative learning activity that builds an organization’s capacity What is Evaluation? Definitions of Evaluation

5. Process The Evaluation Process: Planned, Purposeful, Systematic 1 2 3 Articulate Theory of Change & Develop Program Logic Model Focus the Evaluation Determine Design & Data Collection Methods (numbers, words, pictures) Design Refine Strategy Based on Evaluation Findings Communicating and Reporting 6 5 4 Develop Recommenda-tions and Action Plans Analyze Data and Interpret Findings Collect Data Implementation

6. Uses of Evaluation How Can We Use Evaluation? • Instrumental use - findings reflects changes that can be observed. • Conceptual use - findings may lead to stakeholders having different kinds of conversations, greater insights into future decisions, and/or greater commitment to the program or initiative. • Political or Symbolic use - legitimate uses of the evaluation findings may be applied to lobbying for the program or initiative, securing new or additional funding, or communicating that the evaluation has taken place. Well designed evaluations will serve multiple purposes

7. Stakeholders Different Stakeholders Will Have Different Uses for Evaluation Evaluation Stakeholders • Internal Stakeholders • Board members/Trustees • Senior Leaders/Executives • Program staff • External Stakeholders • Legislators/Policymakers • Project partners • The general public • The media • Researchers • Other funders • Journalism programs • Identifying stakeholders has implications for: • which and how many questions are asked • what indicators will be of interest • how findings will be communicated and reported • how results will be used

8. Example Envision Bay Area– Theory of Change What we are doing to address the issue – how we plan to solve the problem or create the change (Activities) What the issue is and why we are taking action (Assumptions) The change we hope to see if we are successful (Outcomes) • The community needs a regional plan to address climate change and sustainability • There is a low awareness and understanding among residents of the intricacies and implications of regional development decisions • Residents are not engaged in the planning process today • Creation of SVCF’s Venture Fund (with investors) • Development of a web-based information platform for citizens to learn about potential scenarios • Planned convenings on land use and sustainable cities • Awareness campaign (KQED) • Interactive kiosks • Citizens are more informed and aware of key issues • Citizens are more engaged in planning processes • Regional plan is developed with community input This Theory of Change illustrates at a basic level why SVCF is doing what it’s doing Source: FSG interview; used with grantee’s permission

9. As You Think About Your Own KCIC Project Outcomes, Consider the Possibilities Outcomes Outcomes are…the SO WHAT? • The short and/or long term changes, results, and impacts from implementing a project, program, or initiative. Changes may be: • positive or negative, singular or multiple • in knowledge, skills, and/or attitudes • in organizational policies, practices, capacity • at community level in terms of behavior, values, and attitudes What will be different in your community if your project is successful?

10. Types of Outcomes Short term Outcomes • Short term Outcomes • Achieved during program timeframe • Within program control • “Expect to see” • For example: • New knowledge • Changed opinion/values • Increased skills • Changed motivation/intent • Changed attitudes • Changed aspirations • New relationships/networks Adapted from: Innovation Network, Inc. (2005)

11. Types of Outcomes Intermediate Outcomes • Intermediate Outcomes • Achieved at the end/beyond program timeframe • Follow shorter-term outcomes • “Want to see” • For example: • Modified behavior • Changed policies • Changed practices • Changed social action • Changed decisions • Modified structures Adapted from: Innovation Network, Inc. (2005)

12. Longer-term Outcomes Types of Outcomes • Longer-term Outcomes • Achieved after program timeframe • Outside direct program control • “Hope to see” • For example: • Changed human condition • Changed civic condition • Changed economic condition • Changed environmental condition Source: Innovation Network, Inc. (2005)

13. Start Building Your Evaluation Plan by Developing Your Own Theory of Change and Identifying Your Project Outcomes In Practice • From our interviews with each of you, we have attempted to visually depict your project’s theory of change. Review our version of your ToC and change, add/or revise as needed. • If your project were hugely successful in three years, what would your target audience be doing, thinking? (long term outcomes) • After one year, what would your target audience be doing, thinking? (short term outcomes) • What are some indicators that your project is achieving what you set out to do? (outputs) See online assignment form:http://bit.ly/9nREQl

14. Next Steps Next Steps in Designing Your Project’s Evaluation Plan • Developing a Project Logic Model and Key Evaluation Questions (Webinar #1) • Determining Indicators of Online and Offline Behavior (Webinar #1) • Choosing an Evaluation Design and Data Collection Methods (Webinar #2) • Analyzing Quantitative and Qualitative Data (Webinar #2) • Strategies for Communicating and Reporting Evaluation Findings to Multiple Audiences (Webinar #3) • Developing an Evaluation Budget (Webinar #3) • Tips for Hiring an Evaluator (Webinar #3) Throughout 2010 – FSG will provide up to 2 hours of additional evaluation-related technical assistance to each grantee