Communities Connect Network Study of Community Technology in Washington State

1. Measuring and reporting outcomes for BTOP grants:the UW iSchool approachSamantha BeckerResearch Project ManagerU.S. IMPACT Study UW iSchool evaluation framework

2. Communities Connect NetworkStudy of Community Technology in Washington State • In 2007, UW conducted a phone survey across 211 agencies in the state identified as community technology providers. 47of these agencies completed the survey, and 7 sites were visited for more in-depth study. • This was combined with earlier data to provide a snapshot of CT in Washington State UW iSchool evaluation framework

3. Findings from the CCN survey indicated that CT was having an impact on communities. • Three levels of benefits were identified • Individual • Family • Community • Six domains were found to be important • Employment/economic • Academic skills and literacy • Social inclusion and personal growth • Independence • Access to information and resources • Communication From 2007 CCN study UW iSchool evaluation framework

4. This lead to the development of the community technology impact analysis framework • Used to frame the Washington Community Technology Opportunity Program goals • Six domains were found to be important • Employment/economic • Academic skills and literacy • Social inclusion and personal growth • Independence • Access to information and resources • Communication • A similar process was used to frame the current WA BTOP evaluation framework UW iSchool evaluation framework

5. Context Analysis Step 1: Policy issue mapping Identify major policy issues confronting the PCC’s community, such as: • Workforce development • Education • Poverty • Civic engagement And link them to activities used to address these issues, such as: • Access to technology and information • Skills building • Development of local content UW iSchool evaluation framework

6. Context Analysis Step 2: Stakeholder analysis Identify other stakeholders concerned with the policy goals • individuals • groups • organizations • institutions Collect data about what they do and how they experience PCC services Include stakeholders who are working to achieve similar policy goals or who are affected by the PCC UW iSchool evaluation framework

7. Situated Logic Model Step 3: Develop a policy or stakeholder logic model Policy Issue: Workforce Development UW iSchool evaluation framework

8. Situated Logic Model Step 4: Develop a PCC Logic Model to link goals with measurable indicators UW iSchool evaluation framework

9. Sidebar: what’s a measurable indicator? Indicators need to meet certain utilitarian standards. Beyond the actual content of the indicator, they should also be: • specific, unique and unambiguous; • observable, practical, cost effective to collect, and measurable; • understandable and comprehensible; • relevant (measures important dimensions, appropriate and related to the program, that are of significance, predictive and timely); • time bound; and • valid, providing reliable, accurate, unbiased, consistent, and verifiable data (Hatry, 2006) UW iSchool evaluation framework

10. Situated Logic Model Step 4: Bridge the logic models to show how the CTC supports larger policy goals Workforce Development • Example: Workforce Development  Community technology • Workforce development clients use community technology to look for and apply for jobs • Clients get jobs • Workforce is improved 2 3 1 Community Technology UW iSchool evaluation framework

11. Outcome Measurement Step 6: Measure outcomes Identify the most important outcomes to measure: • Tie activities to immediate policy goals in the PCC community • Link to the larger community policy context to evaluate overall impact on stakeholders and the community Step 7: Report outcomes Report outcome measures in the context of the situated logic model to show contribution to community policy goals. UW iSchool evaluation framework

12. Validation and Reflective Practice Step 8: Validate Outcome Measures Re-examine relationship between outcomes and policy issues: • Validate measures • Challenge assumptions • Interview stakeholders Step 9: Reflect on performance Use outcome/impact measures to inform your work: • Establish the value of your work • Improve effectiveness • Understand your organization’s contribution to the community you service UW iSchool evaluation framework

13. Measuring and reporting outcomes for WA BTOP UW iSchool evaluation framework

14. WA BTOP’s reporting system relates to a community technology logic model • SRs are asked to keep track of and report inputs, activities, outputs, and impacts on a quarterly basis for the duration of the grant in order to: • Show the value of the services offered • Show the changes in use as a result of the grant • Evaluate the impact of the grant on the outcomes of clients Measuring and Reporting Outcomes UW iSchool evaluation framework

15. Impact typesfor WA BTOP were informed by the CCN study, BTOP policy goals, and PCC self-identified anticipated outcomes UW iSchool evaluation framework

16. Tracking outcomes • SRs are not expected to have outcomes to report for every category or indicator. • SRs were asked to review the list of possible outcomes and choose those they wish to track. • No single client survey can accommodate all PCCs, but samples and advice are provided for SRs for designing instruments and tracking sheets. UW iSchool evaluation framework