TELECENTRE-EUROPE Summit:

1. TELECENTRE-EUROPE Summit: Third Sector contribution to e-Inclusion in Europe: Impact Measurement and Analysis framework Budapest, 14th October 2010 Gabriel Rissola, JRC-IPTS Maria Garrido, Technology & Social Change Group | University of Washington Joint Research Centre (JRC) Institute for Prospective Technological Studies The European Commission’s Research-Based Policy Support Organisation

2. Institute for Prospective Technological Studies Part of Joint Research Centre of the EC IPTS: Research Institute supporting EU policy-making on socio-economic, scientific and/or technological issues

3. MSP for eInclusion

4. Last TE Summit in Istanbul: IA policy relevance and constraints Demonstrate Impact is useful for: Get a recognition of telecentres role in EU policy making: - Evidence is essential to get policies adopted (including funding programs) - Evidence helps to set priorities, make the right choices Smarter action by combining social innovation and economic innovation, stimulated by technological innovation …but few IA good practices were identified in the field In Vienna Study on Inclusive Innovation for Growth and Cohesion, 1000 cases were analysed but only 52 had measured outputs and outcomes (mostly from UK and IRL), for example: UK Onlines Centres trained 76,000 people; active job seekers increased from 66% to 100% EOL UK digital literacy training helped 76 people to get a job, saving £670,000 in state subsidies FIT Ireland trained 6500 young unemployed people in ICT; 3500 found a job, saving $13 million. Also examples of improved welfare benefits distribution, savings through ICT-enabled home care.

5. Last TE Summit in Istanbul:Measuring Impact (outputs, outcomes) Cristiano Codagnone et al, EC study March 2009 (Vienna Study on Inclusive Innovation for Growth and Cohesion)

6. Last TE Summit in Istanbul:Telecentre eInclusion critical role and challenge • Increasing political awareness on the critical role of Telecentres and NGOs in promoting e-Inclusion goals through different social and digital programs • Wide range of target groups addressed -from immigrant communities, women, impoverished youth to prisoners and isolated people in rural areas to people with physical disabilities and the elderly – which goes even beyond Riga’s targets • However, 3rd Sector organisations working in the eInclusion field don’t balance their efforts in critical areas (like IA)

7. Last TE Summit in Istanbul:Telecentre eInclusion critical role and challenge NGOs often struggle to systematically: collect information from their beneficiaries without overburdening staff capacity articulate how and the extent to which their programs impact the lives of the people they serve use this evidence as a tool to gain more visibility and recognition to their work especially among policy makers and donors Measuring Outputs (quantity of trainees, quantity of Internet users) and End user satisfaction surveys are important, but insufficient; need to go a step forward and find a way to systematically measure Outcomes

8. Practitioner perspectiveTelecottage impact measurement system • Adapted UK’s IMProving Accountability, Clarity and Transparency (IMPACT) to Hungary • Prepare impact report of the real social effect of their work. • Primary data (outputs), with Secondary (outcomes). • Created intended effects and possible indicators, on whether life is better or not. • Most important effects of telecottages: • strengthened community spirit, • improved human conditions, • increase public trust • Challenge: • to adapt intended effects and possible indicators to other local contexts • to measure them in an aggregated way at regional/national/EU level

9. Funder perspectiveMS UP impact measurement system • Driven by need for systematic and credible evidence of how UP helps unemployed & others excluded to get essential skills to find a job. • Most important thing is that evaluation scales globally. Is it improving lives? Are they moving along a trajectory? • UP goes beyond self-reported quarterly reports, doing evaluations and case studies.

10. Funder perspectiveMS UP impact measurement system • Lessons learnt: • track results on ongoing basis and make adjustments as needed • metric that decided to avoid is how many people they put into jobs; • commissioned research can be expensive; • strongest grantees & partners are data driven and report back consistently, which builds a solid picture on impact over time • Need for common indicators, plural of anecdote is not evidence. • Opportunity: telecentres are multipliers

11. Building an IA framework

12. Building an IA frameworkPossible steps • Consensus building on the objectives of measuring impact • to provide funders or investors with data on impact; • to provide a tool for organisations to manage their own choices internally • to better understand long-term processes of social change and impact. • Targeted methodological framework for IA based on the state-of-the-art(there is no “perfect” social metric framework) • Targeted set of indicators • Relevant enough as to measure different e-Inclusion dimensions • Simple and easy to implement by organizations of different sizes • Appropriate enough as to provide consolidated results related to the e-Inclusion goals of the DAE • Compatible with existing indicators but going deeper into the specificity of the measured field.

13. Need to facilitate social organizations self-analysis and self-interpretation of their own local impact. Indicators should be: Available online (save costs and facilitates its distribution) Translated and contextualized in each country (localisation) Complemented by adequate guidelines for a proper data collection, self-analysis of data and production of reports for different organisational scopes Guidelines for implementation and interpretation would surely be necessary Need a software for the implementation of an IA system based on those indicators To facilitates the process of user data collection and impact analysis for NGOs To create with these data a repository that can be aggregated and analyzed globally To protect anonymity of NGO users and of the organizations themselves Easily implementable by the interested organizations in each local context Centrally maintained for the provision of constantly-updated consolidated information Building an IA frameworkPossible steps

14. Measuring indicators is a complex endeavor due to the complexity of isolating the effects of one single variable (e.g. effect of e-Skills training in people’s lives). This process is further complicated by the diversity of NGO programs, target groups, local factors, individual factors, etc. Different types of impactcan be measured (i.e. Access,frequency and purpose of ICT use, skills development, size and diversity of social networks, user perception of usefulness of ICT for improving their lives) However, how to connect them with desired Outcomesremains an issue (e.g. “Qty of DL trainees that become regular Internet users”, “DL trainees that improve their employment status”) Building an IA frameworkIndicators to be measured

15. FEEDBACK FROM THE AUDIENCE • Constraints • Voluntary organisations • Data protection laws • Local advocacy skills • Org needs/Selling points • Carefully select and agree the objectives of IA • Start simple, small • Highlight useful bits of data • Simple and quick implementation • Pay attention to incentives

16. Thank you Gabriel.Rissola@ec.europa.eu http://is.jrc.ec.europa.eu/pages/EAP/eInclusion.html http://is.jrc.ec.europa.eu/pages/EAP/eLearning.html migarrid@uw.edu http://tascha.washington.edu/

17. Last TE Summit in Istanbul One example (female migrant users) Appreciated features of TCs Perceived usefulness of ICT • END USER SATISFACTION IS IMPORTANT, • BUT INSUFFICIENT Satisfaction with Social services + Education and Employment services

18. Building an IA frameworkObjectives of the Metrics Metrics are useful to show what works and what deserves to be grown (from The Open Book of Social Innovation): There are many metrics for judging whether innovations are working – at various stages of development. Metrics can play a decisive role in determining whether innovations are scaled up, or deserve to be. A recent survey found 150 different metrics in use in the non-profit sector. However, relatively few of these are actually used to make decisions. This field has failed to make progress due to the confusion and conflicts of interest between three different tasks performed by metrics: to provide funders or investors with data on impact; to provide a tool for organisations to manage their own choices internally to better understand long-term processes of social change and impact. • 1. NEED TO BUILD A CONSENSUS ON THE • OBJECTIVES OF MEASURING IMPACT

19. There is no “perfect” social metric framework | Pros and Cons Example 1: Sustainable Livelihoods Framework Increasingly popular among development agencies It recognizes that people have a wide range of resources at their disposal and their choice for deploying those resources are critical to assess impact Very closely align to Amartya Sen’s capabilities approach LIMITATION: DIFFICULT TO IMPLEMENT AND MEASURE HOW PEOPLE’S CHOICES AND MOTIVATION AFFECTS THE OUTCOME OF A PROJECT Example 2: Cost-benefit analysis Used primarily by public authorities and government agencies Its goal is to assess a particular project taking into account costs and benefits Benefits and costs are quantified in money terms LIMITATION: ASCRIBING MONETARY VALUES TO CERTAIN SOCIAL IMPACTS THAT ARE VERY DIFFICULT TO MONETIZE Building an IA frameworkSocial metrics system • 2. NEED TO BUILD A TARGETED METHODOLOGICAL FRAMEWORK FOR I.A. BASED ON THE STATE-OF-THE-ART

20. Relevant enough as to measure some dimensions of the impact of e-Inclusion initiatives driven by NGOs in the communities they serve; Simple and easy to implement by organizations of different sizes which are active in the field of e-Inclusion at regional/national level in Europe (extendible to smaller NGOs and other actors in a second stage) Appropriate enough as to provide consolidated results related to the e-Inclusion goals of the Digital Agenda of Europe. Compatible with existing indicators but going deeper into the specificity of the measured field. Building an IA frameworkSelection of indicators 3. NEED A TARGETED SET OF INDICATORS

21. Aggregated level E-Inclusion activities Other social services NGO mission Reach of the work (local, regional, national) Target groups Partnerships Building an IA frameworkSome examples of indicators Organisational level • User demographics • NGO services and activities where users participate • Digital skills level • Frequency of ICT use • Types of ICT use (email, chat, job searching, homework, etc) • Employability situation • Social inclusion situation • Users perception of value of ICT to improve their lives

22. 5. NEED A SOFTWARE FOR THE IMPLEMENTATION OF AN I.A. SYSTEM BASED ON THOSE INDICATORS Building an IA frameworkSoftware for processing data • To facilitates the process of user data collection and program impact analysis for NGOs • To create with these data a repository that can be aggregated and analyzed by country and at European level • In doing so, to protect anonymity of NGO users and of the organizations themselves • Easily implementable by the interested organizations in each local context • Centrally maintained for the provision of constantly-updated consolidated information at European level

23. Stakeholders interested in the metrics to be produced Organisation’s staff (managers, fundraisers, etc) Local policy officers in charge of social services and community development Donors/funders Policy makers operating at different levels, including the EC Users who will collect the data eFacilitators in contact with the end users Volunteers Staff managing the telecentres Telecentre network coordinators Building an IA frameworkStakeholders and Users

24. Review of indicators currently used by EUROSTATS, OECD, ITU, and academic articles Needs assessment to determine the kind of information that social organizations are currently gathering about the beneficiaries of their programs and additional data they are interested in collecting. Piloting the Impact Measurement System in different contexts Wide implementation of the system across Europe promoted by a multistakeholder partnership (MSP) or an umbrella organization who represents several stakeholders (like TE) Building an IA frameworkOther needs