European eGovernment measurement -- eGovMoNet final conference Brussels, 8 April 2010

1. European eGovernment measurement • -- eGovMoNet final conference • Brussels, 8 April 2010 • From measuring • user satisfaction • to measuring • user engagement • Jeremy Millard • Danish Technological Institute

2. Agenda Status of eGovernment satisfactionmeasurement State of art of eGovernment satisfactionmeasurement Step change in eGovernment services and howwemeasurethem Four long term development and measurement trends Implicattions… Usersatisfaction is really a surrogate for the effectiveness of eGovernment services

3. Status of eGovernment measurement – 1 • Progress on policies and measurement 2007-2009 (Millard et al 2009 “i2010 eGovernment Action Plan Progress study”, European Commission) • Usersatisfactionprogress • 2008: c. 33% of citizensratedeGovexperience as ”not satisfactory” (Deloitte & Indigov 2009 ” Study on user satisfaction and impact in EU27”, European Commission) • 2009: usersareonaverageonly ”moderatelysatisfied” (Capgemini 2009 “The new deal: smarter, faster and better eGovernment”, 8th Measurement for the European Commission) • 2009: only 35% of screened web-sites canberated by usersdirecton line (Capgemini 2009)

4. Status of eGovernment measurement – 2 • Progress on user satisfaction measurement 2007-2009 0 = no progress, 0.33 = some progress, 0.66 = good progress, 1 = very significant progress (Millard et al 2009 “i2010 eGovernment Action Plan Progress study”, European Commission)

5. State of art of eGovernment measurement – 1 • Range of usersatisfaction factors & indicators(24 countries, Millard 2010)

6. State of art of eGovernment measurement – 2 • Usersatisfactionmeasurementmethods in use 2009 (24 countries, Millard 2010) • Most widespread: indirect by provider • Basic difference ‘online for specific service’ and other surveys which often survey whole population incl. non-users • Observation & ethnographic: basically observing / monitoring user behaviour • During service use: e..g. mystery user approaches (i.e. surrogates or planted users), following ‘lead’ users, living labs, etc. • User co-design • User charter

7. Summary • Significant progress in satisfaction measurement • A lot of interesting experimentation now starting, although in minority of countries • But huge variations between countries, lack of comparability, lack of real impact on eGovernment performance and policy • Raises question of what is (satisfaction) measurement for? • AND, things are in future likely to become much more diverse posing a whole new set of measurement challenges as the nature of eGovernment services transforms…..(probably!)

8. A step change in eGovernment services and thus how we measure them is on the way these are very strange times….. • The UK is (probably) leading Europe in eGovernment: • Number 4 in world, highest in Europe (UN 2010) • Number 4 in world, 2nd highest in Europe (Waseda 2009) • Joint number 1 in online availability in Europe (EU 2009) • Over last 4 years, UK spent $60 million on main government portal: direct.gov.uk – acclaimed as world-class • BUT this is used LESS than an unofficial site which cost nothing apart from a day’s work by concerned volunteers • WHAT ON EARTH IS GOING ON ??!!

9. Citizens usurp government (1)

10. European eGovernment service citizen use (Source: Eurostat, 2009)

11. European eGovernment service citizen use (Source: Eurostat, 2009)

12. Citizens usurp government (2)

13. Citizens usurp government (3) Ilkeston, Derbys

14. The dominance of portal eGovernment is over…? • Portals are important, especially for large scale, top down ‘administrative’ services (one stop shop concept) • BUT citizen usage is low: • “Why go to a portal first when I am already somewhere else on the web? I want to go direct to the service I need.” • “Everything (services, applications, platforms, infrastructure) is – or will be – in the cloud anyway, so just use Google or other search engines to find what you need.” • “Do we hang on to grandiose portals because they are a showcase – just like an imposing town hall – but what do they really do for all that money?“

15. Next step change: everyday eGovernment • Everyday, not just 2-3 times a year • ‘Public’ services are potentially all around us and could be used constantly – services which are really valuable in people’s everyday lives: health, education, care, transport, infrastructures, utilities, clean and safe environments, congestion & pollution watch, culture, amenities, leisure, sports, security, crime watch, weather, participation, engagement, etc. • Often very location specific – depends where you are and what you are doing • Also very personal – move from one-size-fits-all to precisely-my-size

16. Everyday technologies can deliver this • Smart mobile phones + GPS (digital TV) • Location-based / place-related • Real time, augmented reality • Offered appropriate services as walk down the street • Location or event creates real time opportunities for content, engagement, participation • BUT only very small % of 100,000 iPhone apps are for public services

17. The USA is leading on the ground • Over last 1-2 years, big cities (e.g. New York, San Francisco, Washington) have made public data available for anyone to use • The ‘geeks’ got to work on spreadsheets on train schedules, complaints, potholes, street lamp repairs, city garbage, etc., etc., and created apps for local people to use • Local people got involved and started to realise “government is us” and “we have a responsibility too” • BUTonly where local government releases the data – small cities have no time or expertise (but now San Francisco is creating a standard for municipal data sets for others to use) • “For some cities and for some governments I could understand that transition can be a scary thing….but we feel like it makes governments more accountable, and it makes them function better…..what I really see is a monumental change for how government works. This is just really the starting point."

18. But the UK is catching up, maybe overtaking… the ‘gold standard’ – 2 months ago

19. Four long term development and measurement trends Up the policy value chain Down the government hierarchy Out of the government institution Start from social and user priorities

20. 1) Up the policy value chain • Moving from a focus ONLY on demand side, inputs and outputs, e.g.: • back-office process re-engineering • eService availability and sophistication • access • money spent • number of cases dealt with • Moving to a focus which also INCLUDES outcomes and impacts, e.g.: • front-office successful use • satisfaction and fulfilment • competitiveness, growth, jobs • social cohesion, inclusion, democracy

21. 2) Down the government hierarchy • From ONLY central government macro targets and measures (of mainly large scale administrative services infrequently used) • To INCLUDE local government AND front line staff AND user targets and measures • designing and setting own standards and outcomes – often quite local and quite specific targets and measures • staff and users (family, community, civil organisation) panels • user-centric measures in care, housing, education, crime, transport… • Measurement is more immediate and real time, i.e. don’t ‘wait forever for the decisive evidence’, but feedback is continuous so can measure as you go along, and respond more quickly and precisely – adjusting, learning and exchanging experiences in real time

22. 3) Out of the government institution • Host of otheractors at differentlevels (and thus new business models, especiallybottom-up) • Using new diverse sources of data and otherknowledge and ressource inputs • ’Everyday’ and local services (not just 2-3 times per year large scalecentralisedadmin services) • All this implies much greater risk, as it decreases (central) government control – but is this right? – it spreads risk as it spreads accountability • But does require new forms of coordination and measurement as part of that

23. 4) Start from social and user priorities • Government-determined: howwemeasuretoday Identification of eServices delivered by Gov priorities & structures eGov value contribution to government’s priorities & structures Measuring these gov-determined eServices Gov priorities & structures, e.g. ministries • Social/userdetermined: howwe (should) measure in future Social & users priorities & structures, e.g. what is most used by whom Identification of eServices meeting these social & user priorities eGov value contribution to social/user priorities & structures Measuring these social /user determined eServices

24. Some implications of these long term trends – 1 • Much more difficult to compare and apply across different areas, sectors and countries: • so will need more learning practice, benchlearning, etc. • AND CRITICALLY, the direct use of measurement to improve performance in real time as well as longer term policy • Need to develop measurement tools which can be used by many actors in different contexts and at different scales • But, still need national and international frameworks for coordination, learning, and perhaps some comparison: • and this must be done globally, i.e. in cooperation with e.g. OECD, UN, World Bank • BUT without compromising the local, the small scale and the bottom-up – the two must be complementary • More comparison of like-with-like, e.g. clusters of similar PAs as in Belgium’s Fed-eView

25. Some implications of these long term trends – 2 • Much measurement has been used badly, focusing on factors to move up the rank order rather than what may actually be needed • But measurement can be extremely constructive if used to help countries do better by focusing on factors important for them (e.g. alert countries to factors they can improve and avoid mistakes made by others through good practice, learning, etc.) • There is also a need to link eGovernment much more firmly into a whole-of-government view: • strong trend towards multi-channel integration, management, etc. • ICT is becoming ubiquitous in the public sector, so will increasingly become problematic to talk about a separate something labelled “e”