Impact Assessment Study of eGovernment Projects: Methodology and Some Preliminary Results

1. Impact Assessment Study of eGovernment Projects:Methodology and Some Preliminary Results Subhash Bhatnagar Indian Institute of Management, Ahmedabad (subhash@iimahd.ernet.in) and Advisor e-government, Information Solutions Group (Informatics Program) World Bank, Washinton DC sbhatnagar@worldbank.org

2. Study Objectives • Review the status of impact assessment of eGovernment projects in developing countries • To develop and present an analytical framework to conduct analysis on the impact of e-delivery of services. • To study selected e-delivery projects, assess their impact, and identify key determinants of economic, organizational, and social impact.

3. Study Team • Indian Institute of Management, Ahmedabad (IIMA) • Subhash Bhatnagar, Rama Rao, Nupur Singh, Ranjan Vaidya, Moushami Mandal • London School of Economics • Shirin Madon, Matthew Smith • eGov Practice Group • Deepak Bhatia, Jiro Tominaga, Soren Giggler • Reviewers • WBI, EAP, SAR, OECD, GICT,PREM, DEC • Sponsors • ISG,EAP,IIMA

4. Learning from Past Assessments • Variety of approaches have been used-client satisfaction surveys, expert opinion, ethnographic studies • Client satisfaction survey results can vary over time as bench mark changes—need for counterfactuals • Often studies have been done by agencies that may be seen as being interested in showing positive outcome • Lack of credibility of results-different studies of the same project show very different outcomes • Lack of rigor in sampling-results can not be easily generalized • Lack of rigor in controlling for external influence-need for counterfactuals ignored. • Lack of a standard methodology-making it difficult to compare projects • Hardly any projects do a benchmark survey

5. Critique of Existing Frameworks • Biased towards quantification of short term direct cost savings- quality of service, governance and wider impacts on society not studied. • Conceptual in nature-hardly any frameworks have been applied to assess impact of real projects • Variety in delivery models has not been recognized. Impact is a function of the delivery model and the nature of clients being served • Practical issues of paucity of data have not been taken into account-particularly in a developing country context where baseline surveys are not done and M&E systems are weak

6. Proposed Framework • Small budget exploratory study- focus on retrospective assessment of e-delivery systems(B2C and B2B) • Balanced approach between case study and quantitative analysis • Recognizes that some part of the value to different stakeholders can not be monetized • Understand how inputs lead to outputs and outcomes in different project contexts • A practical methodology that can be used for designing bench mark surveys, M&E systems and prospective evaluation of Bank projects in countries with various delivery models and paucity of data

7. Measurement Framework

8. Projects of e-delivery of Services • Issue of land titles in Karnataka (Bhoomi): 180 Kiosks, 18 million titles issued • Property registration in Karnataka (Kaveri): 230 offices • Computerized Treasury (Khajane): 240 locations • Property Registration in Andhra Pradesh: AP 400 offices. 5.7 million documents, 3.6 million encumbrance certificates, 2 million market valuation slips • eSeva center in Andhra Pradesh: 250 locations in 190 towns, Used by 3.5 million citizens, • eProcurement in Andhra Pradesh • Ahmedabad Municipal Corporation (AMC): 14 Civic Service Centers • Computerized Inter State Check Posts in Gujarat: 8 locations • eProcurement in Chile (Comprasnet) • Income Tax on-line in Chile

9. Methodology Used for Assessment • Selected 10 mature, wide scope and scale projects of e-delivery of services. • Collected data through structured survey from clients, employees, supervisors using counterfactuals ( for old non computerized delivery and new e-delivery system) • Survey instrument customized to each project, adapted in local language • Sampled 240 clients, 30 employees randomly from locations stratified by activity levels and remoteness • Collect data on investments, operating costs, activity levels, revenues, employee strength from agencies. • Develop a case study-organizational context, process reform, change management.

10. Profile of AMC Respondents • Partial data of 139 (out of 240) citizens • 40% graduates, 50% schooled, 10% barely literate • 16% workers, 50% business, 10% white collar workers, 24% supervisor/management • Aware for 28 months, Average Income Rs 7000 pm • Urban population: 38% within 1-3 kms of a center, 25% 4-6 Kms, 20% < 1 Km • Payment of property tax, birth/death registration

11. Desirable Attributes in Service Delivery at AMC Centers • Less time and effort 48% • Less corruption 41.7% • Less cost to the citizen 29.5 • Greater Transparency 27.3 • No need for agents 20.1 • Good complaint handling 19.4 • Convenient time 16.5 • Privacy and security 15.1 • Waiting facilities 12.2 • Helpful attitude of civil servants 10.8 • Error free transactions 9.4 • Fair treatment 8.6 • Accountability 7.9 • More predictable outcome 5.4 • Clarity/simplicity of procedures 5.4

12. AMC: Impact on Client Costs

13. Client Perceptions (Ratings on a 5 Point Scale)

14. Attitude to eGovernment

15. Preliminary Observations • Overall Impact • Significant positive impact on cost of accessing service • Strong endorsement of eGovernment • Variability across different service centers of a project • Reduced corruption-outcome is fragile • Any type of system break down leads to corruption • Agents play a key role in promoting corruption • Private operators also exhibit rent seeking behavior given an opportunity Systematizing queues and appointments helps prevent break down • Small improvements in efficiency can trigger major positive change in perception about quality of governance. • Challenges • No established reporting standards for public agencies- In case of treasuries, the AG office has more information on outcome. • What is the bench mark for evaluation-past performance or potential? • Public agencies are wary of evaluation-difficult to gather data