Geog 469 GIS Workshop

1. Geog 469GIS Workshop Evaluating Implementations of GIS

2. Outline • How do we evaluate a GIS (project) implementation? • What is a successful GIS implementation? How do we define the “success” of GIS implementation? • What are some additional perspectives on GIS implementation?

3. How do we evaluate a GIS (project) implementation? Why is “project” in parentheses?

4. Evaluating GIS • Ad-hoc method • If time is limited, evaluate project based on your insight and previous interaction with clients • Benefit-cost analysis • Compare benefit to cost to determine whether the project is financially justifiable (at any stage) to move forward • Usability testing • Test whether clients’ needs have been met; use prepared questionnaire, interview, observation

5. Ad-hoc • What does your experience tell you? • Data • Software • Hardware • Personnel • Timing of Procedures and Institutional Arrangements • Motivations

6. Benefit-cost analysis • Enumerate all possible costs spent on this project; classify them into one-time cost (data, hw, sw) and operating cost (maintenance, overhead, salary, training) • Cost of leasing workstation for 10 weeks can be obtained from estimates • Enumerate all possible benefits; classify them into tangible (cost reduction, efficiency) and intangible one (improved decision making, better public image, equity); assign monetary value wherever possible • Benefit of using GIS for making map can be calculated from saved time multiplied by salary rate (= actual time spent - time spent if manual method is used) • Discounting: If B & C estimates are based on different time period, discount them given inflation rate • BCR: Calculate the ratio of benefit to cost to determine whether the ratio is over 1 (i.e. economically justifiable) • Time: If possible, project the cost/benefit into the foreseeable future; draw the curve; determine payback period

7. Usability testing • Develop test objectives • What am I testing? e.g. effectiveness of map design, accuracy of data geocoded • Acquire participants • Sampled from potential end users • You can screen them through pre-survey form (ask their age, education level, gender, occupation, and so on wherever appropriate) • Collect qualitative and quantitative performance and preference measurements • Questionnaires • Which map do you prefer? Does prepared map design work? • Task scenario if applicable • Describe the context for use of the information product • Recommend improvements to the design of information product • Read materials on usability testing at http://www.lib.washington.edu/usability/

8. Use an Evaluation to… • Draw conclusions • About significance of what you found • About the effectiveness of your methods • About whether needs were met • About whether research question was answered • Provide recommendations • For improving the completed work • For additional work

9. 2. What is a successful GIS implementation? How can we define the “success” of GIS implementation?

10. Elements of GIS project success and failure

11. Validity of implementation success • Technical validity • The system to be implemented works • no bug, data seems clean • Organizational validity • Congruence between the organization and the system to be implemented • The system is accepted and used by the organizational members • Organizational effectiveness • Improvement in efficiency and effectiveness • Improved decision making • Even perceived as a process of organizational change • Some GIS changes the way the organization does their business From Obermeyer and Pinto 1995

12. What are the criteria that define the success of GIS implementation? • It will be useful to address this question based on two different perspectives • Content model • GIS project is viewed as collection of elements • Process model • GIS project is seen as an activity composed of phases

13. GIS Implementation: Content Models • Focus on key (critical) success factors • Technical aspect • Competent technical support • Staff’s competence in GIS knowledge • Managerial aspect • Clearly defined goals • Sufficient resource allocation • Top-management support • Implementation schedule • Adequate communication channels • Feedback capabilities

14. GIS Implementation: Process Models • Focus on key phases in GIS implementation process • Initiation • Planning phase of the implementation process • Related to “strategies” • Implementation • Actual process (or the action) of the implementation • Related to “tactics” Conceiving GIS implementation as the two-stage “process” helps us look at dynamic nature of GIS in the context of strategic goals of organizations

15. Strategy/tactics effectiveness matrix high Effectiveness of tactics low low high Effectiveness of strategy Successful GIS should be both strategically and tactically effective From Schultz, Slevin, and Pinto (1987)

16. Success in the short-term might become failure in the medium- to long-term • Early false start does not necessarily imply that the project is destined to fail • Bear in mind the dynamic nature of the environment in which GIS is embedded when making assessments

17. 3. What are some additional perspectives on GIS implementation?

18. The implementation of GIS is not merely a technical process, nor one that simply requires the formulation of rational management strategies. • It is as much a social and political process which takes place within unique organizational cultures subject to changing conditions. • Adapted from Campbell and Masser 1995

19. How do organizations interact with technology such as GIS? • Depends on how you see organizations, for example, as… • Machine, system, or culture Three perspectives (theories) on organization activity… • Technological determinism • Widespread adoption and implementation of GIS is dependent upon the quality of the technology • Managerial rationalism • GIS implementation success is based on good management strategy that produces results (e.g. corporate approach) • Social interactionism • Good understanding of the social organization (social attitudes or institutional developments) leads to GIS implementation success

20. Technological determinism From Campbell & Masser, “GIS and Organizations” 1995

21. Managerial rationalism

22. Social interactionism

23. Case studies suggest… • A significant number of GIS (projects) failed when… • User needs are not well-defined • GIS design is not well matched to specific tasks • Ownership and control of GI is not well determined • Not clear how information is to be used in decision-making process • Information sharing is not well coordinated • Organizational members are not ready to adopt GIS or resistant to using GIS

24. Review of critical issues to successful GIS implementation • Identification of simple applications producing information is fundamental to the work of potential users • Awareness for limitations of the organization in terms of accepted practices and available resources is crucial to know limits • User-directed implementation set within a framework based on the commitment and real participation of staff throughout the organization • Ability to create stability or cope with change fosters success through adaptive management…continual review is critical to successful implementation. Campbell & Masser 1995