An Introduction to an Index of Geospatial Web Services

1. An Introduction to an Index of Geospatial Web Services Jeremy D. Bartley www.mapdex.org Kansas Geological Survey

2. Advantages of Building an Index of Dynamic Map Services Why do Geographers/planners/scientists care if there is an easy way to find spatial data over the internet? “Early GIS was impeded by the lack of good algorithms and powerful computers to analyze data and by the difficulties associated with digitizing, which were such that 80% or more of a project’s resources were often consumed in the task of converting paper records to digital databases. Today, a new source of frustration has emerged. Despite all of the on-line digital data now available and the vastly increased power of GIS, it is still common to spend 80% or more of a project’s resources on searching, discovering, retrieving, and reformatting data.” Forward by Michael Goodchild in Internet GIS by Peng and Tsou 2003 Jeremy D. Bartley www.mapdex.org Kansas Geological Survey

3. Advantages of Building an Index of Dynamic Map Services • Many groups have built catalogs of spatial data. • State Clearinghouses • Federal Government Initiatives • They all rely on manual publishing of metadata • More Metadata, more information • Less actual data, more work for the provider Jeremy D. Bartley www.mapdex.org Kansas Geological Survey

4. Advantages of Building an Index of Dynamic Map Services • The fundamental difference between Mapdex and other geospatial catalogs is that Mapdex searches ALL available map services that are openly published on the Internet via ArcIMS or through WMS compliant servers serving through version 1.1.1. • Other data catalogs rely on the owners of the data to “publish” metadata about the mapservices and the underlying data to the catalog • ArcIMS server administrators can enable authentication to not allow Mapdex (and anybody else) to harvest (or use) information about their mapservices without a username and password Jeremy D. Bartley www.mapdex.org Kansas Geological Survey

5. Advantages of Building an Index of Dynamic Map Services • Current Mapdex Statistics • Successfully Indexed: • Almost 1,500 servers containing… • Approximately 25,000 live mapping services (ArcIMS Mapservices spanning… • Over 370,000 spatial layers (300,000 feature layers, 70,000 raster layers) • Over 100 WMS 1.1.x services over 60 servers Jeremy D. Bartley www.mapdex.org Kansas Geological Survey

6. How Map Services Work You get an XML response containing key information about the mapservices that are available to be used by any client. Each server has multiple mapservices available from it: If you post a getclientservices XML request to the ArcIMS server… http://seamless.usgs.gov/servlet/com.esri.esrimap.Esrimap?ServiceName=catalog&ClientVersion=4.0.1 Jeremy D. Bartley www.mapdex.org Kansas Geological Survey

7. How Map Services Work You get an XML response containing key information about the mapservice (in this case Geology). Information like: extent, projection, layer envelope, layer name, layer type, column name, column type, etc. Each mapservice has multiple spatial layers that are available from it: If you post a getserviceinfo XML request to the ArcIMS server… http://seamless.usgs.gov/servlet/com.esri.esrimap.Esrimap?ServiceName=geology&ClientVersion=4.0.1 Jeremy D. Bartley www.mapdex.org Kansas Geological Survey

8. Finding servers that publish mapservices • Number one indexing application: GOOGLE • The main way Google builds its index is by finding pages that link to other pages • Mapdex finds map servers that publish data in a specific format (web service) and are indexed by Google • Most dynamic mapping applications can by identified by either vendor specific content or common key words (zoom, pan, map) within the html page Jeremy D. Bartley www.mapdex.org Kansas Geological Survey

9. Discovering Published mapservices • Search Google for possible mapserver sites. • For each server, check to see if it is publishing mapservices. • If it is a match, then save the response into the database. NULL NULL NULL YES Jeremy D. Bartley www.mapdex.org Kansas Geological Survey

10. Storage of mapservice configuration • Now we have access to over 370,000 GIS layers • This includes approximately 300,000 vector layers (point, line, polygon) and 70,000 raster layers (DEM, orthophotos, satellite images, scanned maps) Jeremy D. Bartley www.mapdex.org Kansas Geological Survey

11. Spatial Envelopes for mapservice Layers The catch--not all mapservices declare their projection In terms of layers that means of the 365,000 layers that Geodiscovery has indexed, approximately 150,000 layers either have their projection set or are assumed geographic Jeremy D. Bartley www.mapdex.org Kansas Geological Survey

12. Spatial envelopes for mapservice layers • Why is this important… • If we know the mapservice projection we can reproject the various layer envelopes to a common projection (geographic-latitude/longitude) • Now we can query mapservice layers by geographic location (all layers within state, county, zip code, etc.) Jeremy D. Bartley www.mapdex.org Kansas Geological Survey

13. Updating the Index • Mapdex is continually updating and discovering new services • Mechanisms are in place to: • Discover new servers • Discover new mapservices • Discover new layers • Remove old servers & services • Update envelope index Jeremy D. Bartley www.mapdex.org Kansas Geological Survey