Evolution of a Geospatial Clearinghouse: From Data Distribution to Map Services

1. Evolution of a Clearinghouse:From FTP to Map ServicesESRI International Users Conference, August 10th, 2006 Maurie Kelly, Ryan Baxter, David Walrath Pennsylvania Spatial Data Access www.pasda.psu.edu

2. In this presentation…. • Historical background of clearinghouse • Past and current challenges and initiatives • Related projects and experience FTP & Metadata Map Services & Customization Tools Relational Database Development

3. www.pasda.psu.edu

4. PASDA … PASDA is the Commonwealth’s node on the NSDI, participates in GOS, the Geography Network, and the NBII Serves as a resource for locating data throughout PA Provides widespread sharing of Geospatial data Eliminates the creation of redundant data sets • Is the official public geospatial data clearinghouse for Pennsylvania

5. PASDA … • A collaborative effort of: • PA Geospatial Technologies Office • Penn State Institutes of the Environment • State agencies, programs such as PAMAP, and • data partners throughout the region • Established in 1996 to serve PA DEP data (35 data sets) via FTP • Expanded in 1999 to officially serve as the public geospatial data clearinghouse for PA

6. PASDA… PASDA • Provides access to about 50,000 data files • Celebrates its 10th anniversary this year and is one of the US’ largest and most established clearinghouses • Offices are located on PSU UP Campus

7. PASDA… Planar Distance Units:meters Geodetic Model: Horizontal Datum Name:North American Datum of 1927 Ellipsoid Name:Clarke Data Storage and Access Planar Distance Units:meters Geodetic Model: Horizontal Datum Name:North American Datum of 1927 Ellipsoid Name:Clarke Metadata Outreach Planar Distance Units:meters Geodetic Model: Horizontal Datum Name:North American Datum of 1927 Ellipsoid Name:Clarke Planar Distance Units:meters Geodetic Model: Horizontal Datum Name:North American Datum of 1927 Ellipsoid Name:Clarke Planar Distance Units:meters Geodetic Model: Horizontal Datum Name:North American Datum of 1927 Ellipsoid Name:Clarke Training & Assistance Data & Metadata • PASDA info. • GIS tutorials Users PASDA • Our services include:

8. How PASDA Works PASDA Staff -Storage -Management -Assistance • Benefit to Providers: • Mechanism for data distribution • Publicity • Provide service to PA • Benefit to Users: • Can access & use valuable data • Avoid duplication of effort • FREE • Data Partners • Data Collection • Mapping • Documentation Distribution Via the Web -Retrieval by user

9. Metadata… PASDA:Data Cataloging • Description: • Abstract:The U.S. Environmental Protection Agency's (EPA) Reach Files are a series of hydrographic databases of the surface waters of the continental United States … Originator:U.S. Environmental Protection Agency/Office of Water/OST Basins WHO Time Period of Content: Time Period Information: Single Date/Time: Calendar Date:1994 Currentness Reference:publication date Status: Progress:Complete Maintenance and Update Frequency:None planned Spatial Domain: Bounding Coordinates: West Bounding Coordinate:-74.739 East Bounding Coordinate:-73.652 North Bounding Coordinate:45.156 South Bounding Coordinate:44.496 Keywords: Theme: Theme Keyword Thesaurus:None Theme Keyword:RF3 alpha Hydrography River Reach streams stream network rivers hydrography ArcInfo navigation drainage Place: Place Keyword Thesaurus:Geographic Names Information System Place Keyword:New York NY New York English-Salmon Access Constraints:none Use Constraints:none Point of Contact: Contact Information: Contact Organization Primary: Contact Organization:U.S. Environmental Protection Agency Contact Person:Dan Parker Contact Address: Address Type:mailing and physical address Address:401 M Street SW (4503F) City:Washington State or Province:DC Postal Code:20460 Country:United States Contact Voice Telephone:1-800-424-9067 Data Set Credit:McKay, Lucinda; Sue Hanson; Robert Horn; Richard Dulaney; Alan Cahoon; Mark Olsen; and Thomas Dewald, 1994. The U.S. EPA Reach File Version 3.0 Alpha Release (RF3-Alpha) Technical Reference. U.S. Environmental Protection Agency, Washington, DC WHY Title:USEPA/OW River Reach File 3 (RF3) Alpha for CONUS, Hawaii, Puerto Rico, and the U.S. Virgin Islands Info about the data Indirect Spatial Reference:English-Salmon - New York Direct Spatial Reference Method:Vector WHAT WHEN Calendar Date:1994 Planar Distance Units:meters Geodetic Model: Horizontal Datum Name:North American Datum of 1927 Ellipsoid Name:Clarke 1866 Spatial Domain: Bounding Coordinates: West Bounding Coordinate:-74.739 East Bounding Coordinate:-73.652 WHERE HOW

10. Types of data on PASDA • Vector and Raster Data • DEMs • DLGs • DRGs • DOQQs • PAMAP • River Conservation Plans • Census Data • Land Use/Land Cover • Wetlands • Historic Markers • Fish Species

11. PASDA Statistics – End of 2005 • Data Storage • 6.5 Terabytes of data • 3 Terabytes in database for interactive mapping/interactive download • Access (during last12 months) • 11.7 million hits (from 253,000 users) • 631,000 data downloads requests • 3.9 Terabytes of data downloaded • 828,000 online maps generated via web GIS

12. PASDA Statistics

13. PASDA Statistics

14. To whom are we providing assistance?

15. First Challenge—Expand Services and Capability beyond FTP • #1 Challenge for any clearinghouse—Technology • Technology changes rapidly both for clearinghouse and for users • Changes in technology mean changes in hardware, software, and architecture of clearinghouse as well as changes in personnel and training needs. • Requires regular planning, benchmarking, and vision • Needs Analysis • 1999 PASDA and ATS of Lancaster undertook a needs analysis to define user requirements for role as official public geospatial data clearinghouse for PA • User requirements included: customization options, web mapping applications, tutorials and guides, more data!! • Impact on Clearinghouse • Increased training needs and knowledge base for staff • Increased technology, software, and hardware needs • More users

16. Results of Needs Analysis Initiative • Strategic Planning effort undertaken and implemented in 2000 facilitated by Avencia Inc. of Philadelphia. • By 2001, the following were complete • Development of first relational database using Db2 & SDE • Pennsylvania Atlas ArcIMS application • First iteration of Data Wizard customization reprojector/clipping tool • By 2002… • Statewide DOQQ mosaic in database • Clipper/reprojector in regular use • Pennsylvania Atlas ArcIMS application clipping/reprojecting tool developed

17. By 2003 Technology Changes on the horizon….. • By 2003 changes in technology were on the horizon… • Mid 2003 User requirements session held. Examined existing tools, data, and interface. Acquired input on changes and expectations of users— • Clearinghouse needed integrated web interface • Clearinghouse needed streamlined metadata • Clearinghouse needed integrated option for accessing imagery • Website and html needed to be streamlined • Search mechanisms needed to be integrated into single environment • Post user session—staff held strategic planning session and developed new multiyear plan to implement additional enhancements to clearinghouse.

18. Challenge…. How to make all of the options accessible? Solution 1: • Approach had been to create new tools for each need • Approach had been to add links and content to site • Result 1: • Confusing Website • Countless links & access points

19. Challenge Part 2…. How to make all of the options accessible? Solution 2: Our approach this past year Complete redesign of the clearinghouse…search interface, metadata architecture, data wizard, integration of applications and map Services into search interface. • Result 2: • Simplified Website • Single, integrated access point to all data types & services

20. Current Architecture… • “Data cart” option added to Data Wizard • Users can search for, select, add data to cart, reproject/clip data, and zip all the data in their cart together into one downloadable file. • Imagery viewer and download utility created for all imagery in database Clip & Reproject Google Earth WMS MapServices WebGIS Applications FTP .ZIP Files ArcIMS MapServices NSDI – GOS Harvesting

21. Current Architecture… How did we accomplish this? One central database table • One record for each ‘dataset’ in the clearinghouse • Basic metadata elements (title, keywords, date…) • Links to FTP files • Links to SDE database feature classes • Links to XML metadata files • Links to WebGIS application URLs • Links to ArcIMS & WMS MapServices

22. Current Architecture… xml FTP .zip xml xml MapServices SDE WebGIS Central table drives the Website and integrates all services Website Direct Download Clip, Reproject & Download Central Table Metadata NSDI -- Geospatial One-Stop ArcIMS -- WMS --Google Earth

23. Data Access Tools Search by keyword, county, quadrangle, etc. Download .zip files View aerial photos; zoom to an address Click on a quarter-quadrangle to download it FTP DOQQs View & Interact with data on a map Clip, reproject & download Map & download census 2000 data All SF1 & SF3 census variables for PA PA Atlas PennCAT - Geography Network Catalog of MapServices Census 2000 Unified interface for all data Add Data to your ‘Data Cart’ Clip, reproject and download multiple datasets at one time Live Internet Data Data Access Wizard

25. View XML metadata

27. Download via FTP

28. Link to separate application

30. Compile datasets for clipping & reprojecting

32. Dynamic Content

34. Specialized Applications • Imagery Viewer and Download Utility • 4 Terabytes of imagery including aerial photos • DRGs, DOQQs • 5 unique tiling schemes • Multiple years of data—spans period from 1993-2005 • All data can be queried via single interface • Search by address,quadrangle, county • Select image and provide with download options http ftp mr. Sid or tiff

35. Specialized Applications:Imagery Viewer

36. Specialized Applications:Imagery Viewer

37. Specialized Applications:National Weather Service Forecast Map Services Near real time data from NWS including:Apparent Temperature, Dew Point Temperature,Maximum Temperature, Minimum Temperature, Precipitation Amount, Probability of Precipitation, Relative Humidity, Sky Cover, Snow Amount, Temperature, Wave Height, Wind Direction, and Wind Speed.

38. Specialized Applications: Tropical Depression Ivan • The Ivan viewer was developed to provide information about flood • damage following the storm to the public, emergency managers, • And other stakeholders in the Commonwealth. • Ivan includes pre and post flood aerial photos, pictures taken at • ground level of flood damage, and supporting framework data.

39. Ivan Imagery Viewer Interface

40. City Island after the flood…

41. Pre-flood imagery of City Island

42. Flood damage—Photo taken on site and contributed to Ivan viewer by citizen

43. Pennsylvania Initiatives that use the clearinghouse…

44. National Information Infrastructure Initiatives…

45. Conclusion • Clearinghouses have evolved from simple FTP • sites to complex information infrastructures • Clearinghouses serve multiple purposes—access to data • Access to services, access to applications, metadata • development, and outreach. • PASDA has undergone regular strategic planning to • develop new initiatives and to benchmark progress. • PASDA has transitioned from a traditional structure to one that • encompasses data, services, and tools for its users. • Map services and applications are now the hallmark of the • clearinghouse • Free access to data supports businesses, non-profits, and other • projects and programs throughout the state and nation.

46. Questions & CommentsESRI International User Conference, August 10th, 2006 Please contact us at pasda@psu.edu if you have any questions or comments. More information is available at http://www.pasda.psu.edu/ Maurie Kelly, Ryan Baxter, David Walrath Pennsylvania Spatial Data Access www.pasda.psu.edu