Satella: Tool for Peer to Peer Landsat Data Sharing

1. Satella: Tool for Peer to Peer Landsat Data Sharing Bruno Margerin (SSAI) John Owens and Dr. Chris Justice (UMd)

2. Rationale • How can scientists share data easily ? • we need a way let data users know - who has what data. • we need a tool to access and provide data. • we want to minimize the burden from those sharing data • We need to reduce the costs of managing NASA data • We need to promote reduce obstacles to increased use of NASA data • The internet provides a mean for electronic data exploration and delivery • Can we take advantage of IT currently being developed in the private sector and apply these advances to our science data needs ?

3. NASA Data Needs: An example • NASA has purchased large volumes of Landsat data to support the science community • Landsat Pathfinder (Landsat 5) Regional Data Buys • Earthsat Data Buy 1990 • Landsat 7 Data Purchases for R and A (LCLUC) • Landsat 7 Earthsat Data Buy Data 2000 • These data are held by individual PI’s – most of whom are willing and able to share data with other scientists • Once purchased by NASA, these Landsat data can be shared openly for science use • We need a way to facilitate data sharing and reduce costs of data • We need to promote free and open sharing of science data • We need to move the burden of data management away from centralized systems to more distributed systems • Other opportunities – MODIS Data, ASTER Data, Landsat Derived Products, Model Output

4. Requirements for the System • A unified method of access to heterogeneous, distributed archives • Better utilization of Internet bandwidth, storage • Low management and maintenance • Easy to install and use • Highly scalable performance

5. The Satella Concept • A scalable, robust system for data search and distribution over a large collection of heterogeneous providers • A means for individual scientists to obtain and share satellite data

6. Satella Conceptualization • A distributed, self-organizing network based on peer-to-peer topology • Integrating the data available in individual archives • Through a freely available client software for search and transfer of the data

7. Satella: Peer 2 Peer System • Privileges Local Peer Interactions. • Replicates information towards Consumer • Discover and Adapt dynamically to new information sources. • Build complex services by combining simple ones. • Increase performance as system ages and more peer participates. • Protect data by replicating information • Build reliability from interchangeable peers

8. Based on JXTA P2P Protocols and Implementation • Open Network Programming platform for P2P Computing • Set of Protocols for interoperating (XML based) • Language and network Agnostic Technology. • Open Source Technology: http://www.jxta.org • Virtual network overlay • Generalize the P2P problem • Solve key technical challenge (security, stability, scalability ...) • Interoperable, Platform independance, ubiquity • 136,000 Downloads, 40+ Projects, 7500 Members (4/25/2001 – 12/8/2001) From http://www.jxta.org/community

9. Software Architecture 1/2 • Prototype Service software allowing sharing and search for Landsat 7 ETM datasets. • 2 Set of Service: • Root Peer Service: • Maintains and provides secure access to Satella Peer Group. • Provide initial topology information of Net Peer Group and Satella Peer Group • Provide Relay and Rendez Vous capability for Satella Peer Group and Net Peer Group. • Data Peer Service: • Provide P2P File Sharing capability • Automated Landsat 7 Metadata creation • Search and Download of files based on their metadata. • Fully Open Source

10. Software Architecture 2/2 Root Peer Service HTTP XML • N-Pier Architecture • UDDI/WSDL/SOAP Web Service Ready Architecture. XML Data Peer Service Tomcat Web Server Web Browser Other P2P Data & Root Svc Interact Comm. Objects MVC: Turbine + Velocity RMI Interface RMI + Web Based GUI RMI Root Peer Service XML XML Data Peer Service UDDI Other Web Service Associate GUI Other P2P Svc Interact Comm. Objects WSDL/SOAP Interface SOAP

11. Satella Development to Date • Phase I:Internal Proof of concept and feasibility. • AWT/SWING based GUI • Search on File Name Only • Basic, download capability. • Partially Multi-Threaded. • Phase II: N-Pier Architecture • Service Based Architecture • 3- Pier Model View Controller (MVC) Web Based GUI • Automatic Landsat 7 Metadata Generation • Improved Download capability. • Search on Landsat 7 Metadata (in progress).

12. Satella Caveats • JXTA is one competing web services platform (there are other competitors e.g. Microsoft Hailstorm), will it be the eventual winner? Mitigated since Other platforms are less mature especially in the P2P Architecture. • Distributed archive depends on participation by potential users/contributors. Success of Napster/Gnutella music file sharing networks illustrate people willing to give as well as take. • Programmatic guidance could encourage PIs and providers to play an active role. Satella would not replace the need for a long-term archiving of satellite data but will share the load and reduce the cost of data distribution

13. Satella: The Way Forward • Future Improvements: • Continued improvements for Satella network performance and reliability • Evolution to a UDDI/WSDL Web Service • Enhanced Semantic, distributed search • Downloading of a single file from multiple available peers • Other improvement dictated by the user community.

14. Satella project Contributions to the Open Source Community • Involvement (Committer or Contributor) on 9 JXTA projects • Provided tutorial for the creation of a Secure Peer Group on JXTA’s Java implementation build 34c (10-12-2001) • Provided segment “Creating a Secure Peer Group”to“Project JXTA: Java[tm] Programmers Guide”. • In collaboration with Robin Johnson, Steer the architectureof the development of Metadata Support into CMS (Content Management System), JXTA File Sharing engine.

15. A look at the future • The costs of data distribution are becoming unaffordable – DAAC’s are struggling to manage the large data volumes - new and innovative solutions are needed to improve data access and reduce costs to NASA and the PI – we need to invest in new approaches and long term solutions • Data distribution is low compared to the size of the archives and the potential user community – we need to stimulate data use • NASA Science requires regional time-series analysis i.e. large volume data sets giving an increased burden to existing data systems already proving to be unaffordable • The NASA Earthsat Data Buys provide an important asset for the global change research community – need to ensure their widest possible use – extensive data sharing is now feasible • Future Landsat Continuity Missions will hopefully allow free and open sharing of 30m data • The user community can and is willing to play an increasing role in data management – we need to develop sustainable ways of doing business • In the context of the NEWDISS program NASA needs to support innovative use of advances in information technology to solve its data management challenges

16. Credits: • John Owens and Dr. Chris Justice From University of Maryland, Dept of Geography. • Bernard Traversat and Juan C. Soto From Sun Microsystems • All participating JXTA members.

17. Satella Phase I: SnapShots

18. Contact Information • Bruno Margerin: bruno@turbodog.gsfc.nasa.gov • John Owens: jowens@umd.edu • Dr. Chris Justice: justice@hermes.geog.umd.edu • Web Site (to come): http://satella.geog.umd.edu