Enhancing Space Data Access with SECAA Web Services

1. Importance of SECAA Web Services for the Integrated SEC Data Environment <http://spdf.gsfc.nasa.gov/> Robert M. Candey (Code 632) Bernard T. Harris (Code 583) Reine A. Chimiak (Code 583) David B. Han (Code 586) Rita C. Johnson (QSS, Code 630) Colin A. Klipsch (QSS, Code 630) Tamara J. Kovalick (QSS, Code 630) Howard A. Leckner (QSS, Code 630) Michael H. Liu (Raytheon ITSS, Code 630) Robert E. McGuire (Code 632) NASA Goddard Space Flight Center, Greenbelt MD 20771

2. SECAA/SPDF providing services today • Accessible via FTP, HTML browser, and now web services APIs and new clients that use the web services<http://spdf.gsfc.nasa.gov/> • Space physics data access - Coordinated Data Analysis Web (CDAWeb) and Coordinated Data Analysis System (CDAS) • World-wide data finding across public FTP sites and in CDAWeb (Public data file finder) • Return URLs to files matching a range of time in many FTP and some HTTP data sites (filenaming format based on strptime % strings) • Example XML (required tags in bold) <dataset ID="ace_mfi_h0_16sec_cdf_cdaweb" nssdc_ID="None” serviceprovider_ID="ac_h0_mfi"><access filenaming="ac_h0_mfi_%Y%m%d_%Q.cdf" protocol="ftp" subdividedby="%Y" timerange_start="1997-09-02 00:00:12" timerange_stop="2004-03-11 23:59:46"> <URL>ftp://cdaweb.gsfc.nasa.gov/pub/istp/ace/mfi_h0</URL> </access> </dataset> • Orbit locations - Satellite Situation Center Web (SSCweb) • Data Format Translation web service (DTWS) • Radbelts (OMNIweb) solar and magnetospheric indices • More to come

3. Data File Finder • Using just several pieces of metadata, we can return URLs to files matching a range of time in many FTP and some HTTP data sites • Filenaming format based on strptime % strings • Example XML (required tags in bold) <dataset ID="ace_mfi_h0_16sec_cdf_cdaweb" nssdc_ID="None”serviceprovider_ID="ac_h0_mfi"><access filenaming="ac_h0_mfi_%Y%m%d_%Q.cdf" protocol="ftp" subdividedby="%Y" timerange_start="1997-09-02 00:00:12" timerange_stop="2004-03-11 23:59:46"> <URL>ftp://cdaweb.gsfc.nasa.gov/pub/istp/ace/mfi_h0</URL> </access> <mission_group ID="ACE" serviceprovider_ID="ACE"><description short="Advanced Composition Explorer"/> </mission_group> <observatory ID="ACE" nssdc_ID="None" serviceprovider_ID="AC"> <description short="Advanced Composition Explorer"/> </observatory><instrument ID="MAG" nssdc_ID="None" serviceprovider_ID="MAG"><description short="ACE Magnetic Field Instrument"/> </instrument><data_producer affiliation="Bartol Research Institute" name="N. Ness" title="None"/><other_info> <link URL="http://www.srl.caltech.edu/ACE/ASC/level2/index.html" title="the ACE Science Center Level 2 Data website">Release notes and other info available at </link> </other_info> <spase:product_ID>ace_mfi_h0_16sec_cdf_spase</spase:product_ID><instrument_type ID="Magnetic Fields (space)" serviceprovider_ID="Magnetic Fields (space)"/><description short="H0 - ACE Magnetic Field 16-Second Level 2 Data - N. Ness (Bartol Research Institute)"/> </dataset>

4. Data Translation Web Service (DTWS) • Translation between one or more local or remote files into other data formats • Formats: CDF, netCDF, HDF4, HDF5, FITS, • Interoperability with other data formats using XML: CDF Markup Language (CDFML), FITSML, XDF Version of CDFML (CDFX-ML), HDF in XML, Native Data Format (NDF) in XML, ASCII (data object mapping via Extensible Style Language (XSL)) • Input: Files to be translated, translation mode • Output: Direct or e-mail notification with URL links to translated files • <http://nssdc.gsfc.nasa.gov/cdf/html/dtws.html>

5. Satellite Situation Center (SSCweb) • Service for geocentric spacecraft locations with geophysical regions and mappings along magnetic field lines • Supports multiple coordinate systems • Extensive business logic and comprehensive list of active and past space physics missions • Supports complex multi-satellite and ground conjunction queries • Outputs listings and graphs

6. Coordinated Data Analysis System (CDAS) Web services • Simultaneous multi-mission, multi-instrument selection and comparison of science data via graphics, digital listings, file retrieval and merged/subsetted CDF creation • Now comprises >600 datasets of current and past space missions and ground-based facilities (>1M files of science data) • Very popular with 165k user sessions, 94k plots, 622k FTP requests, 67k ASCII listings, 586 CDF create requests, 2882 file retrieval requests in FY2004 • CDAWeb data available through: • FTP file access, including software and documents • Web browser • Web services in addition • Latter two provide data listings, plots, original data files and combined files

7. What is Common Data Format (CDF)? • Self-describing platform-independent scientific data format (object oriented like HDF5) • Most widely used format in Space Physics • ISTP, Stereo, TIMAS • Available on all major computer platforms • Interfaces supported: C, Fortran, Java, and Perl • CDF FAQ <http://nssdc.gsfc.nasa.gov/cdf/html/FAQ.html>

8. Software for CDF • IDL, MATLAB, IBM DX, FlexPro, Application Visualization System • Goddard-developed Web-based systems leverage CDF power • CDAWlib <http://spdf.gsfc.nasa.gov/CDAWlib.html> • Powerful set of IDL routines, including reading/writing CDFs • Available at <ftp://cdaweb.gsfc.nasa.gov/pub/CDAWlib/> • Standard CDF tools • CDFcompare, CDFconvert, CDFedit, CDFexport, CDFstats, SkeletonTable, SkeletonCDF distributed as part of the standard CDF distribution package • SKTeditor metadata editor applying the ISTP Guidelines • MakeCDF converts ASCII and binary custom formats into CDF

9. How to setup your own web service:SECAA implementations • Server Software • Apache HTTPD • Apache Jakarta Tomcat 5 (triggers on “/WS/” in URL) • Java Runtime Environment (JRE) 1.4.x • Sun JAX-RPC 1.1 runtime libraries (could replace with Java Axis) • Apache Jakarta Commons/Net 1.1.0 library (for FTP access) • CDAS and SSC Web application implementing web services • CDAS and SSC core (existing) software • Services described in WSDL files • Development Software • Sun Java Web Services Developers Pack (JWSDP) 1.3 • Sun Java Developers Kit (JDK) 1.4.2 • Java Client Software • Application • Java Runtime Environment (JRE) 1.4.x • Sun JAX-RPC 1.1 runtime libraries • CDAS/SSC Web service client libraries

10. Read create Create execute SOAP request/ response HTTP GET data/response Get dir Get dir Get data file CDAWeb Server CDAS data file/ plot/listing IDL get file CDAS metadata database Apache Tomcat Apache HTTPD read our WS Java code read non-CDAWeb FTP/HTTP site metadata (XML) HTTP Server FTP Server WS Client Remote Computer Remote Computer Remote Computer

11. Tie together services One scenario (connected with XML event lists): • Search bowshock event database • Query indices database for times of no sun spots • Correlate with spacecraft on same field lines • Search for data files • Translate files into preferred format • Plot

12. Tie together with clients • SECAA-provided or external clients (own customer of web services and dependent on service stability) • HTML web page access always for simple access • TIPSOD (Tool for Interactive Plotting, Sonification and 3-D Orbit Display) • CDAS Java client of getting CDAWeb data listings, plots, files (via CDAWlib) and URLs to files on other systems (via FTP) (functions to be merged soon): • Original CDAS Client (no FTP support) • <http://cdaweb/~rchimiak/cdasClient> • CDAS FTP Client (no CDAWlib support) • <http://cdaweb/~rchimiak/cdasFtpClient> • Easy client install (WebStart) • Translator page • VSPO also using CDAS Web services

13. Tool for Interactive Plotting, Sonification and 3-D Orbit Display (TIPSOD) • Shows spacecraft moving in time and 3D space as an interactive movie, while user changes orientation and scale (zoom and rotation) • Interfaces with the new flexible SSC XML Web Services (expanding on operational system) • Exposes relationships between spacecraft visually (in 3D) and temporally • Easy to download and use (Web Start downloads or updates TIPSOD client, Java and Java 3D with a few clicks from the browser) • Runs on Windows, Unix, Linux, Mac OS X (requires Java3D) • Additional functions planned, such as sonification and connections to CDAWeb science data and logic

14. Examples of calling web services • Example IDL program (IDL 6.0 IDL-Java bridge) • Example Java client • Example JavaScript • Example Perl script • SSC Web Services <http://sscweb.gsfc.nasa.gov/WebServices/> • Same techniques used to call CDAS Web services <http://cdaweb.gsfc.nasa.gov/WebServices/>

15. How to use our web services:Example IDL program • Uses IDL 6.0 IDL-Java bridge feature to call the SSCWeb services <http://sscweb.gsfc.nasa.gov/WebServices/RsiIdlExample.html> • Get list of satellites on SSCweb and then get orbit locations for 1 day of Polar in GSE coordinates • Configure IDL-Java bridge provided with the IDL software • Add JAXRPC jar files and sscWebServices-client.jar to CLASSPATH • IDL> .run SSCexample.pro

16. SSCexample.pro rpcStub = OBJ_NEW('IDLJavaObject$Static$JAVAX_XML_RPC_STUB', 'javax.xml.rpc.Stub') rpcStub -> getProperty, ENDPOINT_ADDRESS_PROPERTY=endpointAddressProperty sscWsImpl = OBJ_NEW('IDLJavaObject$SSCWS_IMPL', 'gov.nasa.gsfc.spdf.ssc.SSCWS_Impl') ssc = sscWsImpl -> getSatelliteSituationCenterPort() ssc -> _setProperty, endpointAddressProperty, 'http://sscweb.gsfc.nasa.gov/WS/jaxrpc' sats = ssc -> getAllSatellites() gregorianCalendar=OBJ_NEW('IDLJavaObject$Static$GREGORIANCALENDAR','java.util.GregorianCalendar') calendar = gregorianCalendar -> getInstance() calendar -> getProperty, DATE=calendarDateProperty endDate = calendar -> getTime() calendar -> add, calendarDateProperty, -1 startDate = calendar -> getTime() coordinateSystem = OBJ_NEW('IDLJavaObject$Static$COORDINATESYSTEM', 'gov.nasa.gsfc.spdf.ssc.CoordinateSystem') coordinateSystem -> getProperty, GSE=coordinateSystemGseProperty sats = ['polar'] locInfo = ssc -> getSatelliteLocation(sats, coordinateSystemGseProperty -> toString(), 3L, startDate, endDate) locInfoBeginTime = locInfo[0] -> getBeginTime() locInfoEndTime = locInfo[0] -> getEndTime() coords = locInfo[0] -> getCoordinates() plots, coords[*, 0], coords[*, 1], coords[*, 2] OBJ_DESTROY, rpcStub, sscWsImpl, gregorianCalendar, coordinateSystem END

17. Example Java client package gov.nasa.gsfc.spdf.ssc.test; import java.io.*; import java.util.Date; import java.util.Calendar; import java.util.GregorianCalendar; import java.text.SimpleDateFormat; import javax.xml.rpc.Stub; import gov.nasa.gsfc.spdf.ssc.*; public class WsExample { public static void main(String args[]) throws Exception { System.setProperty("http.agent", "WsExample (" + System.getProperty("os.name") + " " + System.getProperty("os.arch") + ")"); Stub stub = (Stub)(new SSCWS_Impl().getSatelliteSituationCenterPort()); stub._setProperty(Stub.ENDPOINT_ADDRESS_PROPERTY, args[0]); SatelliteSituationCenter ssc = (SatelliteSituationCenter)stub; String[][] satellites = ssc.getAllSatellites(); for (int i = 0; i < satellites.length; i++) System.out.println(" " + satellites[i][0] + " " + satellites[i][1]); Calendar calendar = GregorianCalendar.getInstance(); Date endDate = calendar.getTime(); calendar.add(Calendar.DATE, -1); Date startDate = calendar.getTime(); String[] sats = new String[] {"polar"}; SatelliteLocation[] locInfo = ssc.getSatelliteLocation(sats, CoordinateSystem.GSE.toString(), 3, startDate, endDate); double[][] coords = locInfo[0].getCoordinates(); for (int j = 0; j < coords.length; j++) { for (int k = 0; k < coords[j].length; k++) System.out.print(" " + coords[j][k]); } } }

18. Example JavaScript (Mozilla-based browsers) <script type="text/JavaScript"> netscape.security.PrivilegeManager.enablePrivilege("UniversalBrowserRead"); var call = new SOAPCall(); call.transportURI="http://sscweb.gsfc.nasa.gov/WS/jaxrpc"; call.encode(0, "getAllSatellites", "http://sscweb.gsfc.nasa.gov/WS/SSCWS", 0, null, 0, null); var response = call.invoke(); if (response.fault == null) { var result = response.getParameters(true, {}); for(i = 0; i != result.length; i++) { var param = result[i]; var name = param.name; document.write("result[" + i + "].name = " + name + "<br>"); var value = param.value; for (j = 0; j < value.length; j++) document.write("value[" + j + "] = " + value[j] + "<br>"); } } </script>

19. Example Perl script #!/usr/local/bin/perl -w use SOAP::Lite; my $ssc = SOAP::Lite -> proxy('http://sscweb.gsfc.nasa.gov/WS/jaxrpc') -> uri('http://sscweb.gsfc.nasa.gov/WS/SSCWS'); my $result = $ssc -> getAllSatellites(); unless ($result -> fault) { print "getAllSatellites() returned:\n"; my $satellites = $result->valueof('//result'); foreach $satellite (@$satellites) { foreach $name (@$satellite) { print " $name"; } } } my $satVar = SOAP::Data->type('string')->name('String_1')->value('polar'); $result = $ssc -> getSatelliteResolution($satVar); unless ($result -> fault) { my $value = $result->valueof('//result'); print "getSatelliteResolution() = $value\n"; } $result = $ssc -> getSatelliteTime($satVar); unless ($result -> fault) { my $values = $result->valueof('//result'); print "getSatelliteTime() returned:\n"; print " $$values[0] - $$values[1]\n"; }

20. Web Services Definition Language (WSDL) • XML description of web service purpose, calling methods and location • WSDL file specifying web service can describe both SOAP and CGI interfaces • CGI calls are lighter and less opaque but not as powerful as SOAP calls. See REST discussions by Mark Baker <http://webservices.devchannel.org/article.pl?sid=02/12/10/1849249>, Paul Prescod <http://www.prescod.net/> and <http://www.xml.com/lpt/a/2002/02/06/rest.html> • Example CDAWeb CGI call for plots of the lat, long and altitude variables for NOAA5/6 datasets for 1 day: <http://cdaweb.gsfc.nasa.gov/cgi-bin/cdaweb/eval3.cgi? dataset=NOAA05_H0_SEM+NOAA06_H0_SEM& start=1980%2F11%2F17+03%3A00%3A00& stop=1980%2F11%2F18+03%3A00%3A00& auth=hello&index=radiation&action=plot& var=NOAA05_H0_SEM+Lat&var=NOAA05_H0_SEM+Lon&var=NOAA05_H0_SEM+Alt& var=NOAA06_H0_SEM+Lat&var=NOAA06_H0_SEM+Lon&var=NOAA06_H0_SEM+Alt

21. Lessons we are learning (and concerns) • Our services require underlying standards, although still manual input for each dataset stored in master CDFs • Many firewalls limit ports available and may impose timeout on sessions (had to modify Ethernet stack to send Keep-alive packets) • Web service standards are evolving, subject to interpretation, and implementations may choose which options to support • Different data bind options (RPC-encoded, RPC-literal, document-literal) • Different communication model options (synchronous RPC, asynchronous messaging) • Handling of null Date values (pass by reference [Java] or value [.NET]) • Support for passing multi-dimension arrays (ASP .NET) • Handling of empty arrays • Our Web services were initially implemented before Web Services Interoperability (WS-I) Basic Profile 1.0 and use RPC-encoded binding style. We’ll be changing to document-literal to be compliant with WS-I and .NET. • Need to be flexible and rewrite code but stay stable for clients (how to update API without breaking existing clients: add new methods?, add whole new API?) • Difficult error handling (error codes) between applications • Need careful attention to how existing services are done • Created duplicate functionality that we will have to merge software to remove (Perl interface to call web services)

22. Call to action • Cooperative identification among major services and clients to aid in understanding usage and various reporting questions: Set user-agent string to identify calling routine/institution in all web services calls • Try out our services and give us feedback

23. Joint effort of the NASA GSFCSun-Earth Connections Active Archive (SECAA)in the Space Physics Data Facility (SPDF) and National Space Science Data Center (NSSDC) See <http://spdf.gsfc.nasa.gov/> for links to these tools and services or contact Robert.M.Candey@nasa.gov

24. What are web services? • Services Oriented Architecture (vs. client-server) • Distributed software to software communication, analogous to older technologies such as RPC, DCOM, CORBA, RMI • No HTML or human interaction required • Cross-platform and language-independent • Enables others to develop tools and services leveraging core logic and science data and orbit information • Everyone can use their own clients/tools

25. Lower level components of VOs as a set of interoperable services • Strung together in many combinations • Form an integrated system that is much greater than the sum of its parts • Easily extendable • Open to other systems and external applications by using standard distributed Application Program Interfaces (APIs) • Based on XML and Simple Object Access Protocol [SOAP] standards and/or HTTP calling interface • Tie together existing data holdings • Standardize and simplify their use • Enable much enhanced interoperability and data analysis.

26. Distributed components of a space physics virtual observatory • Work together via standard interfaces and metadata agreements to form a system • Appears as a single super-instrument providing geophysical measurements and models across time and space • Enables researchers to easily and seamlessly analyze data from many more sources than possible before

27. Coordinated Data Analysis Web (CDAWeb) • HTML user interface supports graphics, digital listings, file retrieval and merged/subsetted CDF creation, and simultaneous multi-mission, multi-instrument selection and comparison of science data among a wide range of current and past space missions and ground-based facilities • Now comprises >600 datasets and >1M files of science data • Very popular with 165k user sessions, 94k plots, 622k FTP requests, 67k ASCII listings, 586 CDF create requests, 2882 file retrieval requests in FY2004

28. CDAWeb data available through • FTP file access, including software and documents • Web browser • Web services in addition • Latter two provide data listings, plots, original data files and combined files

29. CDF Markup Language (CDFML) • XML Markup Language describing Common Data Format (CDF) data and metadata • CDFML file (XML file) is an ASCII representation of a CDF file • Used as a mechanism for data interoperability with other science data formats (since translation between data formats via XSL is easy)

30. Simple Object Access Protocol (SOAP) • An XML-based protocol for exchanging information in a distributed environment • Defines a way to perform remote procedure calls (RPCs) - method name and a list of parameters • HTTP is the most popular option for data transport (since it is simple, stable, and widely deployed), but not mandatory • SOAP messages masqueraded as HTTP messages can pass through some firewalls - can be good or bad • Supported in C, C++, Java, C#, Perl, Python, and JavaScript

31. Future and Improved Web Services • Adding more I/O formats to CDAWeb (FITS, HDF, netCDF, IDL savesets) • Smart client issues (WebStart) • Other web services • OMNI_near_earth field and plasma data and indices • ATMOweb ionospheric data • ModelWeb • Orbit_regions and cross-spacecraft queries • COHO_heliospheric data • NMC_metadata (NSSDC Master Catalog) • SPASE_dictionary (Space Physics Archive Search and Exchange) • Community-wide client project? plugins? (NOAA plot tool) • Grid support now that Globus 3 is OGSA/web services compatible

32. What’s important to you? • What services and features do you want? • What other web services are being developed? (VHO, VSO, EGSO, STARS) • ???

33. SECAA Goals • Ease of use • Open source, modular and reusable software • Access to multi-spacecraft and multi-instrument data • Performance (reasonable response times) • Broad base of users • ???

34. Importance of Standards and Agreements • Wide variety of science data file formats (CDF, netCDF, HDF, FITS, etc. and many custom formats) impedes science research across instruments and missions by greatly increasing learning and translating time for each dataset • Custom binary and ASCII formats can be converted into CDF with the makeCDF tool for possible further conversion to other formats • Optional mapping templates handles dataset-specific translations • These tools and service will increase interoperability and improve cross-mission dataset transparency • Cooperative identification among major services and clients to aid in understanding usage and various reporting questions (set user-agent string)

35. Poster specific stuff Fall 2004 AGU Special Session SF05 - Data and Services for Space and Earth Sciences (SF05-9402)

36. Outline • What we’ve done • Why scientists should care • How to use web services (example calls from IDL, Java, Javascript and Perl) • How to setup your own web service • Lessons we are learning (and concerns) • Client development

37. Importance of SECAA Web Services for the Integrated SEC Data Environment

38. Authors Robert M. Candey (Code 632) Reine A. Chimiak (Code 583) David B. Han (Code 586) Bernard T. Harris (Code 583) Rita C. Johnson (QSS, Code 630) Colin A. Klipsch (QSS, Code 630) Tamara J. Kovalick (QSS, Code 630) Howard A. Leckner (QSS, Code 630) Michael H. Liu (Raytheon ITSS, Code 630) Robert E. McGuire (Code 632) NASA Goddard Space Flight Center, Greenbelt MD 20771

39. Abstract The Space Physics Data Facility (SPDF) at NASA GSFC has developed new and state-of-the-art, distributed programming (web services) interfaces to SPDF’s space science mission services and data. This work is an essential step in integrating these powerful and popular SPDF software systems (and the NASA mission data they now serve) with external science data and model software systems under a NASA Virtual Observatory paradigm for enhancing the scientific return of space physics research. The distributed components of a space physics virtual observatory work together via standard interfaces and metadata agreements to form a system that appears as a single super-instrument providing geophysical measurements and models across time and space, enabling researchers to easily and seamlessly analyze data from many more sources than possible before. We are providing a critical set of these lower level components, leveraging our data format expertise and our existing and very popular science and orbit data web-based services, Coordinated Data Analysis Web [CDAWeb] and Satellite Situation Center Web [SSCweb], by adding web services for orbit location, data finding across FTP sites and in CDAWeb, data file format translation, and display. These services tie together existing data holdings, standardize and simplify their use, and enable much enhanced interoperability and data analysis. A Java3D application, TIPSOD (Tool for Interactive Plotting, Sonification and 3-D Or!bit Display), developed in-house, makes use of these web services. CDAWeb, SSCWeb, CDF tools, and web services are a joint effort of the Sun-Earth Connections (SEC) Active Archive (SECAA) in the NASA GSFC Space Physics Data Facility (SPDF) and the National Space Science Data Center (NSSDC); see <http://spdf.gsfc.nasa.gov/>.

40. Outline • What we’ve done (virtual observatories (VOs)) • Why scientists should care • Overview of technologies (web services, public file service, standards such as CDF, translators, 3D clients, etc.) • SECAA services (importance to evolution of integrated SEC data environment for effective multi-mission space physics) • How to use (call from IDL, Java, Javascript and Perl) • How to setup your own web service • Emphasize what’s now working (VSPO, Visbard, ???) • Lessons we are learning • Future • Call to action • Demos (Tipsod and CDAS clients)

41. SECAA SOAP implementations • SOAP uses HTTP calls (port 80) to Apache which sees "/WS/" in the path and calls Tomcat (or Tomcat use own port but lose logging and other features Apache provides). Other URLs are handled by Apache directly • Tomcat <http://jakarta.apache.org/tomcat/> calls calls Java SOAP servlet to parse the incoming message, call the underlying tool, and martial the SOAP response <http://ws.apache.org/soap/docs/install/tomcat.html> • Axis <http://ws.apache.org/axis/> called from Tomcat in place of Sun Jax-RPC