1 / 31

Finding, browsing, and getting data easily using SPDF web services

Finding, browsing, and getting data easily using SPDF web services. Space Physics Data Facility <http:// spdf.gsfc.nasa.gov > NASA Goddard Space Flight Center Greenbelt MD 20771. CDAWeb data browser

santo
Download Presentation

Finding, browsing, and getting data easily using SPDF web services

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Finding, browsing, and getting data easily using SPDF web services Space Physics Data Facility <http://spdf.gsfc.nasa.gov> NASA Goddard Space Flight Center Greenbelt MD 20771

  2. CDAWeb data browser • Handles vast heterogeneity of instruments and parameters (>1000 datasets, 2M files, 10 TB, 8500 parameters) • Yet simple browsing interface provides 80% of researchers’ needs without many bells & whistles • Based on • standard file format (CDF) • standardized metadata • metadata-driven IDL software • provider metadata over-ride by Master CDFs

  3. CDAWeb usage statistics

  4. CDAWeb process • Top static web page calls Perl routines to generate selection forms, with instrument-types, spacecraft, instruments, datasets and available time ranges filled from text lists (compiled each night from data CDFs, no formal database) • Form POST calls a Perl routine to generate IDL calls to determine data files to fill user’s request, followed by a call to read_myCDF and then either plotmaster, list_mystruct or write_mycdf, depending on the user’s request. • CDAWlibIDL routines read data CDFs into IDL structures, plot or list data, and return URLs to temporary output files (ASCII listings, movie or static GIF files, or created sub- or super-setted CDF files) • CDAS SOAP and REST API calls can also query the text lists and generate the IDL calls

  5. Metadata (ISTP and Cluster) • Global attributes provide mission, instrument type, dataset info (project, source_name, data_type, descriptor, etc.) • Variable attributes (Catdesc, Fieldnam, var_notes) • Depend_0 _1 _2, Delta_plus/minus_var point to related variables • Display_type, Var_type (data, support data) • Fill_val, Validmin/max for ranges • LablAxis/Label_ptr, Units/Unit_ptr for plots • Format for listing • Virtual variables to create variables on-the-fly

  6. CDAWlib IDL library<http://spdf.gsfc.nasa.gov/CDAWlib.html> • Read_MyCDF reads CDF variables into an IDL structure, along with support variables and metadata • PlotMaster determines best display for the variables and creates plots in GIF and PS/PDF • Assorted plot routines: time_series, spectrogram, ionogram, radar_vector, image, orbit, stack_plot, map, plasmagram, movie, map_movie, time_text, etc.

  7. Other useful CDAWlib routines • List_MyStruct creates text listings using all available/defined global and variable level metadata, e.g. label_axis, label_ptr1, format, depends, units, etc. • CDFx <http://cdaweb.gsfc.nasa.gov/cdaweb/cdfx/> IDL GUI to read, list and plot data from CDFs located on user’s machine, allows some customization of plotting options, e.g. color pallets, scale ranges, etc. • Auroral_Image maps 2-D data using map projections and various fill and expansion techniques (generally auroral images onto the Earth) • Spectrogram plots a color spectrogram of a 2-dim variable in contiguous or non-contiguous blocks

  8. Making CDFs • Create skeleton text file (hopefully with standard metadata) by hand or easier with SKTeditor<http://SSCweb.gsfc.nasa.gov/skteditor/> • Convert skeleton file to empty CDF with SkeletonCDF (or automatically in SKTeditor) • Add data with IDL CDAWlib routines or use MakeCDF C-language tool to put ASCII and binary data into a CDF file <http://spdf.gsfc.nasa.gov/makecdf.html> • ISTP/IACG Guidelines for recommended naming of datasets and filenames, global and variable attributes <http://spdf.gsfc.nasa.gov/sp_use_of_cdf.html>

  9. Making CDFs with CDAWlib • Create empty CDF with SKTeditor or manually • Read CDF structure into an IDL structure with read_master_cdf.pro • Fill IDL structure data fields with data • Create new CDF file with the contents of the filled structure with write_data_to_cdf.pro (both in IDLmakecdf.pro)

  10. CDAWeb-resident data direct to IDL (Beta) • <http://cdaweb.gsfc.nasa.gov/WebServices/REST/CdasIdlLibrary.html> • d = spdfgetdata('AC_K2_MFI', ['Magnitude' , 'BGSEc'], ['2009-06-01T00:00:00.000Z', '2009-06-03T00:00:00.000Z']) • IDL GUI interface to CDAWeb-held data: spdfcdawebchooser

  11. CDAWeb Lessons • Standard data file format and metadata enable powerful and extendible services • Code is metadata-driven (from the data files themselves), rather than coding lots of special cases (with attendant high maintenance) • Variable type and dimensions is often enough to determine desirable plot format, but can be over-ridden by metadata • Metadata from Master (no data) CDFs over-rides metadata (or lack of) in data CDFs

  12. SPDF Web Services • Satellite Situation Center (SSC) • SOAP RPC-encoded since 2002 • SOAP document-literal since 2007 • Coordinated Data Analysis System (CDAS) • SOAP RPC-encoded since 2003 • REST since 2010

  13. Web Service Styles • Remote Procedure Call (RPC) • Tightly coupled • Early SOAP, CORBA, DCOM, RMI • Service-Oriented Architecture (SOA) • Loosely coupled • Later (message/document) SOAP • Resource-Oriented Architecture (ROA) • Loosely coupled • Representational State Transfer (REST)

  14. SOAP Web Services • Popular before REST • Supported by most major software vendors • Criticized for being complex (but simple to use with advanced tools/frameworks) • Focuses on “message-oriented” services • Supported in most popular programming environments (Java, .NET, PHP, Python, Perl, etc.)

  15. REST Web Services • Focus on interacting with stateless resources using well-known HTTP standard operations (GET, POST, PUT, DELETE, etc.) • Simpler protocol requires simpler libraries/frameworks • Available in more programming environments • Details less defined than SOAP (e.g., messages can be plain text instead of complex XML conforming to SOAP specifications) • Different services can be very different • “RESTful Web Services”, Leonard Richardson & Sam Ruby, 2007 O'Reilly Media, Inc.

  16. REST vs SOAP • CDAWeb REST sits on top of CDAS SOAP • Trades: ugly URLs but easy to understand, use in browser • SOAP easy to call from Java with SOAP libraries • Opendap, FTP, HTTP, TSDS

  17. More SOAP vs REST • IDL interface much easier with REST rather than convincing scientists to install the Java bridge and our library along with the IDL code • Finally and most important, we can create a system where spacecraft, instruments, datasets, and time periods are treated as objects and can be referenced in the event list server and in papers/reports • SOAP is opaque and more difficult to explain and to use • Can describe a REST service in WSDL, particularly version 2 and also in WADL <http://research.sun.com/spotlight/2006/2006-04-24-TR153.html>, although WSDL alone doesn’t always work for SOAP either.

  18. CDAWeb REST Complications • Metadata is complicated, requiring REST interface to return XML/JSON • Results are complicated, requiring REST interface to return XML/JSON • Resources don’t conform to a simple hierarchical structure, requiring REST to support a POST (with XML request) for some resources in addition to the simpler GET method

  19. Calling CDAWeb SOAP from IDL • Use Java Bridge (Java's SOAP) and CDAWeb-specific JAR (automatically produced from WSDL) • Newer versions of IDL include/configure Java during installation • 1 JAR to add to classpath • IDL call to CDAWeb: • dataviews = cdas->getAllViewDescriptions() • Calling CDAWeb REST from IDL • Use IDLnetURL and IDLffXMLDOM • Requires 1000s of lines* of hand-written, CDAWeb-specific data-binding and serialization/deserialization code (that uses IDLffXMLDOM for marshalling/unmarshalling) • IDL call to CDAWeb: • dataviews = cdas->getDataviews()

  20. CDAS REST IDL Library: > 4518 LOC (including comment and blank lines)

  21. Calling CDAWeb REST from Java • JAXB (Java API for XML Binding) can produce the thousands of lines of code required by IDL (from the CDAS.xsd schema) • Only requires a little more code (to send/receive HTTP) than SOAP. • Conclusion: • Calling REST is only a little more work than SOAP if the client environment has something like JAXB (which IDL doesn’t) or if someone else writes the extra code.

  22. SOAP Lesson 1 Applied To REST • Cannot reduce metadata to simple array of string values • That is, metadata is more complicated (structured) and simplification makes it less useful • This resulted in the REST interface returning XML/JSON instead of something simpler

  23. SOAP Lesson 2 Applied To REST • CDAWeb results (plot, listing, CDF) are too complicated to be returned in a HTTP entity body • A single request can produce multiple image files • A result may have many message, status, warning, and error entities associated with it • Results may be thumbnail images that require special processing to obtain expanded images • REST interface returns XML/JSON that contains URL of actual result (plot, listing, CDF)

  24. CDAWeb Resource URI Design • Attempted to design URI that was meaningful, well structured, and used path variables when possible • /dataviews/{dataview}/datasets/{dataset}/data/{start-time},{stop-time}/{var1},{varn}?format=... • Cannot represent multi-dataset (with dataset associated variables) resources with a single path • Multi-dataset resources must be requested with a POST containing an XML description of request

  25. Backups

  26. Abstract The NASA GSFC Space Physics Data Facility (SPDF) provides heliophysics science-enabling information services for enhancing scientific research and enabling integration of these services into the Heliophysics Data Environment paradigm, via SOAP and REST web services in addition to web browser, FTP, and OPeNDAP interfaces. We describe these interfaces and the philosophies behind these web services, and show how to call them from various languages, such as IDL and Perl. We are working towards a "one simple line to call" philosophy extolled in the recent VxO discussions. Combining data from many instruments and missions enables broad research analysis and correlation and coordination with other experiments and missions.

  27. Coordinated Data Analysis Web (CDAWeb) <http://cdaweb.gsfc.nasa.gov> • Data browsing system provides plotting, listing and open access via FTP, HTTP, and web services (REST, SOAP, OPeNDAP) for data from most NASA Heliophysics missions • Combining data from many instruments and missions enables broad research analysis and correlation and coordination with other experiments and missions • Collecting and making data usable biggest effort but most important • Space weather, planetary studies, in situ/remote data

  28. Common Data Format (CDF) <http://cdf.gsfc.nasa.gov> • Standard self-describing multidimensional data format • Platform- and discipline-independent • Associated scientific data management package (“CDF Library”) makes actual data format completely transparent to the user and accessible through a consistent set of interface routines • IDL and Matlab routines • Also callable by Fortran, C, C#, Perl, Java • Open source • Internal compression, checksums

  29. Format Translator • CDF project also maintains software and services for translating between many standard formats (CDF, netCDF, HDF, FITS, XML) <http://cdf.gsfc.nasa.gov/html/dtws.html>

  30. IDL CDF Interface Creating CDFs: CDF_CREATE, CDF_VARCREATE, CDF_ATTPUT, CDF_VARPUT, CDF_CLOSE Reading CDFs: CDF_OPEN, CDF_INQUIRE, CDF_CONTROL, CDF_VARINQ, CDF_VARGET, CDF_ATTINQ, CDF_ATTGET, CDF_CLOSE Info: CDF_CONTROL, CDF_COMPRESSION, CDF_DOC, CDF_INQUIRE, CDF_LIB_INFO Time: CDF_ENCODE_EPOCH, CDF_EPOCH, CDF_PARSE_EPOCH, CDF_EPOCH_DIFF, CDF_EPOCH_COMPARE Other: CDF_SET_MD5CHECKSUM, CDF_SET_CDF27_BACKWARD_COMPATIBLE

More Related