1 / 36

RBSP Science Data Portal

RBSP Science Data Portal. Michele Weiss SWG Data System Technical Lead michele.weiss@jhuapl.edu 240-228-4806. Outline. Virtual Observatories History and Introduction Report on Virtual Observatories SPASE and SPASEQL Introduction and Definition How the Virtual Observatories use SPASE

brice
Download Presentation

RBSP Science Data Portal

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. RBSP Science Data Portal Michele Weiss SWG Data System Technical Lead michele.weiss@jhuapl.edu 240-228-4806 Radiation Belt Storm Probes

  2. Outline • Virtual Observatories • History and Introduction • Report on Virtual Observatories • SPASE and SPASEQL • Introduction and Definition • How the Virtual Observatories use SPASE • How SPASEQL fits in • Requirements on Instrument Teams • Common Data Tools • RBSP Science Data Portal • Autoplot presentation by Jerome Faden • MIDL presentation by Jon Vandegriff RBSP SWG March 4-5, 2009

  3. History of Virtual Observatories (VOs) • In 2007, NASA formalized the Heliophysics Data Environment (HPDE) in the HP Science Data Management Policy. This was endorsed by the Heliophysics Data and Computing Working Group. • http://hpde.gsfc.nasa.gov/index.html • Newest components of the HPDE are discipline specific Virtual Observatories (VxOs) and post-mission Resident Archives • History: • VSO approved by a Senior Review in 2001; Initial studies led to full funding in 2003 (granddaddy of Heliophysics VOs) • 5 VxOs funded in 2006/2007. These are currently in their 3rd year of funding. • Virtual Energetic Particle Observatory (VEPO) - funded 2008, Virtual Wave Observatory (VWO) - funded 2008 and Virtual Model Repository – funded 2009 • In initial phases, but talk to them if you think your data would fit well there. RBSP SWG March 4-5, 2009

  4. Current VOs of Potential Interest to RBSP – RBSP team met w/ each of these VOs • Virtual Heliospheric Observatory (VHO) • PI: Adam Szabo • NASA GSFC • http://vho.nasa.gov/ • Virtual Magnetospheric Observatory (VMO) – GSFC • PI: Jan Merka • NASA GSFC • http://vmo.nasa.gov/ • Virtual Magnetospheric Observatory – UCLA • PI: Ray Walker • Institute of Geophysics and Planetary Physics – UCLA • http://vmo.igpp.ucla.edu/ • Virtual Ionosphere Thermosphere Observatory (VITMO) • PI: Daniel Morrison • JHU/APL • http://vitmo.jhuapl.edu/ • Virtual Radiation Belt Observatory (ViRBO) • PI: Bob Weigel • George Mason University • http://virbo.org RBSP SWG March 4-5, 2009

  5. Virtual Heliospheric Observatory (VHO) & Virtual Magnetospheric Observatory (VMO) – GSFChttp://vmo.nasa.gov/ and http://vho.nasa.gov/ Why use the VHO/VMO? • Latest data available • Descriptions associated with data products • Comprehensive data collection • Data search uses multiple criteria • Date/time • Product name • Measurement type • Parameter values (statistical estimators) • Observatory positions • … more are coming: event lists, regions, ... RBSP SWG March 4-5, 2009

  6. VHO/VMO – Pre-calculated Statistical Estimators • Used in parameter value searches as proxies • A unique feature of VHO/VMO RBSP SWG March 4-5, 2009

  7. VHO/VMO – Data Finder RBSP SWG March 4-5, 2009

  8. Services Support Domain experts (X-men) Generation, validation, referential tools. Search Structured/Form Word-based (google-like) SPASE-QL Download Granules Collections (multi-granule) Rendering Autoplot Data Spacecraft AMPTE_CCE Interball-1 AMPTE_IRM Interball-Tail AMPTE_UKS ISEE-1 DE-1 ISEE-2 DE-2 POLAR GOES10 ST5-094 GOES8 ST5-224 GOES9 THEMIS Geotail Wind IMP8 Groundbased Augsburg UCLA ICESTAR WDC Virtual Magnetospheric Observatory (VMO) – UCLAhttp://vmo.igpp.ucla.edu/ The Virtual Magnetospheric Observatory (VMO) is a distributed data environment with a web -accessible graphical user interface and application programming interface that provides a unified data discovery and retrieval.The VMO offers both services and data. RBSP SWG March 4-5, 2009

  9. VMO Homepage (UCLA & GSFC) –they share the same web entry point Software Search For Data All news Our Vision Helpful Information New Additions Current Events RBSP SWG March 4-5, 2009

  10. Virtual Ionosphere Thermosphere Observatory (VITMO)http://vitmo.jhuapl.edu/ • Interface that is keywords-based using keywords that cover phenomena, fundamental physical properties, and common data measurements. • Enhanced search services – coincidences between satellite and ground and restricted by geophysical conditions. • Single data delivery mechanism for all data, pointers to tools and documentation. • Support for Summary Images and Search for Data by Browsing Images • Data providers web sites integrated into data browse process. 10 RBSP SWG March 4-5, 2009

  11. VITMOIntegrates Data Providers Web Site into the Browse Process RBSP SWG March 4-5, 2009

  12. Virtual Radiation Belt Observatory (ViRBO)http://virbo.org ViRBO Capabilities/Advantages Data • An expanding list of data that we are continually making more accessible (for example, all 2-second NOAA SEM2 data, along with corrected data, including NOAA 15, 16, 17, 18, and MetOp02 are available in merged CDF files. Previously only available in daily binary files.) • Early work with ViRBO will simplify two-way RBSP/ViRBO connections at both the metadata and data level. Metadata • Workflow for contributing or editing existing metadata by entering a web form. See http://virbo.org/meta/ • Experience with SPASE metadata issues involved with particle data RBSP SWG March 4-5, 2009

  13. ViRBO – ViRBO Capabilities/Advantages Server • Using OPeNDAP and a few extensions developed by a former OPeNDAP employee (D. Lindholm at CU/LASP) that will end up in the OPeNDAP distribution. All server features are not yet active. Expected completion by June. Demos available by request. • Turns data browsing inside-out: instead of viewing data day-by-day, perform interactive drill down from full time range. (See Autoplot demo by J. Faden). • Allows range requests and data subsetting (e.g., time>2001-01-01&flux>1e6) with expected response times and non-trivial data volumes. • Data requests will return a short IDL or MATLAB script that hides the data transport from the user. (Standard file output will be supported.) • For performance in plotting and subsetting, server uses caching on data as needed and as allowed. (Traditional long list of files associated with query is still an option.) Visualization • Autoplot design and implementation effort led by ViRBO. RBSP SWG March 4-5, 2009

  14. ViRBO – ViRBO Capabilities/Advantages - continued Other • Host a SVN and GIT repository and have a wiki page that is used for meetings - increasing visibility on Radiation Belt searches. • Thinking a lot about the issue of how to make contributing data simpler. The current model will not scale to 100s of active small satellites. Idea would be to provide a simple IDL/MATLAB script that allows the user to “push” data into our cache or permanent storage. Once data is there, all high-level services are available. Working with Das2/QData and OPeNDAP developers on this. RBSP SWG March 4-5, 2009

  15. What a Virtual Observatory does for you • VOs are continuing to evolve but all have the following common goals • Queryable data catalog of distributed space physics data products within a single user interface • Allow scientists to access data from many missions from a single “data portal” or even from their own s/w applications directly • Simple, uniform access to data from distributed, heterogeneous sources can also enable services such as visualizations, or format translation that enhance the use of these data • May provide tools for cross-mission analysis and visualization • Enhances ability to obtain and use data efficiently across a broad range of observatories, instruments and data formats • Enables integration of complex data systems • Could eliminate need for specialized integration software • Promotes the distribution of your data RBSP SWG March 4-5, 2009

  16. What a Virtual Observatory does NOT do for you • Store your data • Most VOs only store information or metadata about the data product, rather than the data itself. This will include the following types of metadata: • Descriptive metadata typically includes: • Program, satellite and instrument information, as well as data details like observed phenomena, units of measure, time and spatial coverages, provenance, and processing level • Structural or access metadata typically includes: • Details for data access and retrieval • Resolution of data product • Ancillary metadata typically includes: • Services and tools for a data product • Some VOs may store your data when it doesn’t have a home anywhere else • Strong interaction necessary between you and VO(s) to properly describe your data • The better your metadata, the better a VO and a VO user can use your data RBSP SWG March 4-5, 2009

  17. Making your data available to a VO(s) • It is your responsibility to make your data available to the VO(s) • To make your data available to VO(s): • Your data must be already publicly available. The mechanism, as supported by the VO you are delivering your products to, may include: • ftp • http (i.e. a website) • a service • a database • Metadata must exist and be current for your data • Metadata requirements are determined by the VO you are delivering your products to • Most VOs use SPASE to define data products metadata • More on metadata and SPASE in subsequent slides • You can make your data available to multiple VOs but each one may require different metadata standards and protocols RBSP SWG March 4-5, 2009

  18. RBSP Project and Virtual Observatories • RBSP Science Data Management Plan received instrument team buy-in to make their products available to a VO • RBSP Project will not recommend a VO or VOs, however it is hoped that at least one VO (a “universal” VO) will contain all of the data products for the entire RBSP mission • RBSP SWG will be the decider of a “universal” RBSP VO RBSP SWG March 4-5, 2009

  19. Future of VxOs • The 5 VOs identified here will be part of a Senior Review proposal in May, 2009 • Results of proposal will identify the future level of funding for the VOs as well as help determine their long-term future • NASA Heliophysics Science Data Management Policy states: • “The data from missions is to be made available by them both directly and via Virtual Observatories (VOs) that will provide one-stop access to data from many missions along with tools for cross-mission analysis and visualization.  NASA HQ will provide the vision for the DE and ensure, with community input from the HP Data and Computing Working Group (HPDCWG) and Senior Reviews of both missions and Data Centers, that the vision is carried out.  NASA Data Centers will provide various services, including archives and cross-disciplinary access to data and services, and coordination of the VOs and other HPDE components.  All components of the HPDE will involve competitive selections and reviews to ensure the best quality. ” RBSP SWG March 4-5, 2009

  20. Which VO(s) to use? • Each VO is currently discipline specific • Large overlap across the disciplines • Look at the VOs • URLs are provided here as well as PIs and institutions • Look at datasets they are currently providing and plan to provide • Look at ancillary tools and services that may be of utility to you • Talk to the individual VOs • Many of you are already affiliated w/ one or even several • PIs and affiliations are provided here • Currently at differing levels of maturity • This will change significantly by the time RBSP is flying • Note: AGU meetings usually have at least one VO session per meeting in either SA, SH, SM or IN (Earth Space Science Informatics) RBSP SWG March 4-5, 2009

  21. Spase Physics Search and Extract (SPASE)http://www.spase-group.org/ • Defines metadata that describes Space & Solar Physics datasets • SPASE is a consortium consisting of an international team of space and solar physicists and information scientists • Provides a unified method to describe data and other resources • SPASE data model is comparable to the data models developed by PDS and the International Virtual Observatory Alliance (IVOA) for planetary and astronomical data • Major creators of SPASE-based product descriptions are expected to be the VOs, data centers and individual data and model providers • Which could be you if your selected VO does not do it for you • SPASE data model tries to provide enough detail to allow a scientist to understand the content of data products as well as access and contact information • i.e. a set of files for 3 second resolution Geotail magnetic field data for 1992 to 2005 RBSP SWG March 4-5, 2009

  22. SPASE - continued • Most VOs are: • Using SPASE to define data product metadata • VITMO and VSO are current exceptions however their metadata needs are similar to SPASE • Currently defining metadata for data providers. This may change so you may be responsible for your own product metadata in the future. • Typically, only need one metadata file for every different type of data product that you produce and make available through a VO • i.e., if you are producing 5 different data products 15 times a day for the life of the mission, you would only need 5 metadata files and your metadata file could specify the frequency of the data product and the resolution of the product • Some of the VOs may have finer granularity on their metadata definitions of your data products • It is up to you and your selected VO(s) to adequately define your data products to make them useful to both you and your users • Your data products are only as good as their metadata RBSP SWG March 4-5, 2009

  23. SPASE Metadata • The following SPASE metadata examples are one VO’s implementation of SPASE – the VHO • Each VO may have their own implementation of SPASE so may require different fields • SPASE data model is available at http://www.spase-group.org/ RBSP SWG March 4-5, 2009

  24. SPASE Metadata Example – Spacecraft/Observatory • SPASE record describing the ACE spacecraft RBSP SWG March 4-5, 2009

  25. SPASE Metadata Example - Instrument • SPASE record describing the ACE EPAM instrument RBSP SWG March 4-5, 2009

  26. SPASE Metadata Example – Numerical Data • SPASE record describing the ACE EPAM 1-hour Level 2 data product RBSP SWG March 4-5, 2009

  27. SPASE Metadata Example – Physical Parameters • SPASE record describing some of the parameters whose values are part of the ACE EPAM 1-hour Level 2 data product RBSP SWG March 4-5, 2009

  28. SPASE Query Language (SPASEQL) • SPASEQL is based on the SPASE metadata model • SPASEQL will allow the VOs to communicate with each other in a standardized manner • User would go into their favorite VO to initiate a query. SPASEQL would then provide a frontend to each VO that allows them to speak the same language to each other so your VO could then query another VO to get the data you are requesting • As a data provider, you will never need to implement SPASEQL or learn anything about it. It is simply a VO to VO language and will be transparent to you RBSP SWG March 4-5, 2009

  29. Requirements on RBSP Instrument Teams • You should make your public data available to at least one VO or possibly multiple VOs • RBSP SWG should select at least one “universal” VO that all RBSP data gets sent to • You will need to work with your selected VO(s) to ensure that the metadata accurately and adequately defines and provides access to your data products RBSP SWG March 4-5, 2009

  30. Common Tools that might be of Interest • Look directions & calibrations • Physical & empirical fields and particles models to both understand and model the radiation belts • Orientation of spacecraft • Energy flux & pitch angle distribution plots w/ magnetic field data • Alternative sources for time conversions • Examples on the following slides • More on Autoplot and MIDL later • Discussion on tools of interest • Discussion on RBSP Science Data Portal RBSP SWG March 4-5, 2009

  31. Common Tools that might be of Interest Coincidence locator for other relevant missions • Orbit viewer • Orientation of spacecraft • Coincidence locator for other relevant missions • Overlays of relevant models • Planning tools • Satellite locations • Coordinated observations calendar View from north pole along with orbit RBSP SWG March 4-5, 2009

  32. Common Tools of Interest - continued 2-D pitch angle plots Energy flux & pitch angle distribution plots w/ magnetic field data Field model at a given L-Shell Angle to Angle plots RBSP SWG March 4-5, 2009

  33. Common Tools of Interest - continued Magnetic field line trace of a simple dipole Magnetic field line trace of a more realistic magnetic field model RBSP SWG March 4-5, 2009

  34. Common Tools of Interest - continued Solar & Geomagnetic Indices Event Lists Planned and as flown Timelines IMF Data RBSP SWG March 4-5, 2009

  35. Common Tools of Interest - continued Publication List Documentation RBSP SWG March 4-5, 2009

  36. RBSP Science Data Portalhttp://rbsp.jhuapl.edu • Will not duplicate VO functionality • Comprised of things that could be of utility to the RBSP instrument teams such as: • Common tools repository • Shared software repository • Catalog of papers and presentations • Documentation • IMF data • Solar & Geomagnetic indices • Latest Space Weather Forecast • Relevant links • Etc. • Data portal is a team resource governed by RBSP SWG New tab called Data Portal RBSP SWG March 4-5, 2009

More Related