1 / 29

Joint Technical Meeting between EGSO, VSO and CoSEC

Joint Technical Meeting between EGSO, VSO and CoSEC. San Francisco 7-8 December 2003. Overview. Of necessity the solar community needs to move towards a virtual environment to access solar and related data

virgo
Download Presentation

Joint Technical Meeting between EGSO, VSO and CoSEC

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Joint Technical MeetingbetweenEGSO, VSO and CoSEC San Francisco 7-8 December 2003

  2. Overview • Of necessity the solar community needs to move towards avirtual environment to access solar and related data • The required environment is rich and it is not clear any of the VO initiatives can fully realize it by themselves • Means we should try to work together to produce a bigger and better whole • Consider how to share resources, including special services • Agree on ways of interfacing to resources, etc. • Etc… • Tools to allow us to build capabilities, so we need to think generically … • Desire to ensure that what VSInc does in interoperable with the heliospheric, etc. communities – e.g. for ILWS

  3. Areas of cooperation in VSInc. • Descriptions of resources understood by all projects • Particularly data providers • Understood protocols/formats for passing information between systems and resources • e.g. VOTable (XML) used to pass results from SEC Server • Consider ways of sharing general and special services • Where possible make them complimentary • Where possible, provide VO (web service?) interfaces into other (existing) special resources • e.g. QLK data, Max Millenium etc. Catalogues, etc. • “Shared” interface between projects into providers • Not scope for three interfaces into each provider • Consider if there are other areas we can cooperate on: • Query languages, workflow, etc…

  4. Data Models, metadata, etc. • If we want to be able to share resources, we need to describe them in a way we all understand • To do this, we need to develop interoperable Data Models • EGSO has released a “Unified Model for Solar Data” (here) • Deliverable to the European Commission (EGSO-WP4-D1) – available on the EGSO Web site (www.egso.org) • Describes both solar and heliospheric data • Recently, we have been taking a more detailed look at how various components of EGSO are related – data, metadata (catalogues), registries, etc – this is shown (here)

  5. Special Servers (I) • Context Rendering of ancillary data • Preparation of “images” used in consumer for refined data selection (spatial and temporal) • Removes(?) need for SSW installation on GUI • How do we handle ZOOMing?? • Over-plotting of location of events on summary images, synoptic maps, etc. (results from an SEC search) • Over-plotting of feature details from the SFC on a reference image • Preparation of special (derived) products • GOES temperature plots, etc. • Synthetic spectra (e.g. using CHIANTI) • SolarSoft Servers located at (meta)data warehouses • Specialist knowledge on access & processing of data • Essential for some types of legacy data • Pipes to other sites…

  6. Special Servers (II) • Processing of data “at source” • SolarSoft Server(s) able to support pipeline processing of requested data • Extraction and calibration of subset • Preferably local to the data, but this is not essential • Reduces dependency on user capabilities • User receives calibrated products and does not need to know how to process data and does not have to have all supporting data installed • Essential as more datasets are added • Work by CoSEC very interesting as way of providing these capabilities

  7. Meeting Plan • Sunday • Concentrate mainly on interaction between the EGSO and CoSEC projects • Look at areas where can share resources • Presentations, discussions & splinters • Monday • More general view for all solar and related VO projects • Look at all areas where we could cooperate • Presentations and discussions • Outcome: • Report on how we will work together

  8. Slides about EGSO

  9. The European Gridof Solar Observations Bob Bentley (UCL) & the EGSO Team 6-7 December 2003 VSInc Joint Technical Meeting

  10. EGSO – European Grid of Solar Observations • EGSO is a Grid test-bed related to a particular application • Designed to improve access to solar data for the solar physics and other communities • Addresses the generic problem of a distributed heterogeneous data set and a scattered user community • Funded under the Information Society Technologies (IST) thematic priority of the EC’s Fifth Framework Program (FP5) • Started March 2002; duration of 36 months • Involves 11 groups in Europe and the US, led by UCL-MSSL • 4 in UK, 2 in France, 2 in Italy, 1 in Switzerland, 2 in US • Several associate partners, mainly in the US • EGSO is interacting with many other projects • US-VSO CoSEC and EGSO working closely together • Also working with the ILWS/SDO Project • Collaborated with ESA’s study project SpaceGRID • Involved with other EC funded Grid projects through GRIDSTART

  11. Objectives of EGSO • Support user community scattered around the world • Current and future projects are international collaborations • EGSO funded by EC, but has US partners • Build enhanced search capability for solar data • Analysis of solar data is event driven • Search capability linked to this not currently available • Increasing data volumes, etc. require new methodology • Provide access to solar data centres and observatories around the world • Data available in Europe (or US) not enough for many studies • Provide ability to process data at source • Both pipeline and more complex processing • Includes ability to upload code to some providers Need for these capabilities not unique to solar physics…

  12. The extended EGSO family Partners provide expertise in solar physics and IT • UK • UCL-MSSL, UCL-CS, RAL, Univ. Bradford, Astrium • France • IAS (Orsay), Observatoire de Paris-Meudon • Italy • Istituto Nazionale di Astrofisico, Politecnico di Torino • INAf includes observatories of Turin, Florence, Naples and Trieste • Switzerland • Univ. Applied Sciences (Aargau) • Netherlands • ESA – Solar Group • US • SDAC (NASA-GSFC), National Solar Observatory (VSO) • Stanford University, Montana State University (VSO) • Lockheed-Martin (CoSEC)

  13. Handling the data EGSO will dramatically enhance access to solar data • Data could be located anywhere in the world • User only needs to know observations exist, not where located • System able to optimize use of sources (closest, least used, etc) and handle of replicated data and aggregated sources • Burden on provider minimized to encourage participation • As far as possible, process the data “at source” • Involves extraction and calibration of a subset of the raw data • Software for processing defined by instrument team (IDL, C…) • Processing reduces volumes of data moved around • Simplifies requirements on user’s own system • Standard (pipe-line) processing adequate for many users • More complex problems require ability to upload code • Used in analysis of extended data sets (helioseismology, etc) • System allocates resources; Security an issue • Models and simulations have similar requirements

  14. The EGSO Search Engine In order to provide an enhanced search capability, EGSO will improve the quality and availability of metadata • Enhanced cataloguing describes the data more fully • Standardized metadata versions of observing catalogues tie together the heterogeneous data sets • New types of catalogue allow searches on events, features and phenomena rather than just date & time, pointing, etc… • Ancillary data used to provide additional search criteria • Images, time series, derived products, etc. • Search Registry describes all metadata available for search • It will be possible to access to EGSO through: • A flexible Graphic User Interface (GUI ) – normal route • An Application Program Interface (API) – this provides access for users from other applications, communities or Grids

  15. The enhanced solar catalogues • Unified Observing Catalogues (UOC) • Metadata form of observing catalogues used to tie together the heterogeneous data, leaving the data unchanged • Self describing (e.g. XML), quantised by time and instrument, with no dependencies on ancillary data or proprietary software and any errors corrected • Standards defined for future data sets (e.g. STEREO, ILWS, Solar-B) • Solar Event Catalogues (SEC) • Built from information contained in published lists • Flare lists, CME lists, lists in SGD, etc. • Solar Feature Catalogue (SFC) • Lists of the occurrence of events, phenomena and features provides an alternate means of selecting data • Derived using image recognition software developed in WP5 Similar hierarchical cataloguing required in other data Grid projects

  16. Ancillary Data This is “catch-all” term for all non-catalogue items • Items used to set the context for a search • Images, time series, derived parameters, etc. • Processed products from data-intensive instruments • STP, etc. data could be incorporated in this way • Some data items will have to be derived on-the-fly • Not possible for everything to have been prepared already • Servers will provide this type of complex data products • Products include derived parameters, specialized actions, etc. e.g. GOES temperature, matched areas of images, etc. • SolarSoft packages, e.g. Chianti, could be brought up like this

  17. Ancillary Data This is “catch-all” term for all non-catalogue items • Items used to set the context for a search • Images, time series, derived parameters, etc. • Processed products from data-intensive instruments • STP, etc. data could be incorporated in this way • Some data items will have to be derived on-the-fly • Not possible for everything to have been prepared already • Servers will provide this type of complex data products • Products include derived parameters, specialized actions, etc. e.g. GOES temperature, matched areas of images, etc. • SolarSoft packages, e.g. Chianti, could be brought up like this Objective of the improved metadata, etc. is to pose questions like: Identify events when a filament eruption occurred within 30° of the north-west limb and there were good observations in H, EUV and soft X-rays

  18. Solar Event Catalogue (SEC) Server • Server specializing in event catalogues • Complexity of updating information hidden from the system • RDBMS that can accept SQL queries • Interfaced into EGSO as a Web Service • Test interface available through URL: http://radiosun.ts.astro.it/sec/sec_ui.php • Currently being tested with: • Flare lists from NOAA and REHESSI • NOAA Proton Event list • CME lists from SOHO/LASCO • In process of being added: • NOAA Active Region (NAR) database • Various indices –Sunspot Number (SSN), 10.7cm flux, Kp • Interested in things that could be added…

  19. Solar Feature Catalogue (SFC) • EGSO has a work package (WP5) dedicated to developing tools needed to detect common solar features and then using them to derive the Solar Feature Catalogue • Code to prepare images for recognition already available • Removal of artifacts, regularize shape, etc. • Work in progress on codes to detect the features • Codes for sunspots, active regions and filaments developed and under test • Codes to recognize coronal holes and magnetic neutral lines under investigation • Document discussing techniques available shortly • Trying to define standard way of describing features for the feature catalogue • Preliminary version of SFC has been prepared • Now starting experimenting with the results to determine if objectives can be realized with the stored information

  20. Use of the Solar Feature Catalog • The SFC can be used in at least three ways: • Outline features recognized in one wavelength on an image taken in another (at a different time) • Determine when events related to features have occurred – e.g. filament eruptions, flux emergence • Track relative motion of features – e.g. sunspots • The SFC will be deployed as a Server addressed through Web Services • Not clear whether the SFC Server will be combined with the already deployed SEC Server • Server will be accessible to other VO projects

  21. Current Status of EGSO • Extensive survey of requirements in 2002 • Working architecture defined and detailed during the first half of 2003 • Release 1 of EGSO was demonstrated at IST2003 in Milan (October 2-4) • Demonstration of how the three roles work together • Working prototype of the Solar Event Catalog Server • Release 2 of EGSO due at the end of November • Development of interface to Data Providers • More complex query supported through SEC Server • Greater GUI capabilities • New version of EGSO Data Model document was released recently • Describes both solar and heliospheric data

  22. Slides after this are reserves

  23. Access to Resources EGSO is a Grid and activities depend on access to resources • Resources described by entries in aResource Registry and managed by a Broker. Types include: • Metadata – from prime data providers • Data – from data centres, observatories, etc • Processing – simple, multi-instance processors, HPC(?) • Storage – cache space, on-line mass storage, etc. • Services – support of complex (meta)data products Note: Some providers can support multiple capabilities • The Broker allocates resources and controls: • How much being requested of a particular provider • Processing of data & staging of results • Processing may be at different site to data provider • Broker & Registries replicated to provide system resilience and permit load sharing

  24. EGSO Services & Products EGSO will establish a number of services which can also be used by other communities and Grids • API access to the EGSO Search Engine • Supports complex queries of all metadata as a service • Solar Event & Feature Catalogue Servers • Sites that specialize in providing access to derived solar event and feature data • Servers of complex data products • Servers able to process requested items on the fly • Products include derived parameters, specialized actions, etc. e.g. GOES temperature, matched areas of images, etc. • SolarSoft packages, e.g. Chianti, could be brought up like this • A lot of scope for processing a number of things in this way • Extracted & processed data products • Requested data can be provided in a number of formats • Single file type/format will not satisfy all requirements

  25. ARCHIVES EGSO GRID Cat. Provider GUI Consumer Broker Provider GUI Consumer Broker Provider GUI Simplified Architecture • Architecture defined in terms of roles • Consumer, Broker and Provider After R. Linsolas, IAS

  26. Architecture – Viewed as 3 Tiers

  27. Possible requirements for Data Providers Need to register each dataset. Required info. could include: • Catalogue Information: All observations “should be summarised in observing catalogues” (prime source only) • Data Map: This defines in broad outline what time intervals of the data are actually held. It should be considered dynamic. • Type of storage: On-line, near on-line or off-line. • Means to retrieve data: The exact meaning of this information depends on whether the provider is active or passive. • Active source: address where process data can be retrieved from and the means of retrieval (ftp, http, etc.). • Passive source: map of the physical location of data within the provider system and the means of retrieval. • Resource limits:The resource usagebeyond this a provider switches from active to passive mode needs to be defined. • Details of access restrictions: If any part of a data set is proprietary (for some period) or otherwise restricted. • Frequency of updates: So that system does not have to constantly monitor all data sources

  28. Definition of Requirements • Solicitation of Requirements • User Survey - in collaboration with SpaceGRID, in March 2002, with over 100 responses • Use Cases - covering multiple usage modes and scientific goals • User Consultation - discussions at meetings and other discussions • Brainstorming - Systems Concepts Document, cross-fertilization from other GRID and Virtual Observatory projects • Total of 220 requirements defined • Requirements refined to address system capabilities, not implementation • Specified in formal manner applicable in system design, but annotated to allow for meaning to be readily understood by stakeholders • Currently soliciting feedback and finalizing priority of each requirement

  29. Prototypes of some Services • Solar Event Catalogue (SEC) Server • Server specializing in event catalogues • EGSO interested in things that could be added • Interface into EGSO as a Web Service • Currently being tested with: • Flare lists from NOAA and REHESSI • NOAA Proton Event list • CME lists from SOHO/LASCO • Available through URL: http://radiosun.ts.astro.it/sec/sec_ui.php • Currently being added: • NOAA Active Region (NAR) database • Various indices – Sunspot Number (SSN), 10.7cm flux, Kp • Database for Solar Observatories (DSO) • Registry needed within EGSO to provide more detailed information on (all) possible data sources • Currently being populated and will shortly be accessible through the Web

More Related