HELIO The Heliophysics Integrated Observatory

1. HELIOTheHeliophysics Integrated Observatory Bob Bentley (UCL-MSSL) Andre Csillaghy (FHNW) Jean Abourdarham (Obs. Paris) 20 November 2008 European Space Weather Week, Brussels

2. Heliophysics • Heliophysics explores the Sun-Solar System Connection • Space weather is a subset of Heliophysics – SWx++ • A virtual observatory that supports Heliophysics must facilitate access to data from a number of communities • Solar, heliospheric, magnetospheric and ionospheric physics, aeronomy… • Throughout the solar system – not just Sun-Earth • Globally – beyond national or European level • Heliophysics sits in the boundary between communities • Astrophysics and Planetary sciences (including Earth sciences) • A VO for heliophysics must be aware of the need to support the interests of these communities • Heliophysics Integrated Observatory (HELIO) • Proposal submitted under EC’s FP7

3. Driven by science • Desire to solve science problems that span disciplinary boundaries is driving the need to provide integrated access to data across the communities • The communities have evolved independently over decades • Each has very different ways of describing, storing and exploiting the data from their observations, varying use of standards • To facilitate access, we need to find ways to: • Tie the data together through searches across all the domains • Present any results in a form that does not require a detailed understanding of each discipline U.Alaska

4. HELIO Concept • HELIO concept implemented with a service oriented architecture • Service to curate and access search metadata • Search engine implemented as a service • Services to access data repositories • Services to extract and process required observations • Workflow tool used to bind the services together • User interface closely liked with this tool • Domain interoperability facilitated by semantic-driven approach • Single data model for all domains not practical • Service oriented architecture allows a researcher to use components of HELIO as they wish – as a chain or as individual services • Chaining of services will provide maximum support for new users and allows a diverse community to study complex science problems in heliophysics • Users may only require access to metadata and employ their own search tools or may just wish to use the dataset location capabilities • Format of data products can be tailored to meet community needs

5. Issues related to Metadata • Metadata are the key to accessing observations • Rapidly increasing volumes make good metadata essential • There are many issues related to existing metadata • Poor quality or missing, lack of interoperability… • This affects ability of all users to do science – not just HELIO • To resolve the issues requires the re-evaluation of capabilities provided within each community and some corrective action • HELIO plans to work closely with the community on this • Metadata can be grouped in several ways, one is: • Search metadata • Metadata used to identify interesting time intervals and locations • Observational metadata • Metadata used to describes the observations, e.g. FITS headers • Storage metadata • Metadata that describes how the data are stored and accessed • Administrative metadata • Metadata that allows the system to exploit the available resources

6. Data Storage • Access to data should be a matter of mechanics and generic • Cheap storage and the Internet have greatly enhanced access • Well established protocols for access – http, ftp, … • How data are stored within a data source can make a lot of difference to their accessibility • A variety of file formats should be accommodated • EGSO has the concept of resource-rich and resource-poor providers • Resource-rich – should provide what is needed in response to simple query • Resource-poor – may only be able to make data accessible over Internet • Guidelines/standards on ways that data should be organized could improve capabilities of all providers • Providing data following simple naming conventions in an ordered directory structure would make them simpler to access • Simple catalogue (textual?) might provide additional information • Discussed at VOiG in 2007; IAU WG discussion

7. Observational metadata • Provides information about how the observation was made • Important for exploiting the data; key for HELIO • Often quality issues related to the metadata that is provided • Parameters sometimes missing, or wrong • Inconsistent use of information, “synonyms” for keywords • In solar data, space-based observations much better described than their ground-based counterparts • Ground-based observations are only source in some wavelengths and need access to as many observatories as possible • Researchers often used to deficiencies in their own domain • Difficult for machines to handle if it is not quantified properly • Consequence can be that the data are “unusable” • Need to encourage all organizations that generate data to adopt and comply with agreed standards • Where possible the standards should have generic parts to facilitate interoperability

8. Searches • In heliophysics, we are interested in how an event on the solar surface can propagate through the heliosphere and affect planetary environments • May also want to work backwards and look for the cause of an effect – what solar event caused this ionospheric activity… • Searches should identify interesting time intervals based on a combination of event, features, etc. metadata • Light curves and images my also be used to augment the search • Location of observer affects whether phenomena seen • Each community of some combination of these metadata • Differences in how some quantities are expressed, what are included • There are concerns about the quality and integrity of these metadata and whether they are adequate to support the searches HELIO would like to undertake

9. Solar search metadata • Searches in solar physics are mainly event driven • Phenomena occur on or near the solar surface • Event data gives time and location of phenomena • Feature data provides details of location and size of structures that may be relevant • Time information can be expressed in many ways • Essentially these are the same, with simple transformations • Spatial information can be expressed in terms of: • Coordinates in the observing frame – e.g. arcsecs from disc centre • Coordinates on the rotating body of the Sun – Carrington coordinates • The location of the observer largely ignored • Helio-seismology is an exception • In the bigger picture of Heliophysics, also need to include the viewing perspective (c.f. STEREO)

10. Other search data • Observation of phenomena in the heliosphere and near/on planets are more complex • For in-situ observations in the heliosphere • Time is when a phenomena affected (passed) the observer • Position of the observer relative to the Sun is key to understanding • When the in-situ observations are made on/near a planet • Position of the observer relative to the planet is also important • Relating events that are defined from in-situ data to those on/near the Sun requires an understanding of how events propagate • Details of the velocity structure of CMEs and the solar wind are not easy to determine… • HELIO plans to develop a tool that will use even/feature data to refine a model to trace effects forwards from causes, etc.

11. Simple, but not so simple • In principle this all seems fairly obvious, but lets look in detail at some common solar event data • On 20 January 2005 there was an X7.1 flare that was intensely geo-effective. • The flare was associated with particle event and a CME; it was also observed by ground-level neutron monitors – a GLE. • Many superlatives were used to describe the event • "The solar energetic particle event of January 20 2005 has been called, by some measures, the most intense in 15 years..." (Mewaldt et al., 2005) • ”The fastest rising SEP event of current cycle [cycle 23]" (Rawat et al., 2006) • ”The most spectacular [solar event] of the Space Age" (Tylka et al., 2006) • ”The largest GLE [GLE 69] in half a century" (Bartol Research Institute) • But event is absent from the NOAA SEC list of "Solar Proton Events Affecting the Earth Environment" • When you look at the data and how lists are created, you realize that the lists are deficient in several ways • Humans and SmFCACs can understand what happened, but • It is harder for machines...

12. X7.1 of 20 Jan 2005 • The event was one of several from AR 10720 • Two other X class flares and several M class flares occurred in previous 3 days; others before this

13. X7.1 of 20 Jan 2005

14. X7.1 of 20 Jan 2005 • At the time of the event, the proton levels had not returned to normal after previous events • The criteria fails to recognize a new event • NOAA lists event on 16 Jan • The X-ray data also suffers from problems • The end of an event is defined by when the counts drop to 50% • New events can “interrupt” existing events • The shape and true duration of the decay phase are lost • NOAA gives start 0636; end 0726 • Not all locations are tagged!! • Significant brightenings seen on images not declared as flares

15. Some of the problems • Automated searches are difficult when major events can be “missed” • A search for long duration events would yield spurious results • If the locations of all flares are not known (in a timely fashion), it is impossible to know whether they will be geo-effective • Instrument flare lists have gaps – nights, off times, etc. – but the reason for a null result is not included

16. Some of the problems • Automated searches are difficult when major events can be “missed” • A search for long duration events would yield spurious results • If the locations of all flares are not known (in a timely fashion), it is impossible to know whether they will be geo-effective • Instrument flare lists have gaps – nights, off times, etc. – but the reason for a null result is not included

17. Improving Metadata • Existing event lists can give a distorted picture what has occurred • Such deficiencies make it difficult for non-experts to use them • The community “knowledge” is not written down • Need to re-evaluate/regenerate the event data in all domains with the idea that they will be used in a joint search across the domains • Ensure events are described more accurately • Include information that might explain null results • Metadata should comply with agreed standards that have generic components to ensure interoperability • The situation has changed with enhancements to technology • Providers need to ensure they are more compliant • Virtual observatories should try to handle problems with old metadata • Limits to what can be achieved if metadata is poorly formed • Standards need to be developed in collaboration with the community and funding agencies

18. Conclusions • We have the technology, but developing a virtual observatory to support heliophysics will not be simple • Cooperation of the community is essential if we are to succeed • Some of the possible problems have been highlighted • The quality of observational metadata needs to be improved • We must improve the quality and content of the search metadata • These problems affect the Space Weather community in a similar way to the Heliophysics community • To address these issues we need to engage the communities in all the domains that constitute heliophysics and develop standards that will facilitate the process • HELIO has a large Networking component that is targeted at addressing these issues and fully involving the community