1 / 22

Summary of distributed tools of potential use for JRA3

Summary of distributed tools of potential use for JRA3. Dugan Witherick HPC Programmer for the Miracle Consortium University College London JRA3 Workshop @ UCL: 22-24 April 2010. Talk Outline. Summarise the Goals of JRA3 Tasks 3 and 4

cais
Download Presentation

Summary of distributed tools of potential use for JRA3

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Summary of distributed tools of potential use for JRA3 Dugan Witherick HPC Programmer for the Miracle Consortium University College London JRA3 Workshop @ UCL: 22-24 April 2010

  2. Talk Outline JRA3 Workshop @ UCL: 22-24/04/2010

  3. Summarise the Goals of JRA3 Tasks 3 and 4 • Task 3: Distributed Numerical Modelling Laboratory (DNML) • Task 4: Distributed Data Analysis Laboratory (DDAL) • What is IDIS? • What are other people doing (two examples)? • A possible solution from the Virtual Observatory • Who’s using Virtual Observatory Technology • The Universal Worker Service JRA3 Workshop @ UCL: 22-24/04/2010

  4. Summary of Tasks 3 and 4 Goals (a reminder) JRA3 Workshop @ UCL: 22-24/04/2010

  5. Task 3: • The main goal of this task is to bring together the numerical simulation efforts currently underway in Europe and to create a coordinated European multi-disciplinary modelling facility for Planetary Science: a Distributed Numerical Modelling Laboratory (DNML). • ….feeding into the catalogue part of SA IDIS. • Task 4: • The main goal of this task is to bring together expert teams in data analysis with their present research infrastructures and create a coordinated European interdisciplinary data analysis facility for Planetary Science: a Distributed Data Analysis Laboratory (DDAL). • ….feeding into the catalogue part of SA IDIS. • The main technical requirements for these tasks could be satisfied with the same solution JRA3 Workshop @ UCL: 22-24/04/2010

  6. Integrated and Distributed Information Service(IDIS) JRA3 Workshop @ UCL: 22-24/04/2010

  7. What is the purpose of IDIS? • The task of IDIS is to provide an easy-to-use web-based platform to give access to available data, initiate new research activities needed to interpret observations or to solve open questions, locate teams and laboratories with special knowledge needed, and exploit synergies between space-based missions and capabilities of ground-based observatories. • A set of tools for describing, accessing and combining information and data from different sources will be developed, with the objective to offer a Virtual Observatory like access to many data essential for planetary research from European and non-European sources. • Any solution we come up with must fit in with the existing plans of IDIS JRA3 Workshop @ UCL: 22-24/04/2010

  8. What are other people doing?(two examples) JRA3 Workshop @ UCL: 22-24/04/2010

  9. Automated Multi Dataset Analysis (AMDA) • A generic Webtool for Space Physics data • automated event search and characterisation • catalogue generation and exploitation • automated database conditional extraction • access to remote Data Centers • Web portal located at http://cdpp-amda.cesr.fr/ • The service can be reached and used from other sites according to the Virtual Observatory paradigm. • Not so much a distributed data analysis tool but a single tool linked into the Virtual Observatory JRA3 Workshop @ UCL: 22-24/04/2010

  10. JRA3 Workshop @ UCL: 22-24/04/2010

  11. Community Coordinated Modelling Centre CCMC • http://ccmc.gsfc.nasa.gov/ • Focus on Space Weather • Models are run through an automated request system: • Instant Run – results are provided immediately or • Run on Request – user is informed by email when run has completed • These longer running models are added to a queue to run on their Beowulf Cluster • Capable of running more than one model/analysis tool • Not really distributed • Appears that all work is being completed at one site JRA3 Workshop @ UCL: 22-24/04/2010

  12. JRA3 Workshop @ UCL: 22-24/04/2010

  13. JRA3 Workshop @ UCL: 22-24/04/2010

  14. Remember IDIS (Distributed is a keyword) JRA3 Workshop @ UCL: 22-24/04/2010

  15. JRA3 Workshop @ UCL: 22-24/04/2010

  16. A Possible Solution from the Virtual Observatory JRA3 Workshop @ UCL: 22-24/04/2010

  17. Common Execution Controller (CEC) • Web service for running various asynchronous applications. • Part of the Common Execution Architecture (CEA) suite of software from Astrogrid. • Designed to wrap existing applications so that they are presented as IVOA standard web services. • Commandline applications • Existing web form based HTTP applications • Custom written Java applications • Suitable for applications that: • have no graphical interface • need no user interaction once launch • Supports the following standards: UWS, VOSI, General Security, Registry, Delegation, VOSpace JRA3 Workshop @ UCL: 22-24/04/2010

  18. The Universal Worker Service (UWS) • An International Virtual Observatory Specification • http://www.ivoa.net/Documents/UWS/ • The Universal Worker Service (UWS) pattern defines how to manage asynchronous execution of jobs on a service • Has features over the previous standards of • Control for job lifetime • Access to user sandbox (in order to inspect intermediate files • There are UWS deployments that do not use the CEA • Not necessary to use CEA JRA3 Workshop @ UCL: 22-24/04/2010

  19. Virtual Observatory Registry • Essentially the “Yellow Pages” or Telephone Directory for services • User client (web portal or client tool) is used to search the registry • Information about the resource is returned • Brief summary • Contact Name • Inputs for the model/tool • Location for the model/tool • Two types of registry • Local registry which lists the services installed locally • Full registry harvests the local registries to find out what they have installed • Users search the full registry JRA3 Workshop @ UCL: 22-24/04/2010

  20. JRA3 Workshop @ UCL: 22-24/04/2010

  21. JRA3 Workshop @ UCL: 22-24/04/2010

  22. Advantages to this plan • Allows us to take advantage of the extensive effort made in developing standards • Saves us a lot of effort • We can use/modify existing VO compliant tools • Tool for OPerations on Catalogues And Tables • AladinVO – VO compliant image analysis tool • VOSpec – spectral analysis tool • Possible access to workflow tools • Provides us with VO access to other data/tools • Some IDIS data sources are already published to the VO • Disadvantages to this plan • Unclear whether this would be supported by IDIS • The VO has had its own problems which we may end up getting caught in • This may not be appropriate for fulfilling our deliverables JRA3 Workshop @ UCL: 22-24/04/2010

More Related