1 / 45

China-VO Data Access Service Based on OGSA

Chinese Virtual Observatory. China-VO Data Access Service Based on OGSA. Jian Sang National Astronomical Observatory of China. Outline. VO,Grid and OGSA Build the catalog data service Build the image mosaic service Faced technical difficulties. The total area of astro telescopes in m**2.

saxton
Download Presentation

China-VO Data Access Service Based on OGSA

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Chinese Virtual Observatory China-VO Data Access Service Based on OGSA Jian Sang National Astronomical Observatory of China IVOA Small Projects Meeting 2003

  2. Outline • VO,Grid and OGSA • Build the catalog data service • Build the image mosaic service • Faced technical difficulties IVOA Small Projects Meeting 2003

  3. The total area of astro telescopes in m**2 The total Gigapixels of CCDs. The number of pixels and the data double every year! The Increase Of Astronomical Data IVOA Small Projects Meeting 2003

  4. Challenges • The quantity of data nearly amounts to PB. • The data is distributed and stored in heterogeneous DBMSs in heterogeneous host environments. IVOA Small Projects Meeting 2003

  5. The VO’s Goal • The VO’s initial goal is to federate existing astronomical data archives and provide standard services for manipulating these data. HOW TO REACH THIS GOAL? The Grid technology can solve the problem! IVOA Small Projects Meeting 2003

  6. What is Grid • Grid technology has been driven by genesis from metacomputing, but… • In practice, the Grid is about resource sharing and coordinated problem solving in dynamic, multi-institutional virtual organizations • Focus on how to enable, maintain and control the sharing of resources to achieve a common goal IVOA Small Projects Meeting 2003

  7. What “Grid“ offers: • Resource management protocols and services that support secure remote access to shared data resources and computing and the co-allocation of multiple resources. • Security solutions that support management of credentials and policies. • Information query protocols and services that provide configuration and status information about resources,organizations and services. • Data Management services that locate and transport datasets between storage systems and applications. IVOA Small Projects Meeting 2003

  8. What is OGSA • The Open Grid Services Architecture (OGSA) represents an evolution towards a Grid system architecture based on Web services concepts and technologies.  • The OGSA integrates key Grid technologies (including the Globus Toolkit with Web services mechanisms to create a distributed system framework based around the Open Grid Services Infrastructure (OGSI). In Grids ,Everything is Service IVOA Small Projects Meeting 2003

  9. The Open Grid Services Architecture • Service orientation to virtualize resources • From Web services:( everything is service) -Standard interface definition mechanisms:multiple protocol bindings,multiple implementations,local/remote transparency • Building on Globus Toolkit: -Grid service :semantics for service interactions -Management of transient instances -Factory,Registry,Discovery,other services -Reliable and secure transport • Multiple host environments:J2EE,.NET,C,… IVOA Small Projects Meeting 2003

  10. The Structure of Grid Service IVOA Small Projects Meeting 2003

  11. Grid service interfaces IVOA Small Projects Meeting 2003

  12. Construct The Astronomical Data Grid The astronomical data service is the most fundamental and important component in Virtual Observatory. In the aspect of data share, the VO can be think as a astronomical Data Grid VO=Astronomical Data Grid IVOA Small Projects Meeting 2003

  13. Outline • VO,Grid and OGSA • Build the catalog data access service • Build the image mosaic service • Faced difficulties IVOA Small Projects Meeting 2003

  14. The Classification of Astronomical Data Service • Astronomical Catalog Service • Image Mosaic Service • Spectrum Data Service • Simulation Data Service IVOA Small Projects Meeting 2003

  15. Existing Astronomical Datasets we have IVOA Small Projects Meeting 2003

  16. Build Catalog Data Service How to federate the catalog data into VO,that is, how to build Data Service using the existing databases and programs? IVOA Small Projects Meeting 2003

  17. Define Catalog Service Interface Some standards we used: • Input Query Language : SQL(now),ADQL (plan) • Output Data Format: VOTable 1.0 • Catalog resource metadata registry protocol: VOResource 0.9 input: ADQL query sentence output: VOTable format result it makes service interface/API simple. IVOA Small Projects Meeting 2003

  18. How to use existing databases and programs to create catalog data service How to create a catalog data service that can understand ADQL and generate VOTable format result?? we adopt two ways! • Reconstruct the existing catalog DBMS • Encapsulate search program,like pmm The CDS has offered search program for big catalog like USNO A2,0….. IVOA Small Projects Meeting 2003

  19. Catalog data service based on DB GT3 Interface VOTable ADQL ADQL/SQL Translator VOTable Wrapper SQL ResultSet JDBC Catalog/metadata DBMS IVOA Small Projects Meeting 2003

  20. Advantage and disadvantage • Can sufficiently use the functions of SQL language and implement complex query. • DBMSs offer the most powerful functions for data management and maintenance. • Need many works to reconstruct the DBs. • To big catalogs, like USNOB1.0,2MASS PSC, query efficiency is low IVOA Small Projects Meeting 2003

  21. Data service based on search program GT3 Interface ADQL VOTable ADQL Translator VOTable Wrapper parameters stream JNI/ program Data Files IVOA Small Projects Meeting 2003

  22. Advantage and disadvantage • Positional search is quicker than DB • Only offer search functions that programs could offer. Many programs only offer position search functions,no statistical functions. IVOA Small Projects Meeting 2003

  23. Catalog Access Service Provided by us IVOA Small Projects Meeting 2003

  24. How to call a Catalog data service Resource Registry 1.<Find Factory> <registry> 2.<Factory GSH> 3.<create data service> Data Service Factory Grid Client 4.<Data service GSH> Create Data service 5.<data request(ADQL)> Data Service Instance 6.<result (VOTable)> Database IVOA Small Projects Meeting 2003

  25. End Users Data service Data service Data service Use Data Service to build www service for end user Web Client End user don’t know where the data services are http Data Mining Service Data Visualization Service Data processing Service Web server Grid Client Resources Register Services Register MySQL Oracle 9i Files IVOA Small Projects Meeting 2003

  26. Use data service to create other service Our next work is to build a multi-wavelength cross-identification service (MWCI)based on the catalog data service. What is multi-wavelength cross-identification ? To cross-identify datasets by positional consistency, we can understand objects from different wavelength properties. IVOA Small Projects Meeting 2003

  27. The steps of multi-wavelengthcross-identification • Cross-identify datasets from different wavelengths within error radius. • Divide the result of cross-identification into three situations: one-to-one, one-to-two, one-to-many. • Choose the one-to-one entry for data mining • The other two situations need statistical analysis to determine which source are the true counterpoint. IVOA Small Projects Meeting 2003

  28. Requirements • Locate the datasets that users want to use. (dataset discovery) • How to cross-match the datasets in heterogeneous DBMSs at different locations effectively and efficiently. • Find storage resource to store the results IVOA Small Projects Meeting 2003

  29. Registry MWCI Factory Data Service 2 4 2MASS 1 MWCI 5 MWCI Service Provider User Application . . . 3 6 . . . 5 Data Service storage Factory 7 6 NVSS 4 storage Storage Service Provider IVOA Small Projects Meeting 2003

  30. Outline • VO,Grid and OGSA • Build the catalog data access service • Build the image mosaic service • Faced technical difficulties IVOA Small Projects Meeting 2003

  31. Build The Image Mosaic Service • Use DSS-I sky image build our first image mosaic service. IVOA Small Projects Meeting 2003

  32. the definition of interface of service • Input parameters: 1.RA,2.Dec,3.image height,4.image width • transport protocols :gridFTP • Output Data format :fits IVOA Small Projects Meeting 2003

  33. Realization of DSS-I imagemosaic service GridFTP GT3 Interface JNI/ Fits file parameters GetImage DSS-I Image Files IVOA Small Projects Meeting 2003

  34. Outline • VO,Grid and OGSA • Build the catalog data access service • Build the image mosaic service • Faced technical difficulties IVOA Small Projects Meeting 2003

  35. Technical Difficulties • service/resource registry and discovery! • ADQL2SQL translator • protocol shortcoming IVOA Small Projects Meeting 2003

  36. protocol shortcomings • The shortcomings of VOTable 1.0 protocol • 1.How to encapsulate result of join query!! • 2.The standard to encapsulating spectrum data • 3.the definition of FIELD element is not strict and uncompleted • The shortcoming of UCD • 1.Can’t express concrete meaning,such as “ERROR” ,Error for what?? • 2. incomplete, example:HTMID has no UCD • Lack of standard for Unit IVOA Small Projects Meeting 2003

  37. Thank You Q & A ? www. .org IVOA Small Projects Meeting 2003

  38. Our provided catalogs in Catalog Service IVOA Small Projects Meeting 2003

  39. The Step Of Calling A Data Service IVOA Small Projects Meeting 2003

  40. Transparencies for Astro Data Access • Heterogeneity Transparency • Name Transparency • Distribution Transparency IVOA Small Projects Meeting 2003

  41. What is Grid Service? IVOA Small Projects Meeting 2003

  42. What Is The Data Grid • DataGrid :A dynamic logical namespace that enables coordinated sharing of heterogeneous distributed storage resources and digital entities based on local and global policies across administrative domains in a virtual enterprise. • DataGrid • Logical name space for location independent identifiers • Abstractionsfor storage repositories, information repositories, and access APIs • Latency management IVOA Small Projects Meeting 2003

  43. Data delivered Ask for data • The data is found and returned • Where & how details are managed by data grid Using a Data Grid –in Abstract Data Grid • User asks for data from the data grid IVOA Small Projects Meeting 2003

  44. IVOA Small Projects Meeting 2003

More Related