1 / 39

Virtual Observatory: An Application for the Data Grid Prototype

This paper presents a prototype of the Japanese Virtual Observatory System, which enables seamless access to federated databases, advanced data analysis, and federation with real telescopes using grid technology.

carlljames
Download Presentation

Virtual Observatory: An Application for the Data Grid Prototype

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An Application for the Data Grid :A Prototype of the Japanese Virtual Observatory System Masatoshi Ohishi National Astronomical Observatory of Japan 　　　　　　and Sokendai masatoshi.ohishi@nao.ac.jp PRAGMA in Fukuoka

  2. NAOJ Y. Mizumoto N. Yasuda Y. Shirasaki M. Tanaka Ochanomizu U. Y. Masunaga Fujitsu Ltd. K. Monzen K. Kawarai Y. Ishihara H. Yanaka Y. Yamaguchi M. Ishida N. Saito Collaborators PRAGMA in Fukuoka

  3. Research Grant from the MEXT 　（文部科学省　科研費 　　特定領域研究 　　「情報学」 　　松岡・下條組） Research Grant from the Sokendai （総合研究大学院大学　企画研究） Supported by PRAGMA in Fukuoka

  4. SUBARU telescope in Hawaii　(optical & IR) PRAGMA in Fukuoka

  5. Orion nebula In the IR PRAGMA in Fukuoka

  6. More than 5,000 Galaxies PRAGMA in Fukuoka

  7. 45m Radio Telescope at Nobeyama PRAGMA in Fukuoka

  8. ALMA in Chile (sub-mm wave) PRAGMA in Fukuoka

  9. Time and Spectral DimensionsThe Multiwavelength Crab Nebulae Crab star 1053 AD X-ray, optical, infrared, and radio views of the nearby Crab Nebula, which is now in a state of chaotic expansion after a supernova explosion first sighted in 1054 A.D. by Chinese Astronomers. PRAGMA in Fukuoka Slide courtesy of Robert Brunner @ CalTech.

  10. Unknowns of Astronomy • Physics just after the Big Bang ? • How Galaxies were formed ? • Averaged density of the Universe etc. →　statistical analyses for homogeneous survey data are crucial PRAGMA in Fukuoka

  11. Data Productivity • Nobeyama Radio ：~1TB/yr • Subaru@Hawaii ：~20TB/yr • ALMA（planned）　 ：~PB/yr Flood of excellent data (survey data) Digitized & Archived PRAGMA in Fukuoka

  12. HPC + High Speed NETWORK +GRID Technology → it is possible to connect and seamlessly use remote DBs and computing facilities → Virtual Observatory PRAGMA in Fukuoka

  13. Telescope is behind me ! PRAGMA in Fukuoka

  14. 観 測 データ 検 索 データ アイデア 解 析 アイデア 解 析 実望遠鏡での 仮想望遠鏡での 研究発表 観測 観測 PRAGMA in Fukuoka

  15. JVO consists of • Seamless Access to Federated DBs • Various Data Analysis tools • Advanced Analysis such as Data Mining • Federation with Real Telescopes →　crucial to control data quality in VO PRAGMA in Fukuoka

  16. JVO prototype : Ver. 1 • Establish seamless access to federated DBs by using the Grid technology • Define JVO Query Language, and implement its parser • Test environment is in a closed subnet PRAGMA in Fukuoka

  17. Query request JVO Query Language JVO Browser JVO Viewer ASTRONOMER Security Service Resource Management Data Management GridFTP each query Resolve servers to request queries Web Broser 利用者独自 サービス Catalog Query Service Commands callable from GT2 JVO Portal Parse the JVO QL into each query sequence Catalog DB Data Archive Service JVO Contoroller skycat Globus Toolokit V2 Data Management DB Data UDDI Server （FreeSoft) Globus Toolkit Service Registry Data Analysis Services PRAGMA in Fukuoka

  18. JVO Query Language(1)Catalog Query PRAGMA in Fukuoka

  19. JVO Query Language(2)Image Query PRAGMA in Fukuoka

  20. Hypertext contents JVO Client Netscape Communicator Java2 Plugin JVO Client (applet) Redhat Linux 2Mass DataBase JVO Prototype System Architecture Redhat Linux JVO Service Gb Ether Solaris8 (Possible at Linux) JVO Controller GSDL UDDI Maintenance Tool UDDI Registory / Servlet Engine Globus Toolkit (Server) Java2SE 1.4 Java2SE 1.4 JVO Service (X match) Apache Tomcat Maintenance Tool UDDI4J soapuddi SQL library JVO Service (select) Apache ＳＯＡＰ • register • update • delete JDBC ① Applet DownLoad ③ Find Service ② Request Observation JVO Controller DB Service Image Data (copy) DB Service Java2SE 1.4 PostgreSQL (DBMS) PostgreSQL JVO Server Monitoring Image Data 2Mass DB UDDI4J SUBARU SupCam z-band UDDI Data • get & parse GSDL • pre condition check • execute Service • post condition check SUBARU DataBase Redhat Linux JVO Service ⑦ issue SQL ⑤ execute Service ③ get GSDL GSDL GSDL (copy) ④’ Get GSDL by GridFTP ⑤’ execute Service Check status Globus Toolkit (Client) Globus Toolkit (Server) Internet ⑥’ spawn Service ④ Get GSDL by GridFTP JVO Service (image) AVO JVO Service JVO Service (X match) SUBARU DataBase Globus Toolkit Globus Toolkit (Server) GridFTP GSDL SQL library JVO Service (select) NVO ⑥ spawn Service JVO Service (select) JVO Service (count) SQL library SQL library JVO Service (image) JVO Service (X match) Globus Toolkit SQL library JVO Service (count) ⑦’ issue SQL Image Data (copy) DB Service Image Data Copy imate data by GridFTP Image Data (copy) ORACLE Copy imate data by GridFTP • The following free software is included by globus toolkit. • OpenSSL (Secure Socket Library) • OpenLDAP (LDAP Server) • wu-ftpd (ftp server) SUBARU SupCam i-band PRAGMA in Fukuoka Image Data Free Software In House Software Commercial Software Function

  21. How it works ? • Read JVO QL • Create “Work Flow”, and execute each step • When necessary, access to the registry to expand the step of the work flow • Thus the steps increases dynamically PRAGMA in Fukuoka

  22. PRAGMA in Fukuoka

  23. PRAGMA in Fukuoka

  24. PRAGMA in Fukuoka

  25. create materialized view my_table as select i.POS_EQ_RA_MAIN, i.POS_EQ_DEC_MAIN, i.PHOT_SDSS_I, z.POS_EQ_RA_MAIN, z.POS_EQ_DEC_MAIN, z.PHOT_SDSS_Z, img_i.BOX(POINT(ra1, dec1), 0.005, 0.005) as img_i, img_z.BOX(POINT(ra2, dec2), 0.005, 0.005) as img_z from Subaru.SPcam.i i, Subaru.SPcam.z z, Subaru.SPcam.i img_i, Subaru.SPcam.z img_z where XMATCH(i, z) < 10 arcsec NEAREST and BOX(POINT(201., 27.4), 0.10, 0.10) and i.PHOT_SDSS_I - z.PHOT_SDSS_Z > 0.0 and i.PHOT_SDSS_I < 20.0 and z.PHOT_SDSS_Z < 20.0 PRAGMA in Fukuoka

  26. PRAGMA in Fukuoka

  27. PRAGMA in Fukuoka

  28. PRAGMA in Fukuoka

  29. PRAGMA in Fukuoka

  30. VOTable as output • XML • Standard data exchange format among VOs defined by the IVOA (see below) PRAGMA in Fukuoka

  31. PRAGMA in Fukuoka

  32. PRAGMA in Fukuoka

  33. FAQs on our Prototype Q) Contents of environment variables are not passed to the executing machines. A) Prepare a shell script to get environment variables and another script to execute the command. Then GridFTP these shell scripts , and execute the shells in the remote machine. Q) It is impossible to prepare all users’ account to all relevant machines. A) Map all accounts on a portal to a single representative one, such as globus. PRAGMA in Fukuoka

  34. FAQs Q) GT2 has too large overhead before starting actual jobs !! They are several seconds. A) For batch jobs, such as database accesses, the overhead could be negligible. However users can never wait for several second for interactive data analyses. Request) Overheads must be < 0.1 seconds excluding the network delays. PRAGMA in Fukuoka

  35. Future • Prototype version 2 implement data analysis tools better user interface • Interoperation with VOs in the world We have joined the International Virtual Observatory Alliance, and are contributing standardizing protocols for interoperability. PRAGMA in Fukuoka

  36. VOs in the World • NVO　(USA), AVO (EU),AstroGrid (UK) • Germany, Canada, Russia, Australia • India, China, JAPAN Protocols to interconnect VOs →　International Virtual Observatory Alliance (IVOA) PRAGMA in Fukuoka

  37. www.ivoa.net PRAGMA in Fukuoka

  38. サービス サービス サービス サービス DB/DA サービス 天体カタログ サービス セキュリティ管理 セキュリティ管理 セキュリティ管理 サービス サービス サービス サービス サービス サービス サービス サービス サービス サービス 利用者独自 サービス サービス サービス サービス サービス 資源管理 資源管理 資源管理 他システム 他システム 他システム データ管理 データ管理 データ管理 処理 pipeline 処理 処理 MVC 天文台殿 MVC 天体カタログDB　 天体カタログ MVC Web サーバ JVO ポータル VO用 ブラウザ セキュリティ管理 資源管理 データ管理 仮想観測実行 サービス 仮想観測 指示 研究者 VO用 Viewer ワーク フロー 天文台殿 DA/DB　 DB Data サービス レジストリA セキュリティ管理 資源管理 データ管理 セキュリティ管理 資源管理 データ管理 GRID環境 A サービス レジストリC サービス レジストリB Web サーバ VO ポータル VO用 ブラウザ セキュリティ管理 研究者 VO用 Viewer 資源管理 他システム データ管理 処理 PRAGMA in Fukuoka GRID環境 C GRID環境 B

  39. JVO Home Page http://jvo.nao.ac.jp/ PRAGMA in Fukuoka

More Related