1 / 44

D-Grid and AstroGrid-D

D-Grid and AstroGrid-D. Matthias Steinmetz. Matthias Steinmetz (AIP). Trends in astrophysics. Panchromatic studies Combine information of high energy, X-ray, optical, IR, Radio etc. Surveys, populations instead of individual objects Moore‘s law for Optics und Detectors

eithne
Download Presentation

D-Grid and AstroGrid-D

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. D-Grid and AstroGrid-D Matthias Steinmetz Matthias Steinmetz (AIP)

  2. Trends in astrophysics • Panchromatic studies • Combine information of high energy, X-ray, optical, IR, Radio etc. • Surveys, populations instead of individual objects • Moore‘s law for Optics und Detectors • Time resolved astrophysics (PanStarrs, LSST) • Killer asteroids • Extra solar planets • „Streaming Telescopes“ • LOFAR • LSST

  3. Data Growth and Moore‘s Law 1000000 Data volume (storage): Factor 2 every 12 months Factor 32 every 5 years Factor 1000 every 10 years 100000 10000 1000 100 Moore's Law (integration density on chips): Factor 2 every 18 months Factor 10 every 5 years Factor 100 every 10 years 10 1 1990 1995 2000 2005 2010

  4. National Grid Initiatives in Europe • Initially: substantial funding of a few national Grid initiatives (USA, Japan) • Many national projects in Europe: • Austrian Grid Initiative • DutchGrid • France: e-Toile; ACI Grid • Germany: D-Grid; Unicore • Greece: HellasGrid • Grid Ireland • Italy: INFNGrid; GRID.IT • NorduGrid • UK e-Science: National Grid Service; OMII; GridPP

  5. Funding of VO and Grid for Astrophysics in Germany • Application Layer • GAVO • BMBF “large facilities, energy, basic research” • Middleware • D-Grid initiative → AstroGrid-D • BMBF “information and communication technologies, new services”

  6. Project • specific development Project specific development • Project • specific development • Project • specific development Project specific development HEP-Grid AstroGrid-D MediGRID • Project • specific development C3-Grid InGrid • Project • specific development TextGrid WISENT Community Grids … Community Grid Middleware Community Grid Middleware Community Grid Middleware Community Grid Middleware Community Grid Middleware Community Grid Middleware Community Grid Middleware Integration project Grid-Ressources, Middleware, Services D-Grid: First Phase of funding ('05) • D-Grid Integration Project (DGI) • Seven Community Projects: • ~100 partner institutions, Funding: 01.09.05 until 31.08.08

  7. AstroGrid-DAstrophysics • Form a grid-based infrastructure for data and compute resources • Management of distributed data archives • Management of astronomy specific resources (robotic telescopes, gravitational wave detectors) • Event management (supernovae, gamma-ray bursts) • Networking between major research institutes and smaller university partners

  8. C3-GridCollaborative Climate Community • Provide storage and compute resources for scientists • Form a grid-based infrastructure für data- and compute resources • Develop a unified access scheme for distributed data sources • Support user in the analysis of data using standardized work flows

  9. HEP-Grid High Energy Physics • HEP = high energy and particle physics • Analysis of huge amounts of data delivered by particle accelerators (CERN, Fermilab, Stanford Univ.) • Network including many universities and research institutions Goals: • Form a grid based Infrastructure for data und compute resources • Detection of dark matter and dark energy • Detection of the Higgs particle

  10. In-GridEngineering • Methods and models to solve problems in engineering using the grid: • Knowledge based support if decision processes in engineering • Support of workflows in the area of engineering • Distributed, simulations-based product- und process optimization • Virtual prototypes and modeling in industry employing grid methods • Casting processes • Streaming processes • Ground water transport processes • Magneto-hydrodynamical coupling in fluid structures

  11. MediGridHealth Sciences • MediGrid demonstrates medical research as a gird application using biomedical research with multi dimensional data based on the coupling of various genotypes and phenotypes • Goal is a grid supported improvement in the interdisciplinary and location independent collaboration of research institutions in the area of biomedical research

  12. TextGridHumanities • Modular Grid Platform for distributed and cooperative analysis of text data • A community grid for the humanities • Goals: the joint philological analysis, annotation, edition and publication of text data in philology, linguistics and related fields • To form a interdisciplinary, international and interconnected research platform • To demonstrate and use synergy in scientific text data analysis • Optimizing scientific research owing to an optimized access to primary sources and tools

  13. WISENTEnergy meteorology • To obtain information for characterizing the fluctuating energy production based on solar and wind power • To gain insight owing to interdisciplinary cooperation • Climate and Weather(meteorology) • Transformation of wind and solar energy (physics) • To develop powerful and flexible distributed systems (computer science)

  14. D-Grid 2: 12 projects • AeroGrid: Aviation and Space Research • BauVOGrid: Development and Construction • BIS-Grid: Information systems • Biz2Grid: commercial grids ("SAP & Grid") • D-MON: Monitoring of Resources and Services • F&L-Grid: Grid-based IT-Services for Research & Education • FinGrid: Grid for the Finance Sector (Stock market) • GDI-Grid: Geo-Data Infrastructure • PartnerGrid: Cooperation in industrial organizations (KMUs) • ProGrid: automobile industry • SuGI: Grid-Support für small institutions und KMUs

  15. D-Grid 3 call (expected soon) • New communities • Gap projects • Service grids • Service layer agreements • Knowledge management • Extension of D-Grid 1 community projects

  16. D-Grid-Infrastructure (03/2008) 10.000 CPU-Cores, 3,3 PB Disk, 5,5 PB Tape

  17. AstroGrid-DAstrophysics • Form a grid-based infrastructure for data- und compute-resources • Management of distributed data archives • Management of astronomy specific resources (robotic telescopes, gravitational wave detectors) • Event management (supernovae, gamma-ray bursts) • Networking between major research institutes and smaller university partners

  18. AstroGrid-D: Tasks Data analysis Simulations Observations Virtual Observatory AstroGrid-D Integration of Instruments User Interfaces

  19. Galaxy collisions Post processing LISA sanssouci.aip.de X-NBODY6++ Visualization LOFAR AstroGrid-D: Tasks

  20. Dynamo D-Grid Storage Cluster GridSphere GridSphere Stella I + II AstroGrid-D: Tasks

  21. Organization in three major nodes Potsdam / Berlin AIP, AEI, ZIB Heidelberg ZAH Munich/Garching MPA, MPE, TUM AstroGrid-D Partner institution

  22. Middleware development • Information system StellarIS • Telescope Map • Job Management and Monitoring • Management of Virtual Organizations • Grid Application Toolkit (GAT) and Simple API for Grid Applications (SAGA) • Gridsphere Portal

  23. Application Monitoring &Steering Prototypes of grid-enabled tools for monitoring and steering beta-release for Cactus : • Cactus Simulation Metadata Managementbased on AstroGrid-D Information Service Stellaris • Integration in a Cactus User Portal and einNumerical Relativity Portal using GridSphere • Publicly released Demo & Production Service for Cactus user and NumRel-Community • Cooperation between AstroGrid-D and the Center for Computation & Technology (CCT) at Louisiana State University Baton Rouge, USA

  24. CACTUS user portal (based on GridSphere)

  25. GridSphere-Portal • AstroGrid-D User Portal: http://astrogrid-portal.gac-grid.org • Integration of Grid Services • AstroGrid-D MDS • Core D-Grid MDS • AstroGrid-D Resource Map • Simple job submission

  26. StellarIS: resource monitoring (see talk by Ilyia Nickelt) Grid-Ressource-Map Basiert auf GoogleMap

  27. Stellaris: Job monitoring Minutes Time table for submitted Jobs hours days Webinterface for simple job monitoring

  28. Robotic Telescopes STELLA-I • Development of the REMOTE TELESCOPE MARKUP LANGUAGE (RTML) and application to STELLA-I telescope • Can include dynamical information like, e.g., weather • Development of a Transformation from RTML to RDF to communicate resource information to the AstroGrid-D Information Service • User can find suitable telescopes via SPARQL queries • Integration of Web-services via SPARQL queries (e.g. Grid-Resource Map) Robotic Telescopes STELLA-I & II on Tenerife

  29. Robotic TelescopesExample of a SPARQL Query STELLA-I SPARQL Query: PREFIX RTML:<http://www.rtml.org/v3.1a#> SELECT ?name WHERE { ?telescope RTML:SpectralRegion "optical" . ?telescope RTML:Location ?loc . ?loc RTML:Latitude ?lat . FILTER (?lat > 0) . ?loc RTML:name ?name . ?loc RTML:Height ?height } ORDER BY DESC(?height) RDF Notation 3 (N3): @prefix RTML: <http://www.rtml.org/v3.1a#>. RTML:RTML [ RTML:mode "ressource"; RTML:uid "rtml://DE.aip.STELLA-I"; RTML:version "3.1a"; RTML:Telescope [ RTML:name "STELLA-I"; RTML:Aperture [ RTML:type "geometric"; RTML:value "1.2"^^RTML:meters]; RTML:SpectralRegion "optical"; RTML:Location [ RTML:name "Izana Observatory, Teneriffa, Spain"; RTML:Latitude "28.300000"^^RTML:degrees; RTML:EastLongitude "-16.509722"^^RTML:degrees; RTML:Height "2480"^^RTML:meters ; ]]]

  30. Job Management jsdlproc JSDL RSL/XML GUI Job Submission Description Language (JSDL) Supported by open grid forum (OGF) GT4.0

  31. JobMonitoring Job-Timeline GridRessourceMap Stellaris RDF- Upload SPARQL-Query Status-Queries GT4 Ressourcen User-Host Wrapper EPR Job-Submit globusrun-ws –status –j job.epr

  32. Computers Recent Development: GPU – Graphics Cards GeForce 8800 GTX (NVIDIA) Using CUDA Library Special Interfaces and API from GRACE project ported. Berczik et al. 2008 Spurzem et al. 2008

  33. NBODY6++ UseCase original Codes : S.J.Aarseth, S. Mikkola • 4th order time integrator, hierarchical time steps • Ahmad-Cohen neighbor scheme • Regularization of nearby encounters • NBODY6++ (Spurzem 1999) parallelized • φGRAPE – (Harfst et al. 2006) special hardware GRAPE/MPRACE Hardware accelerator, Gain: factor 50-100 new: Reconfigurable Hardware (FGPA), GRACE-Projekt

  34. NBODY6++ UseCase ARI-ZAH GRAPE Cluster: ~3.2 Tflop/s sustained Up to 4 million stars (record holder for direct N-Body simulations) Harfst, Gualandris, Merritt, Spurzem, Portegies Zwart, Berczik 2007, New Astron. 12, 357

  35. Cluster Finder • Combine Information from optical and X-ray catalogues to identify galaxy clusters • Statistics of thousands of clusters is constraining cosmological parameters • Grid for • Compute power • Data transfer from distributed archives • Logistics • complex, distributed workflows X-ray Cluster Finder optical

  36. AstroGrid-D: GEO600 / LIGO Laser Interferometer Gravitational Wave Observatory

  37. GEO600/LIGO Network von 4 Detectors Hanford (2000m side length) USA Livingston (4000m side length) USA GEO600 ( 600m side length) Germany Virgo (3000m side length) Italy Pathfinder for LISA, Satellite mission with 3 detectors side length: 5*109 m!

  38. Gravitational waves: Data analysis via the Grid • Data analyses via small data packages, “embarrassingly parallel”. • Einstein@Home is, like SETI@Home, suitable to exploit idle cycles on work stations. • Einstein@Home is an ideal, simple Grid application, supporting many operation system. • Checkpointing and Recovery allows a very accurate control of CPU-Requirements and walltime. • Automatic software deployment job submission and job management, a good scalability of grid application can be obtained • Current workload: 30000 CPU h per week

  39. GEO600 – Resource Integration • user friendly User-Management viaVOMRS • Resource information via MDS und StellarIS • Grid Service Monitoring • automatic job submission on D-Grid resources • Job monitoring und job management using a Laptop • data management on Astrogrid-D Storage Cluster

  40. AstroGrid-DAstrophysics • Form a grid-based infrastructure for data- und compute-resources • Management of distributed data archives • Management of astronomy specific resources (robotic telescopes, gravitational wave detectors) • Event management (supernovae, gamma-ray bursts) • Networking between major research institutes and smaller university partners

More Related