440 likes | 577 Views
D-Grid and AstroGrid-D. Matthias Steinmetz. Matthias Steinmetz (AIP). Trends in astrophysics. Panchromatic studies Combine information of high energy, X-ray, optical, IR, Radio etc. Surveys, populations instead of individual objects Moore‘s law for Optics und Detectors
E N D
D-Grid and AstroGrid-D Matthias Steinmetz Matthias Steinmetz (AIP)
Trends in astrophysics • Panchromatic studies • Combine information of high energy, X-ray, optical, IR, Radio etc. • Surveys, populations instead of individual objects • Moore‘s law for Optics und Detectors • Time resolved astrophysics (PanStarrs, LSST) • Killer asteroids • Extra solar planets • „Streaming Telescopes“ • LOFAR • LSST
Data Growth and Moore‘s Law 1000000 Data volume (storage): Factor 2 every 12 months Factor 32 every 5 years Factor 1000 every 10 years 100000 10000 1000 100 Moore's Law (integration density on chips): Factor 2 every 18 months Factor 10 every 5 years Factor 100 every 10 years 10 1 1990 1995 2000 2005 2010
National Grid Initiatives in Europe • Initially: substantial funding of a few national Grid initiatives (USA, Japan) • Many national projects in Europe: • Austrian Grid Initiative • DutchGrid • France: e-Toile; ACI Grid • Germany: D-Grid; Unicore • Greece: HellasGrid • Grid Ireland • Italy: INFNGrid; GRID.IT • NorduGrid • UK e-Science: National Grid Service; OMII; GridPP
Funding of VO and Grid for Astrophysics in Germany • Application Layer • GAVO • BMBF “large facilities, energy, basic research” • Middleware • D-Grid initiative → AstroGrid-D • BMBF “information and communication technologies, new services”
Project • specific development Project specific development • Project • specific development • Project • specific development Project specific development HEP-Grid AstroGrid-D MediGRID • Project • specific development C3-Grid InGrid • Project • specific development TextGrid WISENT Community Grids … Community Grid Middleware Community Grid Middleware Community Grid Middleware Community Grid Middleware Community Grid Middleware Community Grid Middleware Community Grid Middleware Integration project Grid-Ressources, Middleware, Services D-Grid: First Phase of funding ('05) • D-Grid Integration Project (DGI) • Seven Community Projects: • ~100 partner institutions, Funding: 01.09.05 until 31.08.08
AstroGrid-DAstrophysics • Form a grid-based infrastructure for data and compute resources • Management of distributed data archives • Management of astronomy specific resources (robotic telescopes, gravitational wave detectors) • Event management (supernovae, gamma-ray bursts) • Networking between major research institutes and smaller university partners
C3-GridCollaborative Climate Community • Provide storage and compute resources for scientists • Form a grid-based infrastructure für data- and compute resources • Develop a unified access scheme for distributed data sources • Support user in the analysis of data using standardized work flows
HEP-Grid High Energy Physics • HEP = high energy and particle physics • Analysis of huge amounts of data delivered by particle accelerators (CERN, Fermilab, Stanford Univ.) • Network including many universities and research institutions Goals: • Form a grid based Infrastructure for data und compute resources • Detection of dark matter and dark energy • Detection of the Higgs particle
In-GridEngineering • Methods and models to solve problems in engineering using the grid: • Knowledge based support if decision processes in engineering • Support of workflows in the area of engineering • Distributed, simulations-based product- und process optimization • Virtual prototypes and modeling in industry employing grid methods • Casting processes • Streaming processes • Ground water transport processes • Magneto-hydrodynamical coupling in fluid structures
MediGridHealth Sciences • MediGrid demonstrates medical research as a gird application using biomedical research with multi dimensional data based on the coupling of various genotypes and phenotypes • Goal is a grid supported improvement in the interdisciplinary and location independent collaboration of research institutions in the area of biomedical research
TextGridHumanities • Modular Grid Platform for distributed and cooperative analysis of text data • A community grid for the humanities • Goals: the joint philological analysis, annotation, edition and publication of text data in philology, linguistics and related fields • To form a interdisciplinary, international and interconnected research platform • To demonstrate and use synergy in scientific text data analysis • Optimizing scientific research owing to an optimized access to primary sources and tools
WISENTEnergy meteorology • To obtain information for characterizing the fluctuating energy production based on solar and wind power • To gain insight owing to interdisciplinary cooperation • Climate and Weather(meteorology) • Transformation of wind and solar energy (physics) • To develop powerful and flexible distributed systems (computer science)
D-Grid 2: 12 projects • AeroGrid: Aviation and Space Research • BauVOGrid: Development and Construction • BIS-Grid: Information systems • Biz2Grid: commercial grids ("SAP & Grid") • D-MON: Monitoring of Resources and Services • F&L-Grid: Grid-based IT-Services for Research & Education • FinGrid: Grid for the Finance Sector (Stock market) • GDI-Grid: Geo-Data Infrastructure • PartnerGrid: Cooperation in industrial organizations (KMUs) • ProGrid: automobile industry • SuGI: Grid-Support für small institutions und KMUs
D-Grid 3 call (expected soon) • New communities • Gap projects • Service grids • Service layer agreements • Knowledge management • Extension of D-Grid 1 community projects
D-Grid-Infrastructure (03/2008) 10.000 CPU-Cores, 3,3 PB Disk, 5,5 PB Tape
AstroGrid-DAstrophysics • Form a grid-based infrastructure for data- und compute-resources • Management of distributed data archives • Management of astronomy specific resources (robotic telescopes, gravitational wave detectors) • Event management (supernovae, gamma-ray bursts) • Networking between major research institutes and smaller university partners
AstroGrid-D: Tasks Data analysis Simulations Observations Virtual Observatory AstroGrid-D Integration of Instruments User Interfaces
Galaxy collisions Post processing LISA sanssouci.aip.de X-NBODY6++ Visualization LOFAR AstroGrid-D: Tasks
Dynamo D-Grid Storage Cluster GridSphere GridSphere Stella I + II AstroGrid-D: Tasks
Organization in three major nodes Potsdam / Berlin AIP, AEI, ZIB Heidelberg ZAH Munich/Garching MPA, MPE, TUM AstroGrid-D Partner institution
Middleware development • Information system StellarIS • Telescope Map • Job Management and Monitoring • Management of Virtual Organizations • Grid Application Toolkit (GAT) and Simple API for Grid Applications (SAGA) • Gridsphere Portal
Application Monitoring &Steering Prototypes of grid-enabled tools for monitoring and steering beta-release for Cactus : • Cactus Simulation Metadata Managementbased on AstroGrid-D Information Service Stellaris • Integration in a Cactus User Portal and einNumerical Relativity Portal using GridSphere • Publicly released Demo & Production Service for Cactus user and NumRel-Community • Cooperation between AstroGrid-D and the Center for Computation & Technology (CCT) at Louisiana State University Baton Rouge, USA
GridSphere-Portal • AstroGrid-D User Portal: http://astrogrid-portal.gac-grid.org • Integration of Grid Services • AstroGrid-D MDS • Core D-Grid MDS • AstroGrid-D Resource Map • Simple job submission
StellarIS: resource monitoring (see talk by Ilyia Nickelt) Grid-Ressource-Map Basiert auf GoogleMap
Stellaris: Job monitoring Minutes Time table for submitted Jobs hours days Webinterface for simple job monitoring
Robotic Telescopes STELLA-I • Development of the REMOTE TELESCOPE MARKUP LANGUAGE (RTML) and application to STELLA-I telescope • Can include dynamical information like, e.g., weather • Development of a Transformation from RTML to RDF to communicate resource information to the AstroGrid-D Information Service • User can find suitable telescopes via SPARQL queries • Integration of Web-services via SPARQL queries (e.g. Grid-Resource Map) Robotic Telescopes STELLA-I & II on Tenerife
Robotic TelescopesExample of a SPARQL Query STELLA-I SPARQL Query: PREFIX RTML:<http://www.rtml.org/v3.1a#> SELECT ?name WHERE { ?telescope RTML:SpectralRegion "optical" . ?telescope RTML:Location ?loc . ?loc RTML:Latitude ?lat . FILTER (?lat > 0) . ?loc RTML:name ?name . ?loc RTML:Height ?height } ORDER BY DESC(?height) RDF Notation 3 (N3): @prefix RTML: <http://www.rtml.org/v3.1a#>. RTML:RTML [ RTML:mode "ressource"; RTML:uid "rtml://DE.aip.STELLA-I"; RTML:version "3.1a"; RTML:Telescope [ RTML:name "STELLA-I"; RTML:Aperture [ RTML:type "geometric"; RTML:value "1.2"^^RTML:meters]; RTML:SpectralRegion "optical"; RTML:Location [ RTML:name "Izana Observatory, Teneriffa, Spain"; RTML:Latitude "28.300000"^^RTML:degrees; RTML:EastLongitude "-16.509722"^^RTML:degrees; RTML:Height "2480"^^RTML:meters ; ]]]
Job Management jsdlproc JSDL RSL/XML GUI Job Submission Description Language (JSDL) Supported by open grid forum (OGF) GT4.0
JobMonitoring Job-Timeline GridRessourceMap Stellaris RDF- Upload SPARQL-Query Status-Queries GT4 Ressourcen User-Host Wrapper EPR Job-Submit globusrun-ws –status –j job.epr
Computers Recent Development: GPU – Graphics Cards GeForce 8800 GTX (NVIDIA) Using CUDA Library Special Interfaces and API from GRACE project ported. Berczik et al. 2008 Spurzem et al. 2008
NBODY6++ UseCase original Codes : S.J.Aarseth, S. Mikkola • 4th order time integrator, hierarchical time steps • Ahmad-Cohen neighbor scheme • Regularization of nearby encounters • NBODY6++ (Spurzem 1999) parallelized • φGRAPE – (Harfst et al. 2006) special hardware GRAPE/MPRACE Hardware accelerator, Gain: factor 50-100 new: Reconfigurable Hardware (FGPA), GRACE-Projekt
NBODY6++ UseCase ARI-ZAH GRAPE Cluster: ~3.2 Tflop/s sustained Up to 4 million stars (record holder for direct N-Body simulations) Harfst, Gualandris, Merritt, Spurzem, Portegies Zwart, Berczik 2007, New Astron. 12, 357
Cluster Finder • Combine Information from optical and X-ray catalogues to identify galaxy clusters • Statistics of thousands of clusters is constraining cosmological parameters • Grid for • Compute power • Data transfer from distributed archives • Logistics • complex, distributed workflows X-ray Cluster Finder optical
AstroGrid-D: GEO600 / LIGO Laser Interferometer Gravitational Wave Observatory
GEO600/LIGO Network von 4 Detectors Hanford (2000m side length) USA Livingston (4000m side length) USA GEO600 ( 600m side length) Germany Virgo (3000m side length) Italy Pathfinder for LISA, Satellite mission with 3 detectors side length: 5*109 m!
Gravitational waves: Data analysis via the Grid • Data analyses via small data packages, “embarrassingly parallel”. • Einstein@Home is, like SETI@Home, suitable to exploit idle cycles on work stations. • Einstein@Home is an ideal, simple Grid application, supporting many operation system. • Checkpointing and Recovery allows a very accurate control of CPU-Requirements and walltime. • Automatic software deployment job submission and job management, a good scalability of grid application can be obtained • Current workload: 30000 CPU h per week
GEO600 – Resource Integration • user friendly User-Management viaVOMRS • Resource information via MDS und StellarIS • Grid Service Monitoring • automatic job submission on D-Grid resources • Job monitoring und job management using a Laptop • data management on Astrogrid-D Storage Cluster
AstroGrid-DAstrophysics • Form a grid-based infrastructure for data- und compute-resources • Management of distributed data archives • Management of astronomy specific resources (robotic telescopes, gravitational wave detectors) • Event management (supernovae, gamma-ray bursts) • Networking between major research institutes and smaller university partners