The CrossGrid Project

1. The CrossGrid Project Marcel Kunze, FZK representing the X#-Collaboration

2. Main Objectives • New category of Grid enabled applications • Computing and data intensive • Distributed • Interactive, near real time response (a person in a loop) • Layered • New programming tools • Grid more user friendly, secure and efficient • Interoperability with other Grids • Implementation of standards Marcel Kunze - FZK

3. CrossGrid Collaboration Ireland: TCD Dublin Poland: Cyfronet & INP Cracow PSNC Poznan ICM & IPJ Warsaw 21 institutes 11 countries Germany: FZK Karlsruhe TUM Munich USTU Stuttgart Netherlands: UvA Amsterdam Slovakia: II SAS Bratislava Austria: U.Linz Spain: CSIC Santander Valencia & RedIris UAB Barcelona USC Santiago & CESGA Greece: Algosystems Demo Athens AuTh Thessaloniki Portugal: LIP Lisbon Italy: DATAMAT Cyprus: UCY Nikosia Marcel Kunze - FZK

4. IST Grid Project Space EGSO CROSSGRID GRIA EUROGRID DATAGRID GRIP GRIDLAB DAMIEN DATATAG Industry / business Science -Links with European National efforts - Links with US projects (GriPhyN, PPDG, iVDGL,…) Applications Middleware & Tools Underlying Infrastructures Marcel Kunze - FZK

5. Collaboration with other # Projects • Objective – Exchange of • Information • Software components • Partners • DATAGRID • DATATAG • GRIDLAB • EUROGRID and GRIP • GRIDSTART • Participation in GGF Marcel Kunze - FZK

6. Project Phases M 4 - 12: first development phase: design, 1st prototypes, refinement of requirements M 25 -32: third development phase: complete integration, final code versions M 33 -36: final phase: demonstration and documentation M 1 - 3: requirements definition and merging M 13 -24: second development phase: integration of components, 2nd prototypes Marcel Kunze - FZK

7. Structure Overview Interactive simulation and visualisation of a biomedical system Flooding crisis team support Distributed Data Analysis in High Energy Physics Weather forecast and air pollution modeling APPLICATIONS Remote Data Access Optimization Schedulers Portals TOOLS Monitoring Roaming Access Grid Visualization Kernel Benchmarks Grid SERVICES DATAGRID SET OF TOOLS Protocols, Authentication, Authorization, Access policy, Resource management, etc. GLOBUS TOOLKIT , Condor-G, ... FABRIC INFRASTRUCTURE Network infrastructure, archivers, HPC/HPV systems, Labour instruments, etc. Local domain services Marcel Kunze - FZK

8. CrossGrid Architecture Weather Forecast application Biomedical Application Flood Application HEP Interactive Distributed Data Access Application HEP Data Mining on Grid Application HEP High LevelTrigger Applications And Supporting Tools Portal MPI Verification Performance Analysis Metrics and Benchmarks Applications Development Support MPICH-G Globus Replica Manager Interactive Distributed Data Access Grid Visualisation Kernel Data Mining on Grid Distributed Data Collection Roaming Access Grid Common Services Datagrid Job Manager DataGrid Replica Manager Grid Resource Management User Interaction Service Grid Monitoring GRAM Replica Catalog GSI Globus-IO MDS GridFTP GASS Resource Manager Local Resources Resource Manager Resource Manager Resource Manager Resource Manager Resource Manager Resource Manager Secondary Storage CPU Optimization of Data Access Scientific Instruments (Medical Scaners, Satelites, Radars) Detector Local High Level Trigger VR systems (Caves, immerse desks) Visualization tools Tertiary Storage Marcel Kunze - FZK

9. Layered Structure Interactive and Data Intensive Applications (WP1) • Interactive simulation and visualization of a biomedical system • Flooding crisis team support • Distributed data analysis in HEP • Weather forecast and air pollutionmodeling Grid Application Programming Environment (WP2) • MPI code debugging and verification • Metrics and benchmarks • Interactive and semiautomatic performance evaluation tools Grid Visualization Kernel HLA Data Mining New CrossGrid Services (WP3) DataGrid GriPhyN ... Services • Portals and roaming access • Grid resource management • Grid monitoring • Optimization of data access Globus Middleware Fabric Infrastructure (Testbed WP4) Marcel Kunze - FZK

10. Scope of Applications • Applications in health and environment • Data federation, processing and interpretation in geographically distributed locations • Fast, interactive decision making • Interactive access to distributed • Databases • Super computers and High Performance Clusters • Visualisation engines • Medical scanners • Environmental data input devices Marcel Kunze - FZK

11. Application Requirements • High quality presentation • High frame rate • Intuitive interaction • Real-time response • Interactive algorithms • High performance computing and networking • Distributed resources and data Marcel Kunze - FZK

12. Role of Network Latency Communication delay and rendering delay are negligible Marcel Kunze - FZK

13. CrossGrid Application Development (WP1) • Interactive simulation and visualisation of a biomedical system • Grid-based system for pre-treatment planning in vascular interventional and surgical procedures through real-time interactive simulation of vascular structure and flow. • Flooding crisis team support • Distributed interactive data analysis in HEP • Focus on LHC experiments (ALICE, ATLAS, CMS and LHCb) • Weather forecast and air pollution modelling • Porting distributed/parallel codes on Grid • Coupled Ocean/Atmosphere Mesoscale Prediction System • STEM-II Air Pollution Code Marcel Kunze - FZK

14. Interactive Simulation and Visualisation of a Biomedical System • Grid-based prototype system for treatment planning in vascular interventional and surgical procedures through near real-time interactive simulation of vascular structure and flow. • The system will consist of a distributed near real-time simulation environment, in which a user interacts in Virtual Reality (VR) and other interactive display environments. • A 3D model of the arteries, derived using medical imaging techniques, will serve as input to a simulation environment for blood flow calculations. • The user will be allowed to change the structure of the arteries, thus mimicking an interventional or surgical procedure. Marcel Kunze - FZK

15. Diagnosis & Planning Treatment Observation Current Situation Marcel Kunze - FZK

16. Experimental Setup Marcel Kunze - FZK

17. Simulation Based Planning and Treatment Alternatives Preop AFB w/ E-S Prox.Anast. AFB w/ E-E Prox.Anast. Angio w/Fem-Fem Angio w/ Fem-Fem & Fem-Pop Marcel Kunze - FZK

18. VR-Interaction Marcel Kunze - FZK

19. Flood Crisis Prevention • Support system for establishment and operation of Virtual Organization for Flood Forecasting associating a set of individuals and institutions involved in flood prevention and protection. • The system will employ a Grid technology to seamlessly connect together the experts, data and computing resources needed for quick and correct flood management decisions. • The main component of the system will be a highly automated early warning system based on hydro-meteorological (snowmelt) rainfall-runoff simulations. • System will integrate the advanced communication techniques allowing the crisis management teams to consult the decisions with various experts. The experts will be able to run the simulations with changed parameters and analyze the impact. Marcel Kunze - FZK

20. Virtual Organization for Flood Forecasting Data sources meteorological radars • External sources of information • Global and regional centers GTS • EUMETSAT and NOAA • Hydrological services of other countries surface automatic meteorological and hydrological stations systems for acquisition and processing of satellite information HPC, HTC Storage systems Grid infrastructure meteorological models databases hydrological models hydraulic models Users Flood crisis teams • river authorities • energy • insurance companies • navigation • meteorologists • hydrologists • hydraulic engineers • media • public Marcel Kunze - FZK

21. Flood Crisis Prevention Nosice (Outflow point) Strečno (Inflow point) Pilot Site Catchment Area: 2500km2 (above Strečno: 5500km2) Váh River Catchment Area: 19700km2, 1/3 of Slovakia Váh River Pilot Site • Water stages/discharges in the real time operating hydrological stations • Mapping of the flooded areas Marcel Kunze - FZK

22. Flood SimulationsResults Flow + water depths Marcel Kunze - FZK

23. Distributed Analysis in High Energy Physics • Challenging points • Access to large distributed databases in the Grid • Development of distributed data-mining techniques • Definition of a layered application structure • Integration of user-friendly interactive access (based on PROOF) • Focus on LHC experiments (ALICE, ATLAS, CMS and LHCb) Marcel Kunze - FZK

24. Selection Parameters TagDB CPU Procedure PROOF DB1 RDB CPU Proc.C DB2 Proc.C DB3 CPU Proc.C DB4 CPU Proc.C DB5 CPU Proc.C DB6 CPU PROOF Local Remote Marcel Kunze - FZK

25. Weather Forecast and Air Pollution Modeling • Integration of distributed databases into Grid • Migration of data mining algorithms to Grid • Porting distributed atmospheric & wave models to Grid • Porting parallel codes for air quality models to Grid • Integration, testing and demonstration of the application in the testbed environment Marcel Kunze - FZK

26. COAMPSCoupled Ocean/Atmosphere Mesoscale Prediction System: Atmospheric Components • Complex Data Quality Control • Analysis: • Multivariate Optimum Interpolation Analysis of Winds and Heights • Univariate Analyses of Temperature and Moisture • Optimum Interpolation Analysis of Sea Surface Temperature • Initialization: • Variational Hydrostatic Constraint on Analysis Increments • Digital Filter • Atmospheric Model: • Numerics: Nonhydrostatic, Scheme C, Nested Grids, Sigma-z • Physics: Convection, Explicit Moist Physics, Radiation, Surface Layer • Features: • Globally Relocatable (5 Map Projections) • User-Defined Grid Resolutions, Dimensions, and Number of Nested Grids • 6 or 12 Hour Incremental Data Assimilation Cycle • Can be Used for Idealized or Real-Time Applications • Single Configuration Managed System for All Applications • Operational at: • 7 Areas, Twice Daily, using 81/27/9 km or 81/27 km grids • Forecasts to 72 hours • Operational at all Navy Regional Centers (w/GUI Interface) Marcel Kunze - FZK

27. Status Quo … Quo Vadis ? Current state (briefly) • Simulation done on a single system or local clusters • Visualisation on a single system, locally What we are going to achieve • HPC, HTC, HPV in geographically distributed environment • Improved interaction with the end user • Near real time simulations • Different visualisation equipments (adaptive according to the end-user needs), like • PDA • Workstations • VR studio (e.g. CAVE) Marcel Kunze - FZK

28. Grid Application Programming Environment (WP2) • MPI code debugging and verification • Metrics and benchmarks • Interactive and semiautomatic performance evaluation tools • Specify, develop, integrate, test tools for HPC and HTC applications on the Grid Marcel Kunze - FZK

29. New Grid Services and Tools (WP3) • Portals and roaming access • Grid resource management • Grid monitoring • Optimisation of data access • Objectives • To develop interactive compute- and data-intensive applications • To develop user-friendly Grid environments • To offer easy access to the applications and Grid • To have reasonable trade-off between resource usage efficiency and application speedup • To support management issues while accessing resources Marcel Kunze - FZK

30. International Testbed Organisation (WP4) TCD Dublin PSNC Poznan Uv Amsterdam ICM & IPJ Warsaw FZK Karlsruhe CYFRONET Cracow II SAS Bratislava USC Santiago CSIC Santander LIP Lisbon Auth Thessaloniki UABarcelona CSIC Madrid CSIC Valencia DEMO Athens UCY Nikosia • Testbed setup & incremental evolution • Integration with DataGrid • Infrastructure support • Verification & quality control 15sites Marcel Kunze - FZK

31. Summary • Layered structure of all X# applications • Reuse of SW from DataGrid and other # projects • Globus as the bottom layer of the middleware • Heterogeneous computer and storage systems • Distributed development and testing of SW • 12 partners in applications • 14 partners in middleware • 15 partners in testbeds Marcel Kunze - FZK

32. Grid-enabled Applications Prototype Grid Infrastructures Gèant: World Class Networking 1980s: Internet 1990s: Web 2000s: Grid • Where do we need to get to ? • Applications to support an “e-society” (“Cyber-Infrastructure”) • An international Grid infrastructure which hides the complexities from the users (“Invisible Computing”) • A powerful and flexible network infrastructure • Where do we need to invest ? • Applications targeted at realistic problems in “e-science” • Prototypes of Grid infrastructures • Maintain and improve the GEANT network • Expression of Interest for EU FP6 program: • “Enabling Grids and e-Science in Europe (EGEE)” Marcel Kunze - FZK

33. EGEE Project Space EGSO Enabling Grids and E-Science in Europe (EGEE) CROSSGRID GRIA EUROGRID DATAGRID GRIP GRIDLAB DAMIEN DATATAG Industry / business Science Applications Middleware & Tools Underlying Infrastructures Marcel Kunze - FZK

34. First results of EGEE Brainstorming C Jones, D O Williams, et al Marcel Kunze - FZK