Interactive European Grid Environment for HEP Application with Real Time Requirements

1. Interactive European Grid Environment for HEP Application with Real Time Requirements Lukasz Dutka1, Krzysztof Korcyl2, Krzysztof Zielinski1,3, Jacek Kitowski1,3, Renata Slota3, Wlodzimierz Funika3, Kazimierz Balos1, Lukasz Skital1, Bartosz Kryza1, Jan Pieczykolan1 1ACK Cyfronet AGH 2Institute of Nuclear Physics Polish Academy of Sciences 3Univeristy of Science and Technology AGH Cracow Grid Workshp 2006 – Krakow

2. Outline • int.eu.grid mission • HEP application real time requirements • int.eu.grid approach to support HEP • Summary

3. Researchers demand resources int.eu.grid in three slides int.eu.grid VISION • “Interoperable production-level e-Infrastructure for demanding interactive applications to impact the daily work of researchers” • Distributed Parallel (MPI) Interactive Computing & Storage at the Tera level • User Friendly Access • Grid Interactive Desktop Example: Ultrasound Computer Tomography A new method of medical imaging based on the reconstruction by numerical techniques of an image, using as input the data measured by a scanner of ultrasounds which surrounds the object of interest. The application requires analyzing about 20 Gb of data, which would take order of one month in a workstation…

4. A Real Challenge! • the int.eu.grid project aims to change the way researchers can use the available e-Infrastructure, exploiting the interactivity and collaboration possibilities • Researchers need to be convinced that they can: • Transfer and process gigabytes of information in minutes • Foresee more complex algorithms on larger statistics, test and tune them, use more powerful visualization techniques • Collaborate across the network in a rewarding mode, from sharing information to discussing and presenting remotely through enhanced videoconference environments.

5. Interactive European Grid (http://www.interactive-grid.eu) Projectacronym int.eu.grid Contract number 031857 InstrumentI3 Duration 2 years may ´06-april ´08 “providing transparently the researcher’s desktop with the power of a supercomputer, using distributed resources” http://www.interactive-grid.eu Coordinator: CSIC, Jesús Marco, IFCA, Santander, SPAIN [marco@ifca.unican.es]

6. Summary: the int.eu.grid Mission “To deploy and operate a production-quality Grid-empowered eInfrastructure oriented to service research communities supporting demanding interactive applications.” • Deployment of e-Infrastructure • Oriented to interactive use • Site integration support • Grid operations service • Middleware for interactivity and MPI • Adapt/integrate existing middleware • guarantee interoperability with EGEE • Provide a complete interactivity suite • Desktop • roaming access • scheduler with prioritization services • complex visualization. • Support for interactive applications: • setup of collaborative environment and VO • consideration of performance • interactivity and visualization requirements • identification and selection of research oriented interactive applications • Support remote collaboration activities: • research, management, integration, training • Approach target research communities • Provide security measures for interactivity

7. Remote Processing Farms int.eu.grid PF PF PF PF Packet Switched WAN: GEANT SFI SFI SFI lightpath PF SFOs Switch PF int.eu.grid infrastructure HEP Requirements in Brief 1second to process one event 3000 events per second 1.5MB each event When we need more power we should be able to harness grid Back End Network Local Event Processing Farms mass storage CERN

8. HEP Use Case Requirments • To locate processing resources where processing tasks (PT) will be started. These tasks will be waiting to receive data and process the data once they are received. • Access to each of the CPU found should allow to deliver 1.5 MB of event data every second (the infrastructure should not propose more CPUs per site then the available access bandwidth will allow to use) • Event data transfer time should be minimal and should not interfere with processing resources allocated for processing • Sites with allocated CPUs should have recently updated data base (new updates at CERN every day or a few times per day) • The list of sites with current resources complying with the requirements should be presented to ATLAS operator and the final decision on selection of sites will be left up to him. • Operator should be able to pass approx. 1000 events per second to grid infrastructure!

9. How to use the Grid faster? • Job submission for each event is to slow • We need interactive communication! • Pilot job idea • One job to allocate a node and start PT • One PT process many events • Direct communication between PT and EFD • Faster than job submission • EFD provides event (1.5MB/event) • PT responds with events analysis results (1Kb/event) • Limited lifetime of PT to allow dynamic resource allocation

10. SFI SFI SFI PT PT PT PT PT PT PT PT PT PT PT PT Proposed HEP Architecture UI Broker EFD Dispatcher Buffer EFD Buffer proxyPT Infrastructure monitoring HEP VO Database int.eu.grid HEP VO PT PT PT PT PT PT Application Monitoring Local PT Farm CE Events CE Remote PTs WNs WNs

11. Development Status • HEP Application is developed independently to the project • We need to develop additional tools used to gridify application: • at the moment we already designed solution and we started the implementation process • very first and very simple demo will be ready in 3-4 months • a bit more mature prototype should be ready before the review in may • the development involves QoS issues

12. Events wait in many queues Intelligent dispatcher has a view at the current status of computational resources occupied by previously submitted jobs, and it distributes events to best resources Summary • Interactive grid environment helps us to fulfill rigorous time requirements • The intelligent distribution of event meets QoS requirements, exploits and optimize load of the system • The solution designed for HEP application is ready to be used for other applications requiring access to grid computational power in a similar way.

13. Thank you!