The Interactive European Grid Project

1. The Interactive European Grid Project introduced by Dr. Jesús Marco de Lucas CSIC, Instituto de Física de Cantabria, Santander, SPAIN EELA Conference, 4 September 2006, Santiago de Chile

2. Researchers demand resources int.eu.grid in three slides int.eu.grid VISION • “Interoperable production-level e-Infrastructure for demanding interactive applications to impact the daily work of researchers” • Distributed Parallel (MPI) Interactive Computing & Storage at the Tera level • User Friendly Access • Grid Interactive Desktop Example: Ultrasound Computer Tomography A new method of medical imaging based on the reconstruction by numerical techniques of an image, using as input the data measured by a scanner of ultrasounds which surrounds the object of interest. The application requires analyzing about 20 Gb of data, which would take order of one month in a workstation… J. Marco , EELA Conference, 4 September, Santiago de Chile

3. A Real Challenge! • the int.eu.grid project aims to change the way researchers can use the available e-Infrastructure, exploiting the interactivity and collaboration possibilities • Researchers need to be convinced that they can: • Transfer and process gigabytes of information in minutes • Foresee more complex algorithms on larger statistics, test and tune them, use more powerful visualization techniques • Collaborate across the network in a rewarding mode, from sharing information to discussing and presenting remotely through enhanced videoconference environments. J. Marco , EELA Conference, 4 September, Santiago de Chile

4. Interactive European Grid (http://www.interactive-grid.eu) Projectacronym int.eu.grid Contract number 031857 InstrumentI3 Duration 2 years may ´06-april ´08 “providing transparently the researcher’s desktop with the power of a supercomputer, using distributed resources” http://www.interactive-grid.eu Coordinator: CSIC, Jesús Marco, IFCA, Santander, SPAIN [marco@ifca.unican.es] J. Marco , EELA Conference, 4 September, Santiago de Chile

5. My summary of the GRID(testbed/integrator/developer/user view) • The Grid “idea” is as good as difficult to put in practice • Using the network, and open standards, join resources from different sites and make them available to user communities in a transparent way: Sit users in front of a 1000 GHz/ 100 Tbytes computer • Theoretically, it makes sense: • The network is there now (and evolves even faster than CPUs!) • Basic middleware exists (for a “secure” communication sharing resources) • KEY point: “SHARING” & “COLLABORATION” • This is EXTREMELY difficult, why should we just bother????? • Why not use just resources at one site (clusters, SC centers)? • The answer is two-fold: • e-Science scenario • Meta-computing possibilities J. Marco , EELA Conference, 4 September, Santiago de Chile

6. About a 1000GHz/100Tbytes shared computer… • We have a clear idea of how to setup and also of how to use it! • LARGE CLUSTER • Get an account/password from administrator • Develop your code, parallelize/distribute • Use a repository • Compile using MPI, make executable, store it • Set links to data • Submit script to PBS or LSF queue • Monitor job status • Retrieve output & logs from queue • Display output • GRID FRAMEWORK • Get certificate, register in Virtual Organization • Develop your code, use Grid API • Use a CVS repository to share with others • Compile using MPI, make executable, store in Virtual directory (SE) • Define data lfn (will be stored in SE) • Submit to WMS/RB from a Grid Desktop • Check jobstatus • Get output, store in Virtual directory • Display output in Grid Desktop J. Marco , EELA Conference, 4 September, Santiago de Chile

7. Executing Applications: HTC vs. HPC • High Throughput Computing : • Maximize the number of program executions • Example: 1.000.000 independent simulations • Use 1000 processors, do 1000 simulations in each one • Aggregate for final result • Works perfectly in a cluster • Examples: Condor, batch queues: LSF, PBS • Works perfectly in a GRID with a Workload Management System • High Performance Computing: • Minimize the answer time • Example: Train a complex Neural Network • Use 1000 processors in parallel • The information of all of them is required along the job • Needs a “parallelized description” (e.g. MPI) + and a topology (master-workers) • How to handle network latency in a GRID framework? (or in general QoS) • Metacomputing “issues” J. Marco , EELA Conference, 4 September, Santiago de Chile

8. Benefit for applications Parallelization techniques reduce waiting time in Neural Network Application Scale as 1/N 412980 events, 16 variables 16-10-10-1 architecture 1000 epochs for training Time reduction: from 5 hours down to 5 min using 60 nodes! Modeling including latency <300 ms needed ! J. Marco , EELA Conference, 4 September, Santiago de Chile

9. Innovation • What innovation do we need? • The CrossGrid project demonstrated the feasibility of an interactive Grid framework showing applications running in a testbed using middleware developed on top of EDG/LCG services. • int.eu.grid will provide a production quality service supporting MPI and interactivity on top of EGEE services • Web Services approach will be followed (gLite as underlying middleware) • MPI support will be based on PACX-MPI + OPEN-MPI • European MPI software (HLRS) + International effort • MPI across sites without the need for public IP on WNs • Interactivity support will integrate: • User friendly interface to the infrastructure • Powerful visualization capabilities • Consideration of QoS J. Marco , EELA Conference, 4 September, Santiago de Chile 4

10. Grid Desktop • Single sign-on / authorisation • Platform independent • Batch jobs • MPI jobs • Running interactive applications using java plugins or VNC • Monitoring grid applications • Flexible Application framework • User profile management • Easy application add on • Local and grid file management • GridFTP Commander • User Profile Manager • Private Storage Management • VNC/SSH console Desktop tools: • Job Wizard • Job Monitor • Application Container and Plugin J. Marco , EELA Conference, 4 September, Santiago de Chile

11. e-Infrastructure Project Resources • Grid-empowered Infrastructure • Initial core: 13 sites in Europe • linked through GEANT-2 • Computers • Clusters with 10-100 Pentium/Xeon nodes • but also Itanium, Opteron, Power5 2-4 processors nodes, multi core, large RAM • Interoperable • EGEE batch jobs will be able to execute in background mode • Human resources • Excellent subteams already “in place”, examples: • CrossGrid WP4-sites & Integration Team, driven by FZK & LIP • PSNC-UAB-CSIC on interactivity, MPI use, and applications • Operation of infrastructure • Continuous effort along the project • Benefit from support expertise for middleware Integration (experience from CrossGrid) J. Marco , EELA Conference, 4 September, Santiago de Chile 3

12. Sub-objectivesthrough the I3 instrument: • Deployment of e-Infrastructure • Oriented to interactive use • Site integration support • Grid operations service • Middleware for interactivity and MPI • Adapt/integrate existing middleware • guarantee interoperability with EGEE • Provide a complete interactivity suite • Desktop • roaming access • scheduler with prioritization services • complex visualization. • Support for interactive applications: • setup of collaborative environment and VO • consideration of performance • interactivity and visualization requirements • identification and selection of research oriented interactive applications • Support remote collaboration activities: • research, management, integration, training • Approach target research communities • Provide security measures for interactivity Summary: the int.eu.grid Mission “To deploy and operate a production-quality Grid-empowered eInfrastructure oriented to service research communities supporting demanding interactive applications.” J. Marco , EELA Conference, 4 September, Santiago de Chile 2

13. Potential Impact INPUT TO/FROM OTHER PROJECTS On experience, middleware, application support, training, test of infrastructure Input to National Grid initiatives DGRID, AustriaGrid, GridIreland Collaboration with EGEE Aim for interoperability Objective: two way synergy: transfer experience and middleware to/from EGEE Training and Dissemination Offer complementary services to a large user base Dissemination effort Presence in EGEE/GGF meetings Addressing different research communities at their workshops (national and international wide contacts) Ex. FZK, PSNC, LIP, TCD Profit from multidisciplinary framework Ex. CSIC: horizontal program • APPLICATIONS: • Medicine: • Ultrasound Computer Tomography Simulation, Visualization, Remote Access • Environmental studies: • Landscape modelling , Pollution • Physics: • Astrophysics: Planck mission • Fusion: visualization and 3D transport codes HEP: filtering of data produced by ATLAS • Attracting others: • Through dissemination activities • Identification of Applications • Outreach: workshops, seminars, courses • Definition of requirements (and use cases) for a given application/community • Analyze the possibilities for use of the eInfrastructure • Test and train on real infrastructure • Profiting from links with • National Grid Initiatives • EGEE • Other international projects (like EELA) J. Marco , EELA Conference, 4 September, Santiago de Chile 11

14. Collaboration with EELA • The Interactive European Grid open to collaboration • Listen/Support researchers needs for interactive demanding applications • Offer MPI expertise, in particular for distributed applications • Run on resources adapted for MPI (multi cpu, large RAM, etc) • Interactive oriented setup • Test collaborative and user friendly framework • Try Grid Desktop • Add Powerful visualization • Join Access Grid collaborative experience • Collaborate on development/deployment issues • Simplified installation and virtualization • Application Development framework • Common partners: LIP, CSIC Hope this collaboration will be fruitful! Thanks! J. Marco , EELA Conference, 4 September, Santiago de Chile 4