DIET a Cloud Middleware for Seed4C

1. DIET a Cloud Middleware for Seed4C Eddy Caron Join work with Jonathan Rouzaud-Cornabas, Frédéric Desprez, Rajesh Palanichamy and the DIET Team Ecole Normale Supérieure de Lyon AVALON Research Team SysFera

2. DIET’s Goals http://graal.ens-lyon.fr/DIET/ • Our goals • To develop a toolbox for the deployment of environments using the Application Service Provider (ASP) paradigm with different applications • Use as much as possible public domain and standard software • To obtain a high performance and scalable environment • Implement and validate our more theoretical results • Scheduling for heterogeneous platforms, data (re)distribution and replication, performance evaluation, algorithmic for heterogeneous and distributed platforms, … • Based on CORBA and our own software developments • FAST for performance evaluation, • LogService for monitoring, • VizDIET for the visualization, • GoDIET for the deployment • Several applications in different fields (simulation, bioinformatics, …) • Release 2.8 available on the web since november • ACI Grid ASP, RNTL GASP, ANR LEGO CIGC-05-11, ANR Gwendia, Celtic-plus Project SEED4C DIET a Cloud Middleware for Seed4C

3. RPC and Grid-Computing: Grid-RPC • One simple idea • Implementing the RPC programming model over the grid • Using resources accessible through the network • Mixed parallelism model (data-parallel model at the server level and task parallelism between the servers) • Features needed • Load-balancing (resource localization and performance evaluation, scheduling), • IDL, • Data and replica management, • Security, • Fault-tolerance, • Interoperability with other systems, • … • Design of a standard interface • within the OGF (Grid-RPC and SAGA WG) • Existing implementations: NetSolve/GridSolve, Ninf, DIET, OmniRPC DIET a Cloud Middleware for Seed4C

4. RPC and Grid Computing: Grid-RPC Request S2 ! A, B, C Answer (C) AGENT(s) Client Op(C, A, B) S4 S3 S1 S2 DIET a Cloud Middleware for Seed4C

5. Client and server interface • Client side • So easy … • Multi-interface (C, C++, Fortran, Java, Python, Scilab, Web, etc.) • Grid-RPC compliant • Server side • Install and submit new server to agent (LA) • Problem and parameter description • Client IDL transfer from server • Dynamic services • new service • new version • security update • outdated service • Etc. DIET a Cloud Middleware for Seed4C

6. Architecture overview DIET a Cloud Middleware for Seed4C

7. Workflow Management • Workflow representation • Direct Acyclic Graph (DAG) • Each vertex is a task • Each directed edge represents communication between tasks • Goals • Build and execute workflows • Use different heuristics to solve scheduling problems • Extensibility to address multi-workflows submission and large grid platform • Manage heterogeneity and variability of environment • ANR Gwendia • Language definition (MOTEUR & MADAG) • Comparison on Grid’5000 vs EGI DIET a Cloud Middleware for Seed4C

8. DIET Scheduling: Plug-in Schedulers • SeD level • Performance estimation function • Estimation Metric Vector - dynamic collection of performance estimation values • Performance measures available through DIET • FAST-NWS performance metrics • Time elapsed since the last execution • CoRI (Collector of Resource Information) • Developer defined values • Aggregation Methods • Defining mechanism to sort SeD responses: associated with the service and defined at SeD level • Tunable comparison/aggregation routines for scheduling • Priority Scheduler • Performs pairwise server estimation comparisons returning a sorted list of server responses; • Can minimize or maximize based on SeD estimations and taking into consideration the order in which the request for those performance estimations was specified at SeD level. DIET a Cloud Middleware for Seed4C

9. Data Management • DAGDA • Joining task scheduling and data management • Standardized through GridRPC OGF WG. • Data Arrangement for Grid and Distributed Applications • Explicit data replication: Using the API. • Implicit data replication. • Data replacement algorithm: • LRU, LFU AND FIFO • Transfer optimization by selectingthe more convenient source. • Storage resources usage management. • Data status backup/restoration. • Join work with Gaël LeMahec (UPJV/MIS) DIET a Cloud Middleware for Seed4C

10. Parallel and batch submissions • Parallel & sequential jobs • transparent for the user • system dependent submission • SeDBatch • Many batch systems • Batch schedulers behaviour • Internal scheduling process • Monitoring & Performance prediction • Simulation (Simbatch) MA LA SeD// NFS SeD SeDBatch OAR SLURM OGE Loadleveler PBS LSF DIET a Cloud Middleware for Seed4C

11. DIET Cloud: Prototype • Inside the Cloud • DIET platform is virtualized inside the cloud. (as Xen image for example) • Very flexible and scalable as DIET nodes can be launched • Scheduling is more complex • DIET as a Cloud manager • Eucalyptus interface • Eucalyptus is treated as a new Batch System • Provide a new implementation for the BatchSystem abstract class DIET a Cloud Middleware for Seed4C

12. DIET Cloud: New Architecture DIET a Cloud Middleware for Seed4C

13. Grid’5000 Grid’5000 • Building a nation wide experimental platform for • Grid & P2P researches (like a particle accelerator for the computer scientists) • 9 geographically distributed sites hosting clusters with 256 CPUs to 1K CPUs) • All sites are connected by RENATER (French Res. and Edu. Net.) • Design and develop a system/middleware environment for safely test and repeat experiments • Use the platform for Grid experiments in real life conditions • 4 main features: • A high security for Grid’5000 and the Internet, despite the deep reconfiguration feature • Single sign-on • High-performance LRMS: OAR • A user toolkit to reconfigure the nodes and monitor experiment: Kadeploy • DIET deployment over a maximum of processors • 1 MA, 8 LA, 540 SeDs • 1120 clients on 140 machines • DGEMM requests (2000x2000 matrices) • Simple round-robin scheduling DIET a Cloud Middleware for Seed4C

14. Conclusions • Grid-RPC • Interesting approach for several applications • Simple, flexible, and efficient • Many interesting research issues (scheduling, data management, resource discovery and reservation, deployment, fault-tolerance, …) • DIET • Scalable, open-source, and multi-application platform • Concentration on several issues like resource discovery, scheduling (distributed scheduling and plugin schedulers), deployment (GoDIET), performance evaluation (CoRI), monitoring (LogService and VizDIET), data management and replication (DAGDA), Cloud support • Large scale validation on the Grid’5000 platform • A middleware designed and tunable for different applications http://www.grid5000.org/ DIET a Cloud Middleware for Seed4C

15. Results • A complete Middleware for heterogeneous infrastructure • DIET is light to use • Dedicated to many applications • Designed for Grid and Cloud • Efficient even in comparison to commercial tools • DIET is high tunability middleware • Used in production • The DIET Team • SysFera Compagny (16 persons today) • http://www.sysfera.com DIET a Cloud Middleware for Seed4C

16. Questions ? http://graal.ens-lyon.fr/DIET