1 / 16

DIET a Cloud Middleware for Seed4C

DIET a Cloud Middleware for Seed4C. Eddy Caron Join work with Jonathan Rouzaud-Cornabas, Frédéric Desprez, Rajesh Palanichamy and the DIET Team Ecole Normale Supérieure de Lyon AVALON Research Team SysFera. DIET’s Goals. http://graal.ens-lyon.fr/DIET/. Our goals

winter
Download Presentation

DIET a Cloud Middleware for Seed4C

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. DIET a Cloud Middleware for Seed4C Eddy Caron Join work with Jonathan Rouzaud-Cornabas, Frédéric Desprez, Rajesh Palanichamy and the DIET Team Ecole Normale Supérieure de Lyon AVALON Research Team SysFera

  2. DIET’s Goals http://graal.ens-lyon.fr/DIET/ • Our goals • To develop a toolbox for the deployment of environments using the Application Service Provider (ASP) paradigm with different applications • Use as much as possible public domain and standard software • To obtain a high performance and scalable environment • Implement and validate our more theoretical results • Scheduling for heterogeneous platforms, data (re)distribution and replication, performance evaluation, algorithmic for heterogeneous and distributed platforms, … • Based on CORBA and our own software developments • FAST for performance evaluation, • LogService for monitoring, • VizDIET for the visualization, • GoDIET for the deployment • Several applications in different fields (simulation, bioinformatics, …) • Release 2.8 available on the web since november • ACI Grid ASP, RNTL GASP, ANR LEGO CIGC-05-11, ANR Gwendia, Celtic-plus Project SEED4C DIET a Cloud Middleware for Seed4C

  3. RPC and Grid-Computing: Grid-RPC • One simple idea • Implementing the RPC programming model over the grid • Using resources accessible through the network • Mixed parallelism model (data-parallel model at the server level and task parallelism between the servers) • Features needed • Load-balancing (resource localization and performance evaluation, scheduling), • IDL, • Data and replica management, • Security, • Fault-tolerance, • Interoperability with other systems, • … • Design of a standard interface • within the OGF (Grid-RPC and SAGA WG) • Existing implementations: NetSolve/GridSolve, Ninf, DIET, OmniRPC DIET a Cloud Middleware for Seed4C

  4. RPC and Grid Computing: Grid-RPC Request S2 ! A, B, C Answer (C) AGENT(s) Client Op(C, A, B) S4 S3 S1 S2 DIET a Cloud Middleware for Seed4C

  5. Client and server interface • Client side • So easy … • Multi-interface (C, C++, Fortran, Java, Python, Scilab, Web, etc.) • Grid-RPC compliant • Server side • Install and submit new server to agent (LA) • Problem and parameter description • Client IDL transfer from server • Dynamic services • new service • new version • security update • outdated service • Etc. DIET a Cloud Middleware for Seed4C

  6. Architecture overview DIET a Cloud Middleware for Seed4C

  7. Workflow Management • Workflow representation • Direct Acyclic Graph (DAG) • Each vertex is a task • Each directed edge represents communication between tasks • Goals • Build and execute workflows • Use different heuristics to solve scheduling problems • Extensibility to address multi-workflows submission and large grid platform • Manage heterogeneity and variability of environment • ANR Gwendia • Language definition (MOTEUR & MADAG) • Comparison on Grid’5000 vs EGI DIET a Cloud Middleware for Seed4C

  8. DIET Scheduling: Plug-in Schedulers • SeD level • Performance estimation function • Estimation Metric Vector - dynamic collection of performance estimation values • Performance measures available through DIET • FAST-NWS performance metrics • Time elapsed since the last execution • CoRI (Collector of Resource Information) • Developer defined values • Aggregation Methods • Defining mechanism to sort SeD responses: associated with the service and defined at SeD level • Tunable comparison/aggregation routines for scheduling • Priority Scheduler • Performs pairwise server estimation comparisons returning a sorted list of server responses; • Can minimize or maximize based on SeD estimations and taking into consideration the order in which the request for those performance estimations was specified at SeD level. DIET a Cloud Middleware for Seed4C

  9. Data Management • DAGDA • Joining task scheduling and data management • Standardized through GridRPC OGF WG. • Data Arrangement for Grid and Distributed Applications • Explicit data replication: Using the API. • Implicit data replication. • Data replacement algorithm: • LRU, LFU AND FIFO • Transfer optimization by selectingthe more convenient source. • Storage resources usage management. • Data status backup/restoration. • Join work with Gaël LeMahec (UPJV/MIS) DIET a Cloud Middleware for Seed4C

  10. Parallel and batch submissions • Parallel & sequential jobs • transparent for the user • system dependent submission • SeDBatch • Many batch systems • Batch schedulers behaviour • Internal scheduling process • Monitoring & Performance prediction • Simulation (Simbatch) MA LA SeD// NFS SeD SeDBatch OAR SLURM OGE Loadleveler PBS LSF DIET a Cloud Middleware for Seed4C

  11. DIET Cloud: Prototype • Inside the Cloud • DIET platform is virtualized inside the cloud. (as Xen image for example) • Very flexible and scalable as DIET nodes can be launched • Scheduling is more complex • DIET as a Cloud manager • Eucalyptus interface • Eucalyptus is treated as a new Batch System • Provide a new implementation for the BatchSystem abstract class DIET a Cloud Middleware for Seed4C

  12. DIET Cloud: New Architecture DIET a Cloud Middleware for Seed4C

  13. Grid’5000 Grid’5000 • Building a nation wide experimental platform for • Grid & P2P researches (like a particle accelerator for the computer scientists) • 9 geographically distributed sites hosting clusters with 256 CPUs to 1K CPUs) • All sites are connected by RENATER (French Res. and Edu. Net.) • Design and develop a system/middleware environment for safely test and repeat experiments • Use the platform for Grid experiments in real life conditions • 4 main features: • A high security for Grid’5000 and the Internet, despite the deep reconfiguration feature • Single sign-on • High-performance LRMS: OAR • A user toolkit to reconfigure the nodes and monitor experiment: Kadeploy • DIET deployment over a maximum of processors • 1 MA, 8 LA, 540 SeDs • 1120 clients on 140 machines • DGEMM requests (2000x2000 matrices) • Simple round-robin scheduling DIET a Cloud Middleware for Seed4C

  14. Conclusions • Grid-RPC • Interesting approach for several applications • Simple, flexible, and efficient • Many interesting research issues (scheduling, data management, resource discovery and reservation, deployment, fault-tolerance, …) • DIET • Scalable, open-source, and multi-application platform • Concentration on several issues like resource discovery, scheduling (distributed scheduling and plugin schedulers), deployment (GoDIET), performance evaluation (CoRI), monitoring (LogService and VizDIET), data management and replication (DAGDA), Cloud support • Large scale validation on the Grid’5000 platform • A middleware designed and tunable for different applications http://www.grid5000.org/ DIET a Cloud Middleware for Seed4C

  15. Results • A complete Middleware for heterogeneous infrastructure • DIET is light to use • Dedicated to many applications • Designed for Grid and Cloud • Efficient even in comparison to commercial tools • DIET is high tunability middleware • Used in production • The DIET Team • SysFera Compagny (16 persons today) • http://www.sysfera.com DIET a Cloud Middleware for Seed4C

  16. Questions ? http://graal.ens-lyon.fr/DIET

More Related