1 / 17

The ‘Promise of the Grid’

The Ibis Project aims to drastically simplify grid programming and deployment, providing easy-to-use and efficient wall-socket computing over a distributed set of resources. This project addresses performance, scalability, heterogeneous environments, low-level programming interfaces, connectivity issues, fault tolerance, and malleability.

emichelle
Download Presentation

The ‘Promise of the Grid’

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Ibis Project:Simplifying Grid Programming & DeploymentHenri Bal, Jason Maassen, Rob van Nieuwpoort, Thilo Kielmann, Niels Drost, Ceriel Jacobs,Frank Seinstra, Roelof Kemp, Kees VerstoepVrije Universiteit Amsterdam

  2. The ‘Promise of the Grid’ Efficient and transparent (i.e. easy-to-use) wall-socket computing over a distributed set of resources [Sunderam ICCS’2004, based on Foster/Kesselman]

  3. Reality: ‘Problems of the Grid’ • Performance & scalability • Heterogeneous • Low-level & changingprogramming interfaces • writing & deploying grid applications is hard • Connectivity issues • Fault tolerance • Malleability ! User Wide-Area Grid Systems

  4. The Ibis Project • Goal: • drastically simplify grid programming/deployment • ``Grids as promised’’

  5. Approach • Minimal assumptions about execution environment • Virtual Machines (Java) deal with heterogeneity • Use middleware-independent APIs • Different programming abstractions • Designed for dynamic/hostile grid environments • Modular and flexible: can replace Ibis components by external ones

  6. Global picture

  7. Grid programming • Programming models: • Message passing (RMI, MPJ) • Divide-and-conquer (Satin) • IPL (Ibis Portability Layer) • Java-centric “run-anywhere” library, sent with application • Can handle fault-tolerance, malleability • SmartSockets • Solve connectivity problems automatically (firewalls, NAT, addressing problems)

  8. Grid deployment • JavaGAT: Java Grid Application Toolkit • Make applications independent of underlying grid • Used for file copying, resource discovery, job submission & monitoring, user authentication • API is currently standardized (SAGA) • Zorilla P2P system • Job management, gossiping,clustering, flood scheduling

  9. Ibis applications • e-Science (VL-e) • Brain MEG-imaging • Mass spectroscopy • Multimedia content analysis • Video processing • Many HPC applications • SAT-solver, N-body, grammar learning, ….

  10. Grid’5000 European users • D-Grid: Workflow engine for astronomy • U. Erlangen: grid file system • INRIA: ProActive on Ibis RMI • U. Patras: Jylab scientific computing • UPC Barcelona: Grid Superscalar • HITACHI: Peta-scale data management

  11. DAS-3 272 nodes(AMD Opterons) 792 cores 1TB memory LAN: Myrinet 10G Gigabit Ethernet WAN (StarPlane): 20-40 Gb/s OPN See CCGrid’08 session 11 (Verstoep)

  12. Gene sequence comparison in Satin (on DAS-3) Speedup on 1 cluster Run times on 5 clusters • 78% efficiency on 5 clusters (with 1462 WAN-msgs/sec)

  13. Multimedia content analysis • Analyzes video streams to recognize objects • Extract feature vectors from images • Describe properties (color, shape) • Data-parallel task • Compute on consecutive images • Task-parallelism on a grid See CCGrid’08 SCALE challenge & sessions 12+16

  14. High Resolution Video Processing Realtime processing of CineGrid movie data 3840x2160 (4xHD) @ 30 fps = 1424 MB/sec Multi-cluster processing pipeline Using DAS-3, StarPlane and Ibis

  15. Summary • Goal: Simplify grid programming/deployment • Key ideas in Ibis • Virtual machines (JVM) deal with heterogeneity • High-level programming abstractions (Satin) • Handle fault-tolerance, malleability, connectivity problems automatically (Satin, SmartSockets) • Middleware-independent APIs (JavaGAT) • Modular

  16. Future work? • Clouds as promised (CCGrid’09 Shanghai) • Exaflop as promised (CCGrid’10 Melbourne) • Cold fusion as promised • Stock market prediction as promised • Indiana Jones 5 as promised (2032?)

  17. More information • Ibis can be downloaded from • http://www.cs.vu.nl/ibis • Papers: • Satin [PPoPP’07], SmartSockets [HPDC’07], Gossiping [HPDC’07], JavaGAT [SC’07],MMCA [IEEE Multimedia’07, CCGrid’08],DAS-3/StarPlane [CCGrid’08] • Ibis tutorials • CCGrid’08

More Related