Grid eXplorer (GdX) An Instrument for eXploring the GRID

1. Grid eXplorer (GdX)AnInstrumentfor eXploring the GRID F. Cappello, O. Richard, P. Sens (fci@lri.fr, olivier.richard@imag.fr, pierre.sens@lip6.fr) LRI, ID-Imag, Lip6, etc. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

2. What we know about Grid/P2P? Infrastructure features: • Large scale distributed systems • Dynamic infrastructures (network, nodes) • Dynamic workloads • Heterogeneous (parallel & uniproc. nodes) • High Computation/Communication perf. Ratio F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

3. What we know about Grid/P2P? Application features: • Large data sets (data bases, physics instruments, generation of very large results) • Huge computations (in terms of #operations) • Long run times • Large number of data operations (movements, transactions, etc.) F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

4. What we know about Grid/P2P? Utilization: • A large number of users (researcher, industry, individual, etc.) • A large set of applications (research, business) • High throughput (file download, multi-parameters, computation, etc.) • Real time (Sensors, Entertainment, HPC, etc.) F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

5. What are open issues? • Security (GSI, CAS) • Data Storage/consultation/movement • Multi users/ Multi applications scheduling • Coordination (virtual, ephemeral infrastructure) • Programming • Fault Tolerance! • Scalability • Performance • Easy/efficient deployment techniques • Application characterization techniques • Etc. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

6. What is Grid today? • Middleware: Globus, Legion, Netsolve, Unicore, DIET, Condor, XtremWeb, Boinc, NWS  they are actually working! • Testbed: DataGRID, TeraGRID, Etoile, Grads, XW, Boinc  Difficult to build (debug, human factor, etc.) • Dedicated applications (SETI, Kazaa, Jabber, etc.)  they are working already (at large scale!),  BUT they address much less issues! F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

7. Why Grid is so difficult? • Grids are very complex systems (we still have problems with large scale parallel computers!, we have less control on Grid resources! ) • Many issues are addressed simultaneously  We need a methodology enabling the study of Grid issues, independently but realistically. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

8. What are the current approaches? • Simulators: SimGRID, MicroGRID, etc.  they have strong limitations (scalability, != than execution of real codes, validation) • Experimental testbed (is there any?)  Most testbed are for production, each testeb is specific, representativeness? • We have no way to test: ideas independently, at a significant scale, with realistic parameters and behaviors! F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

9. What is missing? • A full fledge scientific environment (reproducible realistic experimental conditions) • Probes measuring the performance of real resources and networks (Ganglia, NWS, la grenouille) • Fully experimental testbed, (GRID 5000 would remove this lack)  Not enough, we need instruments with parametrisable reproducible experimental conditions F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

10. Methodology • When physicists can’t measure a phenomenon because a) the item to measure is not reachable or b) the phenomenon is hidden by others, • They build instruments! • And they observe… F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

11. Grid eXplorer A tool set for conducting experiments & measurements An experimental conditions database or generation An experimental hardware platform F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

12. Grid eXplorer:Analogy with physic instruments A set of sensors Inside real platform A tool set for Observation Results Analysis A Hardware Platform for Emulation Simulation A database of Real platform Validation on real platforms F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

13. Grid eXplorer:Analogy with physic instruments Experimental Conditions database/ injector 1k Cpu Emulator The Cosmotron. This was the first accelerator in the world to send particles to energies in the billion electron volt, or GeV, region, 1953. Measurement units F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

14. Grid eXplorer (GdX): Instrument for Grid exploration: • An instrument for exploring Grid middleware, algorithms, performance and applications under reproducible experimental conditions • A tool set for emulation/simulation of large scale distributed systems • 1K CPU clusters + configurable network & OS • For Grid, P2P, etc. • Potentially connected to Grid testbeds F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

15. Grid eXplorer (GdX): eXperimental conditions database • A set of sensors (Nodes, Networks):  Academic Networks (x K nodes, GRID 5000)  ADSL (la grenouille  60 K nodes) • A common format for traces • A tool set for accessing, managing traces • Tools for trace analysis F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

16. Grid eXplorer (GdX): Hardware platform for eXperiments: • 1K CPU clusters • configurable network • configurable OS • Multi-users • Potentially located/managed by IDRIS or CEA F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

17. Grid eXplorer (GdX): Tool set: • Emulators (folding 10K virtual nodes on 1K nodes) • Parallel simulators (difficult!) • Virtual GRID environment (1k virtual nodes on 1k nodes) • Measurement tools, • Visualization tools. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

18. Grid eXplorer (GdX): eXperiments: • Real scale pseudo-emulation (1:1 scale) • Large scale emulation (10:1 scale) • Large scale simulation (100:1 scale) • Application test (bottleneck discovery, performance evaluation, optimization) • Infrastructure test (bottleneck discovery, performance evaluation, optimization) • Connection to Grid 5000, International Grids, etc. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

19. Grid eXplorer (GdX): Instrument related projects: • Instrument projects • Cluster management (ID Imag), • Sensor event data base (LRI-fci, la grenouille), • Event analysis (ID Imag), • Emulators (LRI-fci, etc.), • Experimental conditions injection, • Result visualization (???), • Result analysis, • Validation methodology, • User validation (physicists, biologists, chemists, etc.) F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

20. Grid eXplorer (GdX): Grid/P2P research projects: • Grid • DataGRID technology (???) • GRIDRPC (LIP, Desprez?) • Code coupling (IRISA?) • P2P • Large scale desktop Grids (LRI-fci) • Large scale storage (Laria, Irisa?) • Large scale data consistency (Lip6-P. sens) • Large scale fault tolerance (LRI-Beauquier) • Network • Emulation of latency, packet loss, bandwidth • Emulation of topology F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

21. Grid eXplorer (GdX): User applications: • Bio Grid (Genomining, IBBMC, etc.) • Medi GRID (???) • Geo GRID (IFP, CGG) • Power GRID (EDF) • Industry GRID (EADS, Alcatel Space) • Academic GRID (Orsay, Berkeley, SDSC, Suitzerland, etc.) • Industry Partners (GridXpert???, IBM???) F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

22. Grid eXplorer (GdX)Project organisation • A Virtual laboratory • A director (or a group), • A scientific director, an administrative director • A scientific council (with physicists), • A Researcher group • A group of engineers, • A user group • A set of scientific events F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

23. Grid eXplorer GdX • A long term effort • A medium term milestone: 4 years  A fully functional prototype • Many scientific issues (large scale emulation, experimental conditions injection, distance to reality, etc.) • A tool for Grid users or potential users • A tool for Grid/P2P developers F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

24. “Enabling Grid 5000”Proposition d’Action Spécifique RTP Grid Grid 5000 Grid eXplorer F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal