TCM: Virtualization of Resources / Infrastructure

1. Solutions for Quasi-opportunistic Supercomputers G. Mécheneau Platform Computing gmecheneau@platform.com TCM: Virtualization of Resources / Infrastructure

2. Takeaway • Focus of the project on • Infrastructure for net-aware apps • Real use cases (9) • Virtualization of grid topology • Is critical for complex applications • Is possible • Existing, deployable technologies

3. Virtualization ? The internet is a fantastic place

4. Virtualization ??? A definition as presented by the science guy in:

5. Virtualization ! Through Wikipedia : Virtualization is the process by which "the supercomputer analyzes your molecular structure through the scanners and breaks down your atoms before digitalizing them and recreating a digital incarnation in the virtual world."

6. No really, virtualization. "the supercomputer analyzes your molecular structure through the scanners and breaks down your atoms before digitalizing them and recreating a digital incarnation in the virtual world."

7. QosCosGrid Vision:Grid ≈ virtual supercomputer ≈ This workwassupported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project 10/11/2014 7

8. What for ?(Or: Our target use cases) • Complex Systems • Systems composed of interacting components • "The whole is more than the sum of its parts" • Typically not amenable to analytic solutions

10. Complex SystemsSimulations (Examples) • N-Particle Systems • Protein Interactions • Metabolic Pathways • Financial Markets • Market Research • Supply Chain Optimization • Ecological and Population Dynamics • Stellar Systems This workwassupported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project

11. Virtualization in practice JSDL RTG GRMS • Description language • For the jobs • For the grid • Programming environnement • Resource management : fabric • Metascheduler • Able to understand workflows of simpler entities

12. Architecture Overview This workwassupported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project

13. Implementation QCG Parallel Toolkit This workwassupported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project

14. Testbed This workwassupported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project

15. QosCosGrid workflow • On-demand resource acquisition and formation of application specific Grids: QosCosGrid • QosCosGrid (= Quasi Opportunistic Supercomputing for Complex Systems on the Grid) Complex Systems applications use the QosCosGrid-Toolbox to parallelize their workload in order to use distributed resources. • Towards the QosCosGrid-Broker applications express requirements and behavior by xml “job profile” • The Broker acquires resources on-demand and form an application specific Grid, creates RTG = resource topology graph • RTG is used to map the application to the resources, placing MPI communicators at the right place This workwassupported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project

16. NExTml: QCG Job profile • The QCG Job Profile document is inspired by an existing XML-based job description language supported by one of the main components of the QCG middlewarecalled GRMS. • End users can describe topology and resource requirements, in particular: • required aggregations and hierarchies of resources (computing nodes, clusters, sub-clusters, storage elements etc.), • required resource properties (operating system, memory, number of CPUs, speed of the CPU on a resource), • required network and connection properties (bandwidth, latency and capacity), • required applications and licenses available at destination computing resources. This workwassupported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project

17. NExTml :QCG Resource Description Model • RTG (Resource Topology Graph) • A common XML resource description language • Provide description of: • Resources, tasks, processes • Topology • Communication properties • Serves as a “bridge” between the various system components • Used to describe, publish, evaluate, reserve and monitor heterogeneous resources across the QosCos Grid • Supplementary Java implementation: • Functional behavior and logic • XML to Java objects marshaling/un-marshaling • Specialized types of RTG objects, according to the middleware requirements (i.e. Resource advertisement, Meta-scheduling, SLA’s, Monitoring, and User requirements.) This workwassupported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project

18. Example job on a QosCosGrid • Write your description • Submit it to GRMS • GRMS • queries existing Grid Topology • allocates tasks of the workflow on the grid

19. Example job on a QosCosGrid AO AO AO QCG-RMI RMI AO AO AO AO AO AO Usecase 8: AITIA • Predator-prey ecology • Cellular Automaton (CA)‏ • Partitioning • Divide CA • Adapt to available number of nodes • Master / slave division • Using QCGProactive

20. NExTml descriptor - Topology <topology> <processes processesId="Slaves_A" > <processesCount> <value> 8 </value> </processesCount> </processes> ... <link from="Slaves_A" to="Master" > <bandwith min="1Gb"/> </link> ... <processes processesId="Master" masterGroup="true" > <processesCount> <value> 1 </value> </processesCount> </processes> </topology>

21. Example QosCos Grid

22. Job scheduling

23. Job scheduling

24. Job scheduling

25. Job scheduling 1 1

27. Plug’n’Play deployment • Our target: "as simple as firing up a VM" • Existing resource manager in place does not change • LSF, PBS, Maui… • Full control of resources is maintained • No additional deployment on the grid nodes, no kernel changes, no specific linux. No nothing. • ‘Just’ deploy a QCG Head & allocate a budget

28. Oui Nide Iou Further deployments Further adoption of NExTml Porting to other schedulers, dev env, etc.

29. Performance

30. Performance

31. Performance