Evaluating Meta-Scheduling Algorithms in VLAM-G Environment V.Korkhov, A.Belloum, L.O.Hertzberger

1. Evaluating Meta-Scheduling Algorithms in VLAM-G Environment V.Korkhov, A.Belloum, L.O.Hertzberger FNWI, University of Amsterdam VLAM-G VLAM-G, the Grid-based Virtual Laboratory AMsterdam, provides a science portal for distributed analysis in applied scientific research. VLAM-G provides tools and instruments designed to help scientists in performing experiments by providing high-level interface to Grid environment. Virtual laboratory can spread over multiple organizations, enabling access to resources available across different organization domains. The core of VLAM-G concept is a virtual experiment which is composed of distributed processing modules and can be considered as a meta-application. Application with QoS Front-End Resource Manager Session Manager VIMCO Run-Time System Grid information services (NWS, Globus MDS) Grid resource allocation services (Globus GRAM) Application model and cost function VLAM-G experiment is represented by a meta-application composed of a number of components. For the application component Ci we define: comp(Ci) - the computational load of the component Ci, may be counted in the number of instructions to be executed; comm(Ci,Cj) - the communicational load between Ci and Cj, may be counted in the number of bytes transferred between the components Consider Sk, (k<=M), M – number of sites. We define: compT(Ci,Sk) - the computation time for the component Ci running on the site Sk when the site provides all its resources to the application. In case other components are scheduled in then the function may be degraded. commT(Ci,Cj,Sk,Sl) - the communication time taken for data transfers between Ci scheduled on Sk and Cj scheduled on Sl. The function may depend on the number of components sharing the link. commTT(Ci,Sk) - the total communication cost for component Ci placed on site Sk, is a function of commTfunctions for the component’s links. In the simplest case is the sum of those costs. The cost function used for pipeline applications is thus: Fabric layer (hosts, networks etc.) Scheduling algorithms used in evaluation Basic Greedy Algorithm(straightforward heuristic): extract set of available resources from Grid indexing service using minimal module requirement; sort resources according to available CPU power; sort modules according to CPU requirement (amount of processing cycles needed); try to map modules to different resources starting with the most powerful ones, only if no suitable resources left map more instances to single resource. Modified Basic Greedy Algorithm:modification of the previous algorithm, allowing to map several instances to single resource unless it decreases overall performance (estimation of partial run-time is made on each step) Computation-Network Prioritized Algorithm: here we introduce means to describe the level of meta-application relative intensity of data transfers and computations. Heuristic coefficient CN defines priority either of networking communications or computational operations for the experiment. The following formulae is used to rank resources: R=CPUrel*CN + BWrel/CN Simulated Annealing:based on probabilistic methods that avoid being stuck at local (non-global) minima. Here an objective function to be minimized is the overall execution time of an experiment. The execution time of each instance is equivalent to the "energy" of an instance. Then, "temperature" is the average of these times. Starting from some initial schedule and initial temperature, the algorithm randomly selects an instance to be remapped, randomly selects a suitable resource and remaps the instance. The total "energy" (execution time) of the experiment is estimated. Any downhill step is accepted and the process repeats. An uphill step may be accepted. This uphill decision is made by the Metropolis criterion. The Metropolis criterion attempts to permit small uphill moves while rejecting large uphill moves. Thus, the algorithm can escape from local minima. VLAM-G Architecture Simulation results For the experiments we simulated a resource pool of 20 available machines of various computing power, memory and storage capabilities. The machines were distributed across 5 domains with different bandwidths within the domains and between the domains. We combined the domains with powerful computational resources linked by a low bandwidth links with fast, but slower resource domains. The resources were represented by abstract values of CPU power (from 500 up to 3000 units), load (0.05-0.95), network bandwidth (2-100 units between domains, 10-200 within domains). Thus we tried to approach the real heterogeneity of Grid environment. To achieve this we simulated the information usually received from Grid indexing service (MDS). All the mentioned algorithms were tested in this simulation environment on several experiment topologies consisting of various number of module instances: from 3 to 10. On the figure we present the results for one set of experiment topologies. The Y axis on the charts corresponds to resulting schedule evaluation, proportional to overall runtime. The less the value is the more efficient is the schedule. The X axis represents all four examined algorithms. • Resource Manager (RM): • receives an experiment topology with module requirements (QoS); • performs resource discovery, location and selection according to module requirements • composes a number of candidate schedules that are estimated using specified cost model and resource state information; • selects optimal schedule The basic greedy algorithm usually gives one of the two worst results. The modified basic greedy algorithm has different behaviors: in topologies with small number of modules it is very effective and gives good results, though the more the size of experiment grows the less effective it becomes. For the topologies with more than 6 modules BGM gives the worst results among all the algorithms. Computation-network prioritized algorithm gives better results than basic greedy algorithms (both standard and modified) only except the case of 3 modules when BGM is the most effective. The best results (except 3 modules case) have been shown by simulated annealing algorithm, though it was the most time consuming one. Thus the simulation results have shown that heuristic mapping algorithms might be effective in some system configurations, especially in small homogeneous environment, but for generic case and complex system topologies the algorithms of random search like simulated annealing are more promising. The lack of such algorithms is increased requirement for execution time caused by much more complex and extensive computations taken. Key VLAM-G applications Material analysis: MACSLab Medicine: MRI Scanner VLAM-G GUI http://www.vl-e.nl/ Contacts: Vladimir Korkhov, e-mail: vkorkhov@science.uva.nl Adam Belloum, e-mail:adam@science.uva.nl Project Leader : L.O. Hertzberger Phone: 020 525 7464 Fax: 020 525 74 90 e-mail: bob@science.uva.nl