Using AppLeS to Improve the Performance of Distributed Applications

1. An increasingly important platform for large scientific applications are collections of distributed resources Resources may include workstation clusters MPPs storage, etc. PROBLEM: How can users schedule applications to achieve performance in multi-user, distributed platforms? SOLUTION: Adapt application execution to the resources which offer the best performance T3E Viz Using AppLeS to Improve the Performanceof Distributed Applications cluster data

2. AppLeS:Adaptive Application Scheduling in Production Distributed Environments • Project Lead: Fran Berman, UCSD • AppLeS = Application-Level Scheduler • Each AppLeS is a scheduling agent for an individual distributed application • Each AppLeS develops and executes a custom application schedule • AppLeS schedules adapt to predicted deliverable resource performance at execution time. • For NAVO PET applications, AppLeS targets applications to Legion platforms • infrastructure based on object-oriented model • AppLeS uses Legion’s Kerberos model for handling security

3. PET AppLeS Project:Results for Interactive Legion Platform • Demonstration: Developed adaptive AppLeS scheduler for Legion PMHD3D (magneto-hydrodynamics) application • representative regular stencil-based application • AppLeS complements Legion by providing adaptive application scheduling mechanism • AppLeS PMHD3D Scheduler • partitions application based on prediction of deliverable performance of target resources • more work assigned to unloaded processors • Wolski’s Network Weather Service provides resource forecasts • AppLeS dynamically schedules computations on best resources, achieves improved performance over static techniques

4. Experimental Results on Legion Cluster • Profile of heavily loaded processor, when load is heavy, application performance is likely to be poor • AppLeS scheduler uses Network Weather Service to forecast when load will be heavy, develops custom application schedule which targets tasks to faster, more lightly loaded processors • In experiments on Legion cluster, static scheduler always picks the same target processors no matter what the load • Static scheduler chose this processor whereas AppLeS adaptive scheduler chose faster, more lightly loaded processors CPU Utilization

5. Improving Application Performance on the Legion Cluster • Comparison of statically-scheduled PMHD3D execution and AppLeS- scheduled PMHD3D execution when multiple users present in system. • AppLeS schedule leverages best resources at execution time, statically scheduled execution cannot • Execution time for statically-scheduled application takes 2.5 times longer on average than adaptive AppLeS-scheduled execution. • Adaptive scheduling key to performance improvement for PMHD3D.

6. Proposed Year 4-5 Activities • Project Goal: Develop AppLeS schedulers to minimizeturnaround time = wait time + execution time by leveraging both batch and interactive environments • Project Goals: • Develop AppLeS to target T3E and other batch MPPs • Develop strategy for predicting execution time on available batch and interactive environments • Develop AppLeS which reduces application turnaround time by running on batch MP, interactive cluster, orbothenvironments simultaneously -- whichever will deliver the best performance • AppLeS Home page • http://www-cse.ucsd.edu/groups/hpcl/apples/