360 likes | 495 Views
CSF4 Meta-Scheduler. Name: Zhaohui Ding, Xiaohui Wei Email : Zhaohui.ding@email.jlu.edu.cn , weixh@jlu.edu.cn College of Computer Science & Technology Jilin University. Agenda. Meta-Scheduler & CSF4 Introduction CSF4 Architecture CSF4 Functionalities Future Work Demo and Practice.
E N D
CSF4 Meta-Scheduler Name: Zhaohui Ding, Xiaohui Wei Email : Zhaohui.ding@email.jlu.edu.cn, weixh@jlu.edu.cn College of Computer Science & Technology Jilin University
Agenda • Meta-Scheduler & CSF4 Introduction • CSF4 Architecture • CSF4 Functionalities • Future Work • Demo and Practice
What is Meta-Scheduler • Resource Allocation & Management • Heterogeneous • Distributed • Dynamic • Local Scheduler VS Meta-scheduler
Meta-Scheduler VS Local Scheduler • Local Scheduler • LSF (Load Sharing Facility) • PBS (Portable Batch System) • SGE (Sun Grid Engine) • Condor • IBM Loadleveler • Meta-Scheduler • CSF • Maui (Silver) • Gridway • Nimrod-G • Condor-G
What is CSF4 • What is CSF Meta-Scheduler • Full Name: Community Scheduler Framework • CSF4 contains a group of grid services host in GT4 • CSF4 is a full WSRF compliant meta-scheduler. • Open Source project and can be accessed at http://sourceforge.net/projects/gcsf • Developed by Jilin University, China • CSF4 has been added to Globus Toolkit 4 as an Execution Component
Meta-Scheduler VS Local Scheduler A typical deployment for Meta-Scheduler and Local Schedulers
What CSF4 Can Do? • Basic Functionalities • Submit jobs to Grid without Specifying Cluster • Monitor and Control Jobs • Provide Queuing Service • Schedule jobs and resource by custom-built polices • CSF4 Portlet (A Web browser based User Interface)
What CSF4 Can Do? (cont.) • Advanced Functionalities • Multiple Domains Resource Information Sharing • Automatic user credentials delegation • Automatic data-staging • Extensible scheduling policies • Supporting grid parallel jobs (MPICH-G2)
CSF4 – Architecture (cont.) Resource View NOTE: One RM Gram is for one cluster only Local clusters
CSF4 – Functionalities • Local Scheduler Supported • LSF • PBS • SGE • Condor • Infrastructure Supported • Globus Toolkit 4 • Globus Toolkit 2
CSF4 – Functionalities Scheduling Plug-in Framework Designed For Queuing Service Provide A set of policies Customizable Extensible
Existent Scheduling Policies • FCFS (First Come First Serve) round-robin • Default policy • Throttle • Restrict the number of jobs in a scheduling cycle • Array Job Plug-in • Design for life science applications (such as AutoDock, BLAST) • MPICH-G2 Plug-in • By using VJM, the plug-in guarantee the synchronized resource allocation can be successful • Data intensive applications plug-in (under-developing)
Schedule plug-in & scheduling policies Each policy is implemented inside a scheduling plugin module A queue can load multiple plugin modules
Array Job • AutoDock and Blast-like applications • A large number of sub-jobs. • Execute same binary • Different input/output files
Array Job (cont.) • Advantages • Submit job only once • Save submission time and memory storage
Resource Information Sharing • A MDS information provider for CSF4 • Multiple CSF4 can share the resource information
CSF4 – Functionalities (cont.) Deploy Multiple CSF4 in a Grid Community
User Credential Delegation • Automatic user credential delegation • GT4 Delegation Service • Full/Limited Delegation
Data Staging • Manual Data Staging • Which clusters I can use? • Which clusters my jobs will running on? • Where is the output data? • When will the job finish, so that I can stage-out the output data?
Automatic Data Staging • Stage-in and Stage-out without Meta-Scheduler
Automatic Data Staging • With CSF4 Automatic Data Staging
CSF4 – User interface (cont.) • Command line interface - Services available for grid users: • JobService csf-job-create, csf-job-start, csf-job-submit, csf-job-status, csf-job-stop, csf-job-resume, csf-job-list, csf-resource-list, csf-mpi-run • ReservationService csf-rsv-create, csf-rsv-status, csf-rsv-cancel csf-job-submit, … • QueuingService csf-queue-create, csf-queue-conf, csf-queue-data
CSF4 Portlet • Web Browser based Interface • Friendly interface
How to get CSF4? • Homepage: • http://sourceforge.net/projects/gcsf • Release Version (Recommended) : • http://sourceforge.net/project/showfiles.php?group_id=103105 • Developing Version (CVS, the newest but not stable) • http://sourceforge.net/cvs/?group_id=103105 • Globus Toolkit 4 Package : • In “contrib” directory
Future Work • Plug-in Framework Improvement • More scheduling policies • Optimize for Parallel Job • Optimize for Data intensive job • Workflow supported
Practices • View Credential Info • Query Available Resource • Submit A Simple Job • Submit An Array Job • Query Jobs History • View Job Output