Using the NSF TeraGrid for Parametric Sweep CMS Applications

1. Jeffrey P. Gardner Edward Walker Vladimir Litvin Pittsburgh Supercomputing Center Texas Advanced Computing Center California Institute of Technology Using the NSF TeraGrid for Parametric Sweep CMS Applications

2. The NSF TeraGrid • Links nine resource provider sites via a high speed 10/30 Gbps network backbone, providing, in aggregate, 20 teraflops of computing power, 1 petabyte of storage capacity, and high-end facilities for remote visualization of computing results. • Compute resources composed of a heterogeneous mix of clusters with different architectures and operating systems, running different workload management systems, with different local configuration, specifying different queues, with different job submission and run-time limits.

3. Requirements • Schedulers at each Teragrid site • Should be able to gracefully handle ~1000 single-processor jobs at a time • Metascheduler (Teragrid doesn’t have metascheduler) • distributes jobs across the Teragrid • Metascheduler must be able to handle ~100,000 jobs

4. Solution: PBS/LSF? • In theory, existing Teragrid schedulers like PBS or LSF should provide the answer. • In practice, this does not work. • Teragrid nodes are multiprocessor • Only 1 PBS job per node • Teragrid machines frequently restrict the number of jobs a single user may run • Asking for many processors at once communicates your actual resource requirements.

5. Solution: Clever shell scripts? • We could submit a single PBS job that uses many processors. • Now we have a reasonable number of PBS jobs. • Scheduling priority would reflect our actual resource usage. • This still has problems. • Each job takes a different amount of time to run: we are using resources inefficiently.

6. Metacheduler Solution: Condor-G? • Condor-G will schedule an arbitrarily large number of jobs across multiple grid resources using Globus. • However, 1 serial Condor-G job = 1 PBS job, so we are left with the same PBS limitations as before: • Teragrid nodes are multiprocessor • Only 1 PBS job per node • Teragrid machines frequently restrict the number of jobs a single user may run.

7. PBS Job PE PE PE PE PE PE The Real Solution: Condor + GridShell • The real solution is to submit one large PBS on each Teragrid node, then use a private scheduler to manage serial work units within the PBS job. Private Scheduler Serial Work Units • Vocabulary: • JOB: (n) a thing that is submitted via Globus or PBS • WORK UNIT: (n) An independent unit of work (usually serial), such as the analysis of a single run

8. PBS Job PE PE PE PE PE PE The Real Solution: Condor + GridShell • The real solution is to submit one large PBS on each Teragrid node, then use a private scheduler to manage serial work units within the PBS job. GridShell Condor Private Scheduler Serial Work Units • Vocabulary: • JOB: (n) a thing that is submitted via Globus or PBS • WORK UNIT: (n) An independent unit of work (usually serial), such as the analysis of a single run

9. Features: GridShell/Condor on the TeraGrid • Automatically throttles condor_startd job submissions based on configurable local site policies. • Tolerates and recovers in the presence of transient faults in the WAN and login nodes. • Balances condor_startd jobs between sites based on queue wait times. • Shunts condor_startd jobs off temporary faulty sites. • Automatically renews grid proxy credentials across sites within the command environment. • Current users: Caltech CMS and NVO. SUBMITTED 70,000 jobs through clusters on TeraGrid to date.

10. Why GridShell/Condor? • scalability - the actual parametric job submission is done directly to the compute nodes and not through the front-end node of the cluster; • fault-tolerance –agents at a front-end node of a cluster maintain the condor_startd submissions locally, allowing transient WAN outages and periodic front-end node reboots to be resolved independently, in isolation from the rest of the system; and • usability – the entire Condor submission, monitor, and control infrastructure is leveraged as a common job management environment for the user.

11. GridShell/Condor Process Architecture

12. Condor Overview • Condor was first designed as a CPU cycle harvester for workstations sitting on people’s desks. • Condor is designed to schedule large numbers of jobs across a distributed, heterogeneous and dynamic set of computational resources.

13. Advantages of Condor • Condor is flexible • Resources can be any mix of architectures • Resources do not need a common filesystem • Resources do not need common user accounting • Condor is dynamic • Resources can disappear and reappear • Condor is fault-tolerant • Jobs are automatically migrated to new resources if existing one become unavailable.

14. Central Manager Submission Machine Execution Machine startd schedd negotiator collector Condor Daemon Layout (very simplified) (To simplify this example, the functions of the Negotiator are combined with the Collector) Startd sends system specifications (ClassAds) and system status to Central Manager

15. Central Manager Submission Machine Execution Machine startd schedd collector Condor Daemon Layout (very simplified) Schedd sends job info to Central Manager User submits Condor job

16. Central Manager Submission Machine Execution Machine startd schedd collector Condor Daemon Layout (very simplified) Central Manager uses information to match Schedd jobs to available Startds

17. Submission Machine Execution Machine Central Manager startd schedd collector Condor Daemon Layout (very simplified) Schedd sends job to Startd on assigned execution node

18. “Personal” Condor on a Teragrid Platform • Condor daemons can be run as a normal user. • Condor “GlideIn”™ ability supports the ability to launch condor_startd’s on nodes within an LSF or PBS job.

19. Central Manager Submission Machine Execution PE Execution PE Execution PE startd startd startd schedd collector “Personal” Condor on a Teragrid Platform(Condor runs with normal user permissions) Login Node PBS Job on Compute Nodes- GlideIn

20. GridShell Overview • Allows users to interact with distributed grid computing resources from a simple shell-like interface. • extends TCSH version 6.12 to incorporates grid-enabled features: • parallel inter-script message-passing and synchronization • output redirection to remote files • parametric sweep

21. GridShell Examples Redirecting the standard output of a command to a remote file location using GlobusFTP: a.out > gsiftp://tg-login.ncsa.teragrid.org/data Message passing between 2 parallel tasks: if ( $_GRID_TASKID == 0) then echo "hello" > task_1 else Set msg=`cat < task_0` endif Executing 256 instances of a job: a.out on 256 procs

22. Merging GridShell with Condor • Use GridShell to launch Condor GlideIn jobs at multiple grid sites • All Condor GlideIn jobs report back to a central collector • This converts the entire Teragrid into your own personal Condor pool!

23. Login Node Gridshell event monitor Merging GridShell with Condor Purdue (IA32) UC (IA32) TACC (IA32) GridShell process User starts GridShell Session at TACC

24. Login Node Login Node Login Node Gridshell event monitor Gridshell event monitor Gridshell event monitor Merging GridShell with Condor Purdue (IA32) UC (IA32) TACC (IA32) GridShell session starts event monitor on remote login nodes via Globus GridShell process Condor process

25. Login Node Login Node Login Node Gridshell event monitor Gridshell event monitor Gridshell event monitor collector schedd Merging GridShell with Condor Purdue (IA32) UC (IA32) TACC (IA32) Local event monitor starts condor daemons on login node GridShell process Condor process

26. Login Node PBS Job Login Node PBS Job PBS Job Login Node Gridshell event monitor Gridshell event monitor Gridshell event monitor gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex Master gtcsh-exec Master gtcsh-exec Master gtcsh-exec collector schedd All event monitors submit PBS/LSF jobs. These jobs start GridShell gtcsh-exec on all processors Purdue (IA32) UC (IA32) TACC (IA32) GridShell process Condor process

27. Login Node PBS Job Login Node PBS Job Login Node PBS Job startd startd startd startd startd startd startd startd startd Gridshell event monitor Gridshell event monitor Gridshell event monitor gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex Master gtcsh-exec Master gtcsh-exec Master gtcsh-exec collector schedd gtcsh-exec on each processor starts a Condor startd. Heartbeat is maintained between all gtcsh-exec processes Purdue (IA32) UC (IA32) TACC (IA32) “Heartbeat” GridShell process Condor process

28. Login Node PBS Job Login Node PBS Job Login Node PBS Job startd startd startd startd startd startd startd startd startd Gridshell event monitor Gridshell event monitor Gridshell event monitor gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex Master gtcsh-exec Master gtcsh-exec Master gtcsh-exec collector schedd gtcsh-exec on each processor starts a Condor startd Purdue (IA32) UC (IA32) TACC (IA32) GridShell process Condor process

29. Login Node PBS Job Login Node PBS Job Login Node PBS Job startd startd startd startd startd startd startd startd startd Gridshell event monitor Gridshell event monitor Gridshell event monitor gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex gtcsh-ex Master gtcsh-exec Master gtcsh-exec Master gtcsh-exec collector schedd Condor schedd distributes Condor jobs to compute nodes Purdue (IA32) UC (IA32) TACC (IA32) GridShell process Condor process

30. CMS production chain (preparations) • First of all we are installing all CMS software on clusters and pileup at lonestar (we don’t have enough disk space to set up pileup everywhere) • We are defining the name of the clusters because now we have only one cluster where digitization with pileup and final analysis must be done – lonestar • Copy both the condor_config.master.template and condor_config.glidein.template to our own private directory. Then in our login script (.cshrc or .bashrc), set the environment variable _GRID_TEMPLATE_DIR to point to this directory • Add the unique definition of the cluster in condor_config.glidein.template

31. CMS production chain (preparations) • Define specific variables in .cshrc which will be different from cluster to cluster: • $WORKDIR -where temporary subdir will be created and everything will be run • $WWORKDIR - where temporary subdir will be created and everything will be run, as it is originally defined in OS. Usually it is the same with$WORKDIR, but sometimes it is differ. It is needed for FCrenamePFN to get rid of the absolute paths before tarring files and moving them out. • $OSCAR_SRC - main OSCAR source directory where "eval `scram runtime -csh ` must be done to set everything up. • $ORCA_SRC - main ORCA source directory where "eval `scram runtime -csh ` must be done to set everything up. • $TOPRELEASE - where all releases of CMS software were installed. • $VCAL - place where VCAL shower library was placed • $DETGEOMFILE - place where cms133.rz is placed • $PILEUP - where are the pileup data and PoolFileCatalog_pileup.xml resided

32. Start GridShell Session Write a simple GridShell configuration script: # vo.conf: # A GridShell config script tg-login.uc.teragrid.org tg-login.purdue.teragrid.org Start GridShell session. This submits PBS jobs at each site and starts local Condor daemons % vo-login –n 4:16 –H ~/vo.conf –G –W 600 Spawning on tg-login.uc.teragrid.org Spawning on tg-login.purdue.teragrid.org waiting for VO participants to callback... ###########Done. -n 4:16 Start 4 PBS job per Teragrid machine, each with 16 processors -W 600 Each PBS job has a wallclock limit of 600 minutes -H vo.conf Configuration file

33. CMS production chain • One main preparation script which massaging the scripts and putting all run numbers etc in place (walltime is in secs): • ./createCondorSubmit.csh <walltime_oscar_init> <walltime_oscar_job> <walltime_orca_init> <walltime_orca_job> <walltime_tt> • This script has all info hardcoded inside – number of events per run, dataset and owner names, condor submission parameters, etc. So, if you need to check this out and change the parameters – this is the place to look at. • It invokes ./createCondorJob.csh which is making preparations for single job with specified run number.

34. CMS production chain • The main part of createCondorJob.csh: ./makeDAG.pl $titles $counter >>! $TOPDIR/work/${titles}.dag ./makeOscarInitCDR.pl $titles $counter $OSCAR_OWNER $OSCAR_DATASET $wall_time_oscar_init >! $TOPDIR/work/${titles}_oscar_init_$counter.cdr ./makeOscarJobCDR.pl $titles $counter $OSCAR_OWNER $OSCAR_DATASET $wall_time_oscar_job $PYTHIA_DATASET_NAME >! $TOPDIR/work/${titles}_oscar_job_$counter.cdr ./makeOrcaInitCDR.pl $titles $counter $OSCAR_OWNER $OSCAR_DATASET $ORCA_OWNER $ORCA_DATASET $wall_time_orca_init >! $TOPDIR/work/${titles}_orca_init_$counter.cdr ./makeOrcaJobCDR.pl $titles $counter $OSCAR_OWNER $OSCAR_DATASET $ORCA_OWNER $ORCA_DATASET $wall_time_orca_job >! $TOPDIR/work/${titles}_orca_job_$counter.cdr ./makeTTCDR.pl $titles $counter $ORCA_OWNER $ORCA_DATASET $wall_time_tt >! $TOPDIR/work/${titles}_tt_$counter.cdr

35. CMS production chain • DAG template JOB OSCAR_INIT /home/tg456083/Prod2005/aajj/work/aajj_oscar_init.cdr JOB OSCAR_JOB /home/tg456083/Prod2005/aajj/work/aajj_oscar_job.cdr JOB ORCA_INIT /home/tg456083/Prod2005/aajj/work/aajj_orca_init.cdr JOB ORCA_JOB /home/tg456083/Prod2005/aajj/work/aajj_orca_job.cdr JOB TT /home/tg456083/Prod2005/aajj/work/aajj_tt.cdr PARENT OSCAR_INIT CHILD OSCAR_JOB PARENT OSCAR_JOB CHILD ORCA_INIT PARENT ORCA_INIT CHILD ORCA_JOB PARENT ORCA_JOB CHILD TT RETRY OSCAR_INIT 0 RETRY OSCAR_JOB 0 RETRY ORCA_INIT 0 RETRY ORCA_JOB 0

36. CMS production chain • CDR template # Common job definitions" Executable = /home/tg456083/Prod2005/aajj/startup/orca_job_wrapper Getenv = True Notification = Never universe = vanilla Error = /home/tg456083/Prod2005/aajj/work/err_orca_job.err Output = /home/tg456083/Prod2005/aajj/work/out_orca_job.out Log = /home/tg456083/Prod2005/aajj/logs/log_orca_job.log should_transfer_files = TRUE when_to_transfer_output = ON_EXIT arguments = $titles $counter $orca_owner $orca_dataset requirements = TARGET.TimeToLive > _WALLTIME_ORCA_JOB_ Requirements = TARGET.Project == “TACC_RESOURCES" Queue

37. CMS production chain • *_wrapper is doing all job (orca_job_wrapper e.g.): • Checking all setup variables • Transferring files from previous step to the area where this job will be executed • Running the job • Attaching run and making FixColl • Moving data out

38. CMS production chain • Submit the whole DAG to condor # Filename: /home/tg456083/Prod2005/aajj/work/aajj.dag.condor.sub # Generated by condor_submit_dag /home/tg456083/Prod2005/aajj/work/aajj.dag universe = scheduler executable = /home/tg456083/condor2//bin/condor_dagman getenv = True output = /home/tg456083/Prod2005/aajj/work/aajj.dag.lib.out error = /home/tg456083/Prod2005/aajj/work/aajj.dag.lib.out log = /home/tg456083/Prod2005/aajj/work/aajj.dag.dagman.log remove_kill_sig = SIGUSR1 arguments = -f -l . -Debug 3 -Lockfile /home/tg456083/Prod2005/aajj/work/aajj.dag.lock Condorlog /home/tg456083/Prod2005/aajj/logs/personal_aajj.log –Dag /home/tg456083/Prod2005/aajj/work/aajj.dag -Rescue /home/tg456083/Prod2005/aajj /work/aajj.dag.rescue -MaxJobs 200 -MaxPre 200 -MaxPost 200 -NoPostFail environment =_CONDOR_DAGMAN_LOG=/home/tg456083/Prod2005/aajj/work/aajj.dag.dagman.out;_CONDOR_MAX_DAGMAN_LOG=0 queue

39. Conclusion • We have used GridShell to turn the TeraGrid into our own personal Condor pool • We can submit Condor jobs, and Condor will schedule these jobs across multiple TeraGrid site • TeraGrid sites do not need to share architecture or queuing systems • GridShell also allows us to use TeraGrid protocols to transfer our input and output data • All of this fits into existing Teragrid software. • We can run CMS production chain through this system • 40000 were done • Will be demonstrated at SC2005 in Seattle in November 2005