Ganga: Introduction

1. Ganga: Introduction • Object Orientated Interactive Job Submission System • Written in python • Based on the concept of a job object • Developed by Atlas + LHCb • With dedicated (very helpful) support team • Used widely - including outside HEP (e.g. biomedics) • Ensures that it is an experiment-neutral framework • All experiment specific extensions are plug-ins. • Very easy automated install • It just takes two commands and < 5 minutes! • Multiple backends • local, psub, condor as well as GRID (glite-wms-*) • A very powerful feature: run the same job as another process on the same machine or on the GRID just by changing its backend attribute • Both standalone and a framework • Is a complete ready to run system for submitting jobs • Also API on which additional features can be layered • Using local backend is great for development before switching to GRID • e.g. MINOS have developed a fault tolerant batch system Computing Lectures

2. Ganga: The central concept: A job • A job is the central object • Jobs are created, configured, submitted and when complete, examined. • They are persistant • Saved automatically into a Registry • Quit Ganga at any time and come back later and resume • Creation • On creation each gets unique ID • This is a serial number that can be used to access the job • Configuration • At any time can configure all aspects - executable, environment, input and output sandboxes, backend • Cloning • Having configured one job it can be cloned to create others • Submission • Job gets submitted to the configured backend • Polling • A job poller runs in the background and monitors submitted jobs • It reports state changes. • When job is complete it retrieves output and places in the job’s output directory • Resubmit • If a job fails it can be resubmitted • Removal • Once a job is no longer needed it can be removed along with its input and output directories • Global summaries • Get one line per job summary either of all jobs or just a slice (by ID or some other attribute). Computing Lectures

3. Ganga: Two Interfaces Command line • Example • To create job, configure, submit, check status and examine output:-my_job = Job()my_job.application = Executable(exe=File('~/somescript'), args=['1','2','3'])my_job.backend = LCG() my_job.submit()my_job.statusmy_job.peek(‘stdout’) • Ganga comes with ipython • help with objects - they list methods and state e.g.: help(my_job) • tab completion - just type start of data or function and hit tab e.g. my_job.p <tab> gives my_job.peek GUI Computing Lectures

4. Ganga: Other Features • Subjobs • Have a master job and a dataset it is to operate on • Job Splitter creates subjobs each taking part of the dataset • Submitting the master job submits all the subjobs • Subjobs can also be accessed individually e.g. fix and resubmit errors. • When all subjobs complete Job Merger recombines the output • Trees • Initially job repository is a flat structure • Can create folder hierarchy each with own jobs • Can be used to perform global operations on collections of jobs • Templates • Configurable like jobs but cannot be submitted • Used to create preconfigured jobs • Authentication management • GRID certificate – will ask to re-authenticate when expired • AFS token • Configuration of all aspects • By resource file read on start up • Interactively during execution • Verbosity control • By level (CRITICAL, ERROR, WARNING, INFO, and DEBUG.) • By specific parts of GANGA Computing Lectures