1 / 10

RunJob in CMS

RunJob is a versatile tool for creating and configuring jobs in CMS, utilizing metadata for abstracting steps, service invocations, and job parallelization. It supports script-based job creation, external service invocations, looping, and metadata constraint modeling.

lewiswalters
Download Presentation

RunJob in CMS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. RunJob in CMS Greg Graham Discussion Slides

  2. RunJob in CMS • RunJob is an Application Configuration and Job Creation Tool • RunJob uses metadata to abstract application steps and bolt them together into a workflow (Configurators) • RunJob uses metadata to abstract service invocations and iterators to parallelize job creation (other Configurators) • RunJob has an abstract model of jobs. • These can be translated into concrete jobs for a variety of environments (local or grid) later depending on what modules are plugged in

  3. RunJob in CMS • RunJob does job creation in scripts (parallelized or not) • RunJob scripts support external service invocations, looping over data things, inline job submission, metadata constraint modeling among applications and services, persistence, and conventional provenance. • Scripts can be factored into simpler scripts and contexts. Contexts contain site level details, administrative details, logical details about constraints on the simpler scripts. • Saving the context scripts can enable “meta-provenance” in the sense that recording constraints provides clues as to why certain parameters are set the way they are instead of just their value.

  4. RunJob in CMS • Specific Uses • Official Monte Carlo Generation • Over 30 sites, over 75 million GEANT events, controlled centrally by central RefDB (pull model) • Data Challenge 2004 • Used at CERN to govern online reconstruction of simulated data at 25 Hz. • Used at FNAL to create physics plots from reconstructed data transferred there.

  5. RunJob in CMS • Extensions to RunJob • XMLP: An XML persistency mechanism • shREEK: Runtime extensions to RunJob, monitoring, and late job building. • logger: An event logger • scriptObjects: Abstract descriptions of jobs that serve as input to code generators producing actual jobs for specific environments • CAST: A GUI for construction Configurators and workflows.

  6. User Admin RunJob Scripts RunJob Contexts Linker:A container for Configurators that enables constraint propagation, workflow modeling, provenance, scripting, and context resolution of scripts Configurator: An Object Oriented, Metadata based description of services or applications with APIs for extending behaviors to many environments Databases & Catalogs Applications Execution Managers Local/Grid Resources ScriptObjects

  7. Relationship to Physics • Physics Parameters Specification • By physics group: in a context file written by a special user designated by the physics group • Members working within a physics group will inherit these parameters by using a common context • Constrains multiple applications according to the same set of constants • By knowledgeable user: in a personal MCRunjob script file or in a personal context file. • By novice user: on the command line of a pre-existing MCRunjob script written by a knowledgeable user. • In this case, the novice is restricted to specifying parameters foreseen by the author of the MCRunjob script in question.

  8. Admin User -Physics Parameters set by Physics Group eg- HiggsMass=115 GeV/c2 TopMass=273 GeV/c2 Decay Parameters from a Control DB -Physical Description of Site Resources eg- BatchManager=FBS -Configuration of Site Resources eg- FBSQueueName=CMS; Submit Jobs Inline -User’s Job Description eg- attach Pythia attach BatchManager -Physics Parameters of User Interest eg- TopMass=275 GeV/c2 -Specification of Which Context To Use RunJob Scripts RunJob Contexts Result Text -Read User Script -Read Context(s) specified by User Linker:A container for Configurators that enables constraint propagation, workflow modeling, provenance, scripting, and context resolution of scripts -Combine all specified contexts -Fully Specified Workflow Description eg- attach ControlDB attach Pythia HiggsMass=115 GeV/c2 TopMass=275 GeV/c2 Decay Parameters from a Control DB attach FBS FBS QueueName=CMS; Submit Jobs Inline Results from Configurators

  9. Linker:A container for Configurators that enables constraint propagation, workflow modeling, provenance, scripting, and context resolution of scripts -Fully Specified Workflow Description eg- attach ControlDB attach Pythia HiggsMass=115 GeV/c2 TopMass=275 GeV/c2 Decay Parameters from a Control DB attach FBS FBS QueueName=CMS; Submit Job FBS Submit Results Configurators: An Object Oriented, Metadata based description of services or applications with APIs for extending behaviors to many environments attach ControlDB Get decay parameters attach Pythia HiggsMass=115 TopMass=275 DecayParameters {x} attach FBS FBSQueueName=CMS Submit Job Now return decay parameters: {x} FBS Submit Results return Pythia scriptObject Queue CMS MySQL Control DB Pythia FBS

  10. Relationship to Physics • Provenance • By specification of fully reduced workflow specification (all contexts resolved, job level) • By specification of contexts actually used in modifying a workflow (why does a constant have value x?) • By specification of

More Related