Production Manager Tools and Ganga (New Architecture)

Production Manager Tools and Ganga (New Architecture) Ganga/Dirac Data Management meeting 16-17 October 2003 Gennady Kuznetsov

Job Control & Monitoring • Requirements for the Job Control and Monitoring script • Interpreter. • Platform independent. • Well developed language with a rich API. • Object-oriented (we need separate variable scopes). • Candidates: Python, Java, Rugby, shells • We chose Python. (Java required compilation) Gennady Kuznetsov

Current Design Job Configuration, Data Production, Bookkeeping - Design Document, M. Frank (Editor), 04/12/2001 Gennady Kuznetsov

Production Workflow Job Job Job Step Job Job Job Job Job Job Job Job Job Job Job Job Job Job Job Step Step Step Job Job Job Step Step Step Step Module Module Module Module Module Module Module Components We add Module as additional component of the architecture • Levels of usage: • Module – Programmer • Step – Production manager • Workflow – User This module structure allow us to construct any algorithm as a combination of modules. Each step generates job as a Python program. Gennady Kuznetsov

Module • Each variable has • Name • Type (for information only) • Value • Stated as “input” & “output” • Name of the instance of linked module • Name of the variable of linked module • Description Variable n 1 VariableCollection Module ModuleInstance ModuleDefinition n n 1 1 StepDefinition ModuleInstances can be connected with each other and forms a Step. Example of the generated code # Module Instance=iterration3 Type=PrintTestiterration3 = PrintTest()iterration3.value1 = iterration1.result # type=intiterration3.value2 = iterration2.result # type=doubleiterration3.execute() Each Step can be instantiated as a single Job Gennady Kuznetsov

Module Editor Python code of single module. Can be many classes. Module Name Stored as XML file Description Module variables. Gennady Kuznetsov

Step Editor Graphical representation of Module relationship (not ready) Stored as XML file, where all modules are embedded Step Name Description Variables of currently selected instance Definitions of Modules Instances of Modules Selected instance Gennady Kuznetsov

Workflow Editor Not ready yet. • It will link all jobs in a workflow instance. • The simplest workflow is a single step. • It will combine workflow and “Vector” and generate as many instances of workflow as many members in a vector. For example, it can be list of numbers (job number or name) or it can be a list of files for reconstruction or analysis. Gennady Kuznetsov

Layered Architecture Module Library Editors Programmer Module Module Editor Programmer & Production Manager Step Step Editor Production Manager & User Workflow Workflow Editor User GangaWorkflow Editor with limited functionality (configuration only) Module library is a collection of XML files on shared file system (like AFS). Can be updated by anyone. It give us a lot of flexibility and freedom. Gennady Kuznetsov

Combining all together Ganga Permanent part of the code IntegratedEnvironment Used as it is now Ganga Job Execution Code Repacked as Modules Dirac Editors Creates Flexible part of the code Module library User Interfaceto Workflow Use an interface specification from Workflow Workflow Gennady Kuznetsov

Complete Structure Submit Python job Grid, Batch or Submission Service Ganga UI Permanent Use Module LibraryFlexible & Experiment dependent Create and use Submit Python job Dirac EditorsPermanent Gennady Kuznetsov

What To Do? • If this proposal would be accepted: • For the Data challenge (MC Simulation) • Extract execution code from Ganga and reorganize it as a modular structure and pack it as a Module Library using Dirac Editors. (Ganga, Dirac) • To make appropriate modification in the Job Handler class. (Ganga) • For the User Analysis • Design formal specification for the User Interface (XML?) • Implement User Interface Rendering Engine (Ganga) • Reserve field in Workflow (XML) and design editing tool as part of the Workflow Editor (Dirac). Gennady Kuznetsov

Collaboration with Ganga • Layered architecture. • No functional limitations specified at the 1st level. • Modular architecture of the executables. • Combine development power of two groups Flexibility Save initial project development time. Good base for the code evolution. Wide and robust testing. Efficient code maintenance. Satisfy users with wide requirements. • Modular structure • Reusable code • The same modules could be tested in different experiments or by different groups. • Service standardisation • Code sharing • 3 Levels of GUI (Programmer, Production manager, End user). Gennady Kuznetsov

Production Manager Tools and Ganga (New Architecture)