1 / 18

Data Management

Data Management. Status of CVS repository Production databases Production tools. CVS. Separate area for each sub-project Internal structure cmt : requirements <package> : public headers src : source opt: src/<component> Opt: python, cmds, images, etc.

amora
Download Presentation

Data Management

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Management Status of CVS repository Production databases Production tools

  2. CVS • Separate area for each sub-project • Internal structure • cmt : requirements • <package> : public headers • src : sourceopt: src/<component> • Opt: python, cmds, images, etc POOL

  3. Job . Configuration App/ Pack. Mgr Data . Production Prod.Mgr Book . keeping Physicist Components Job Sub- mission Tool Real dataflow Apparent dataflow POOL

  4. Components • Job configuration Markus • Data production Markus • Job submission Andrei • Bookkeeping Sebastien/Francoise POOL

  5. Job Configuration • Database ok. • Simple HTTP based Editor • XML-RPC based interface • Boost + python • Tool to import applications POOL

  6. Job Configuration: Editor POOL

  7. Pythia – v2 1) Generation Gauss - v5 Gauss - v5 Gauss - v5 Gauss - v5 2) Simulation Brunel - v12 Brunel - v12 3)Reconstruct 4) Mini GenTag v7 Work Flow Execution Flow (Jobs) Data Production POOL

  8. Step Step Step Data Production - Details Workflow Run • Name • State • N(evt) to be processed • Name • Step Seq. # • Nevt(Split) • Nevt(Merge) Step Step • State • Center • Machine Step <Rec> Job <Tags> Job • Name • Input file types • Output file types • Application Name Output File Input File Input File Output File FileType File Type Job Configuration DB Workflow description POOL

  9. Data Production • Work flows • Database ok. • Simple HTTP based Editor • XML-RPC based interface • Boost + python • Data Production • Database ok. • Simple HTTP based Editor • XML-RPC based interface • Boost + python Instantiation of work flows to production runs Very primitive POOL

  10. 1) Generation 2) Simulation 3)Reconstruct 4) Mini Work Flow Instantiation Pythia – v2 Gauss - v5 Gauss - v5 Gauss - v5 Gauss - v5 All processes in one job Tools missing… Brunel - v12 Brunel - v12 GenTag v7 Work Flow Execution Flow (Jobs) POOL

  11. Work Flow Instantiation • Script generation through servlet • Select work flow • Instantiate work flow to production run • Add basic information • # of events • CPU requirements etc. • Generate scripts • Possibility to extend existing runs • Scripts are then picked up by Andrei’s client POOL

  12. Interacting Components • Main strategy on component failure(s) • The system must stay operable • Intermediately produced information items are cached locally until servers are up again • Production server (see next slide) • Production client (see Andrei’s talk) • Servers/servlets to publish bookkeeping information(see Sebastiens talk) POOL

  13. Component Collaboration Production Editor Work flow Editor Production DB Production data Scripts Edit Instantiate Workflow Production Server • Job request • Status updates Prod.Mgr Production Client Bookkeeping info XML Bookkeeping POOL

  14. Script Generation setenv PRODUCTION_IDENTIFIER 400000067 setenv JOB_NAME 400000369 # Target operating system: linux # Parameters of step MinBias.Dec.2002 setenv MBIAS_QQ_USER_FILE None setenv MBIAS_EVTYPE 61 # Parameters of step B2pipi.Dec.2002 setenv B2PIPI_QQ_USER_FILE B2pipi.dec setenv B2PIPI_EVTYPE 411300 # Parameters of work flow B2pipi.Dec.2002 setenv SIMULATION_APP SICBMC setenv SIMULATION_VSN v254r2 setenv RECONSTRUCTION_APP Brunel setenv RECONSTRUCTION_VSN v13r1 setenv DBASE_PKG dbase setenv DBASE_VSN v248r4 setenv XMLDDDB_PKG XmlDDDB setenv XMLDDDB_VSN v8r1 setenv QQ_USER_FILE ${MBIAS_QQ_USER_FILE},${B2PIPI_QQ_USER_FILE} setenv mcevtype ${MBIAS_EVTYPE},${B2PIPI_EVTYPE} setenv SPILLOVER on setenv mcmonihisto on “Instantiated”Workflow POOL

  15. Production Editor POOL

  16. Work Flow Editor POOL

  17. Production DBs - Summary • Primitive production tools exist • WWW based • Basic interface for production clients exists • Needs testing • The environment to start developing production tools is in place • Problems with MySQL • MySQL does not support nested SELECT • MySQL will not come in the very near future POOL

  18. Job List • Import Brunel versions used in production run to job configuration database • Construct “reasonable” work flow • Test generated production script(s) • Test production server-client collaboration POOL

More Related