1 / 13

OVAL

OVAL. A testing tool for CMS software http://polywww.in2p3.fr/cms/software/oval. General Description. Original goal : help to d etect automatically the changes in the physics performances of the CMS software used for data production.

reya
Download Presentation

OVAL

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. OVAL A testing tool for CMS software http://polywww.in2p3.fr/cms/software/oval David Chamont (CMS - LLR)

  2. General Description • Original goal : help to detect automatically the changes in the physics performances of the CMS software used for data production. • Current features of the tool : compile the provided testprograms, run them in specified runtime conditions (shell variables + auxiliary files), compare the output with references. • User interface is CVS-like, configuration files are XML-like. It is written in PERL and can be configured for any build system (scram, make). David Chamont (CMS - LLR)

  3. Outline • General description. • Configuration : OvalFile. • Commands : build/run/diff. • Results : log files. • Other features. • Summary. • Plans. David Chamont (CMS - LLR)

  4. Configuration : OvalFile <difflineexpr="^OVAL:"><diffnumberexpr="^OVAL NUM: .* is (.*)$" tolerance="5%"> <file name=".orcarc">MaxEvents = 500Random:Seeds = 0 3</file> <var name="FEDERATION"value="cmsuf01…"> <environment name="pt15"><var name="DATASET"value="eg_ele_pt15"><program name="Clusters.cpp"><program name="Electrons.cpp" args="-geo detailed"></environment> <environment name="flow"><var name="DATASET"value="jm_minbias"><program name="EnergyFlow.cpp"></environment> David Chamont (CMS - LLR)

  5. Commands : build/run/diff… cms038:ElectronPhoton> ls Makefile CVS Clusters.cpp Clusters.ref Electrons.cpp Electrons.ref EnergyFlow.cpp OvalFile David Chamont (CMS - LLR)

  6. Commands : …build/run/diff… cms038:ElectronPhoton> oval prod Clusters: build, run, diff. Electrons: build, run, diff (DIFFS). David Chamont (CMS - LLR)

  7. Commands : …build/run/diff cms038:ElectronPhoton> ls Makefile CVS Clusters.cpp Clusters.log Clusters.ref Electrons.cpp Electrons.log Electrons.ref EnergyFlow.cpp OvalFile electrons.hbook David Chamont (CMS - LLR)

  8. Results : log files… [oval build] ======================================== [oval build] make Electrons [oval build] ======================================== /usr/local/gcc-alt-2.95.2/bin/c++ -O2 -pthread -rdynamic -L/afs/cern.ch/user/c/chamont/ORCA_5/lib/Linux__2.2 -L/afs/cern.ch/cms/Releases/ORCA_5/lib/Linux__2.2 -L/afs/cern.ch/sw/lhcxx/specific/redhat61/gcc-2.95.2/3.6.2/lib -L/afs/cern.ch/cms/cmsim/cms122/lib/i386_linux22 -L/cern/2001/lib Electrons.o -o Electrons -lElectronFacilities -lEgammaH4Support -lEgammaL1Tools … -lmathlib -lblas -lkernlib -lpythia6152 -lz -lnsl -ldl -lg2c -lm David Chamont (CMS - LLR)

  9. Results : …log files… [oval run] ========================================== [oval run] LD_LIBRARY_PATH = [oval run] /afs/cern.ch/user/c/chamont/ORCA_5/lib/Linux__2.2 [oval run] :/afs/cern.ch/cms/Releases/ORCA_5/lib/Linux__2.2 [oval run] :/afs/cern.ch/sw/lhcxx/i386_linux22/Objectivity/5.2.1/lib [oval run] FEDERATION = cmsuf01::/cms/jet0900/jet0900.boot [oval run] DATASET = eg_ele_pt15 ... [oval run] .orcarc: [oval run] MaxEvents = 500 [oval run] Random:Seeds = 0 3 [oval run] ========================================= Welcome to COBRA ... David Chamont (CMS - LLR)

  10. Results : …log files [oval diff] ======================================== [oval diff] diff line : /^OVAL:/ [oval diff] diff number : /^OVAL NUM: .* is (.*)$/ ~5% [oval diff] ======================================== ref#1452!=log#2053 < OVAL: 12 electrons --- > OVAL: 11 electrons ref#1972!~ log#2592 (>5%) < OVAL NUM: energy mean is 29.7275 --- > OVAL NUM: energy mean is 27.4728 David Chamont (CMS - LLR)

  11. Other Features • Oval is not restrained to programs. For complex testing tasks, especially when databases are involved, one can also handle some scripts (no build step needed). • The same program or script can be executed several times with different arguments, and/or within different environments. • If there are many tests, one can organize them in a hierarchy of Unix directories, and create OvalFiles at any level. • The results of an oval command can be mailed to watchers as defined in the configuration. David Chamont (CMS - LLR)

  12. Summary • Scan the output, whatever it is. • Can be combine with any build system. • Very easy to use, except the tuning of regular expressions (more examples available soon). • No helper code provided. The users still have to write their test programs. • CMS use it for regression tests ; some developers run it daily on their code (kind of extreme programming !) • LCG plans to use it together with CppUnit. David Chamont (CMS - LLR)

  13. Plans • Improve infrastructure & support : • improve internal documentation, • add a tutorial, • progressively move to LCG savannah server. • React to users’ feedback • enable the users to provide their own diff command, • add a reuse mechanism for the environments, • automatic generation of trivial test programs for a given set of header files. David Chamont (CMS - LLR)

More Related