150 likes | 220 Views
Introduction Scope Workflow Select Execute Compare Store Publish Authentication Deployment Summary Next Steps. Geant 4 Hadron Validation Framework. Intro: Motivation and Benefits. Motivation: Standardize testing among hadron model developers Benefits:
E N D
Introduction Scope Workflow Select Execute Compare Store Publish Authentication Deployment Summary Next Steps Geant 4 Hadron Validation Framework
Intro: Motivation and Benefits • Motivation: • Standardize testing among hadron model developers • Benefits: • Improve consistency of tests • Quality assurance of test procedures • Access to the results in a central location • Share the tools and resources • Share the comparison references
Intro: Team • Authors: • S.Baerjee – Geant 4 Hadron Model developer (FNAL) • S.Gysin – Software Professional (FNAL) • J. Yarba – Geant 4 developer (FNAL) • Reviewers: • J.Apostolakis CERN/PH • G.Folger CERN/PH • D.Elviria FNAL/CD • C.Green FNAL/CD • V.Ivantchenko CERN/PH • A.Ribon CERN/PH • D.Wright SLAC
Intro: Requirement Document • Describes ‘What’ needs to be done • Does not include ‘How’ it will be done • Will be used • to limit the scope and hence the cost • to choose technologies and resources for the design • as an agreement between the user and developer on the functionality to be developed • The most costly errors in software development are made at the requirements stage. • If we get this one wrong we will build the wrong thing
G4 Validation FrameworkScope: • Hadron models only • Regression tests • compare release to release • Validation Tests • compare to physics output • Metric for comparison • MC/data for validation • Chi-squared for regression • Unit tests and system tests are not included
Workflow – the 5 step program • Selecting and building tests and group of tests • Execute tests • Compare the results • Store the results and tests • Publish the results
Selecting Tests and Test Groups • A test is: • Executable + starting condition • Creating a test means • selecting the 2 parts and giving them a name for reference. • Creating test groups means • Making a collection of tests and giving them a name for reference • Detailed requirements about what on pg. 4
Executing Tests • The 3 magic numbers (much debated): • Design to 40,000 test per release • Currently 20,000 • Design to 200,000 CPU hours • Currently 100,000 • Completion time: 3 weeks • Storing results for 2 months after execution • Access to results by all authenticated users • Interrupt option • Must work for small and large tests • Executed on a central server • Web interface
Comparisons • Experimental data • Reference • Storing the output of the comparisons • Quality Measures: MC/Data and Chi-squared • Centralized • Web interface
Storing Results • Save as a reference • Reference store: 10 years • Short Term store: 2 weeks • Control Access by test owner • Centralized • Web Interface • Web Browsing
Publishing • What is published: • Comparisons (charts) • Tests • References • Create and organize hierarchies • requires admin permission • Web interface for browsing • Load: 20 simultaneous users • Central server
Authentication • Authentication • are you who you say you are? • Authorization • do you have permission to do this? • No Authentication needed to view published results • Editing, comparing and publishing restricted to test Owner
Deployment • Central server • Operations: G4 designated admin
Summary • The G4 Validation is categorized into 5 steps • The most resource intensive steps are: • Executing tests • Publishing/browsing the results • Recommend to design these two areas first. • They can be done in parallel and are mostly independent. • They can be used to estimate the work for the framework
Next Steps • Experts review the requirements and give us their honest opinion whether we are on the right track • Finalize the requirements • Allocate resources to the design • Design ‘executing’ and ‘publishing’ • Estimate and plan the implementation